Commit graph

95 commits

Author SHA1 Message Date
Tom Rogers
4e582fb55d
Improving Self-Hosting and Removing 3rd Party dependencies. (#4513)
Some checks failed
Build Docker Images / Build docker images (push) Has been cancelled
Build Self-Hosting Docker Images / Build self-host docker images (push) Has been cancelled
Run tests / Run Codebase tests (push) Has been cancelled
Run tests / Build docker images (push) Has been cancelled
* fix: Library Header layout shift

* Bump Github Actions versions.

* Self-Hosting Changes

* Fix Minio Environment Variable

* Just make pdfs successful, due to lack of PDFHandler

* Fix issue where flag was set wrong

* Added an NGINX Example file

* Add some documentation for self-hosting via Docker Compose

* Make some adjustments to Puppeteer due to failing sites.

* adjust timings

* Add start of Mail Service

* Fix Docker Files

* More email service stuff

* Add Guide to use Zapier for Email-Importing.

* Ensure that if no env is provided it uses the old email settings

* Add some instructions for self-hosted email

* Add SNS Endpoints for Mail Watcher

* Add steps and functionality for using SES and SNS for email

* Uncomment a few jobs.

* Added option for Firefox for parser. Was having issues with Chromium on Docker.

* Add missing space.

Co-authored-by: Russ Taylor <729694+russtaylor@users.noreply.github.com>

* Fix some wording on the Guide

* update browser extension to handle self-hosted instances

* add slight documentation to options page

* Fix MV

* Do raw handlers for Medium

* Fix images in Medium

* Update self-hosting/GUIDE.md

Co-authored-by: Mike Baker <1426795+mbaker3@users.noreply.github.com>

* Update Guide with other variables

* Add The Verge to JS-less handlers

* Update regex and image-proxy

* Update self-hosting/nginx/nginx.conf

Co-authored-by: Mike Baker <1426795+mbaker3@users.noreply.github.com>

* Update regex and image-proxy

* Update self-hosting/docker-compose/docker-compose.yml

Co-authored-by: Mike Baker <1426795+mbaker3@users.noreply.github.com>

* Fix Minio for Export

* Merge to main

* Update GUIDE with newer NGINX

* Update nginx config to include api/save route

* Enable Native PDF View for PDFS

* Enable Native PDF View for PDFS

* feat:lover packages test

* feat:working build

* feat:alpine build

* docs:api dockerfile docs

* Write a PDF.js wrapper to replace pspdfkit

* Revert changes for replication, set settings to have default mode

* build folder got removed due to gitignore on pdf

* Add Box shadow to pdf pages

* Add Toggle for Progress in PDFS, enabled native viewer toggle

* Update node version to LTS

* Update node version to LTS

* Fix Linting issues

* Fix Linting issues

* Make env variable nullable

* Add touchend listener for mobile

* Make changes to PDF for mobile

* fix(android): change serverUrl to selfhosted first

* feat:2 stage alpine content fetch

* feat:separated start script

* fix:changed to node 22

* Add back youtube functionality and add guide

* trigger build

* Fix cache issue on YouTube

* Allow empty AWS_S3_ENDPOINT

* Allow empty AWS_S3_ENDPOINT

* Add GCHR for all images

* Add GCHR For self hosting.

* Add GCHR For self hosting.

* Test prebuilt.

* Test prebuilt

* Test prebuilt...

* Fix web image

* Remove Web Image (For now)

* Move docker-compose to images

* Move docker-compose files to correct locations

* Remove the need for ARGS

* Update packages, and Typescript versions

* Fix

* Fix issues with build on Web

* Correct push

* Fix Linting issues

* Fix Trace import

* Add missing types

* Fix Tasks

* Add information into guide about self-build

* Fix issues with PDF Viewer

---------

Co-authored-by: keumky2 <keumky2@woowahan.com>
Co-authored-by: William Theaker <wtheaker@nvidia.com>
Co-authored-by: Russ Taylor <729694+russtaylor@users.noreply.github.com>
Co-authored-by: David Adams <david@dadams2.com>
Co-authored-by: Mike Baker <1426795+mbaker3@users.noreply.github.com>
Co-authored-by: m1xxos <66390094+m1xxos@users.noreply.github.com>
Co-authored-by: Adil <mr.adil777@gmail.com>
2025-01-27 13:33:16 +01:00
Jackson Harper
400e69465b Disable non-essential jobs to give more resources to the exporter 2024-10-30 20:57:11 +08:00
Hongbo Wu
70d6e9dcae reduce logs 2024-08-29 18:18:17 +08:00
Hongbo Wu
48b3f736f0 wait for write stream to finish 2024-08-27 14:57:31 +08:00
Hongbo Wu
47cea15480 keep up to 1 hour complete tasks and up to 1 day failed tasks 2024-08-22 15:27:59 +08:00
Hongbo Wu
5bd272dde0 use one queue with different priority for fetching content of rss feed item or saved url 2024-08-21 12:24:35 +08:00
Hongbo Wu
08fbb8aebf use different queues for fast,slow and rss content fetch jobs 2024-08-21 12:24:35 +08:00
Hongbo Wu
87b4ec503e enqueue content-fetch task to the queue 2024-08-21 12:24:35 +08:00
Hongbo Wu
70829a6b88 remove initial observation 2024-07-01 12:08:56 +08:00
Hongbo Wu
6df8c71035 reduce bullmq worker concurrency to 2 2024-06-18 17:28:19 +08:00
Hongbo Wu
88e3d648c8 add a REST API to trigger the expire folder job 2024-06-13 18:36:33 +08:00
Hongbo Wu
6f496b9336 add folder expiration jobs 2024-06-13 18:30:27 +08:00
Hongbo Wu
9ef91ff1ca add an REST API to trigger the prune trash job 2024-06-13 14:48:14 +08:00
Hongbo Wu
71d07caf94 run prune trash job at 3am UTC daily 2024-06-12 19:27:42 +08:00
Hongbo Wu
f29592d671 worker can concurrently process 10 jobs 2024-06-12 18:40:08 +08:00
Hongbo Wu
57695c370c monitor each job latency in prometheus 2024-06-12 13:35:04 +08:00
Hongbo Wu
f052ab55a0 monitor home feed job latency in prometheus 2024-06-12 13:01:36 +08:00
Hongbo Wu
76383d8a33 feat: generate preview content if excerpt less than 180 chars 2024-06-06 19:08:41 +08:00
Hongbo Wu
efb7160373 rename just read feed to home 2024-05-27 21:11:08 +08:00
Hongbo Wu
41b9fef316 score library item after saved 2024-05-27 11:45:51 +08:00
Hongbo Wu
a48a614676 save just read feed in redis sorted set 2024-05-24 14:52:21 +08:00
Hongbo Wu
01ebcbb16b add bulk upload original content job 2024-05-10 14:37:05 +08:00
Hongbo Wu
14e91d338d do not create user profile and topics 2024-04-30 12:48:19 +08:00
Hongbo Wu
2f7c18b363 add more logs 2024-04-23 15:50:15 +08:00
Hongbo Wu
1b6f4976d3 remove timeout 2024-04-23 15:50:15 +08:00
Hongbo Wu
de258ae0bf timeout if job takes more than 10 minutes 2024-04-23 15:50:15 +08:00
Hongbo Wu
71477b1712 implement create digest job 2024-04-16 21:00:51 +08:00
Hongbo Wu
03e497b489 use a separate queue for digest job 2024-04-16 21:00:26 +08:00
Hongbo Wu
12aa7d4447 create digest job api 2024-04-16 21:00:10 +08:00
Hongbo Wu
b4911639c7 stub create digest job 2024-04-16 21:00:09 +08:00
Hongbo Wu
f66d3fa36c migrate inbound-email tasks from pubsub to bullmq 2024-04-11 11:08:22 +08:00
Hongbo Wu
5f239d2dcb update graceful shutdown 2024-04-11 11:08:22 +08:00
Hongbo Wu
8af2bb076c rename send mail functions 2024-04-05 16:59:20 +08:00
Hongbo Wu
10dcb922f2 Enqueue confirmation email 2024-04-05 16:57:02 +08:00
Jackson Harper
b5bea17ffe Export the age of the oldest job in the queue 2024-03-28 10:25:44 +08:00
Hongbo Wu
92abfed2df replace logger.warn with warning 2024-03-27 15:41:13 +08:00
Hongbo Wu
390d12eea1 add export to integration and get task api 2024-03-21 11:14:23 +08:00
Jackson Harper
629c044273 Queue the transcript processing as a separate job
Handle YouTube in two steps, first get metadata then
get the transcript.
2024-03-14 15:09:53 +08:00
Jackson Harper
2dbd16a61e Pull duration and description from YouTube metadata 2024-03-14 12:48:56 +08:00
Jackson Harper
f7225b298a Rebase 2024-03-14 12:48:56 +08:00
Hongbo Wu
d41459a5cd log unhandled error 2024-03-06 12:04:16 +08:00
Jackson Harper
bbe6204b91 summaries 2024-02-28 15:12:09 +08:00
Hongbo Wu
fb576a58a0 initial export 2024-02-22 15:20:24 +08:00
Hongbo Wu
f94267ee1a enqueue export job 2024-02-22 13:38:51 +08:00
Hongbo Wu
ce19218e15 get queue priority from a table 2024-02-21 22:01:04 +08:00
Hongbo Wu
cd311d6454 do not update db options in queue-processor 2024-02-20 19:09:33 +08:00
Hongbo Wu
59d4d78f78 connection idle timeout in 10 seconds 2024-02-20 18:33:21 +08:00
Hongbo Wu
485caec803 append job version to the job id 2024-02-07 18:42:10 +08:00
Hongbo Wu
8c23cff17b remove limit on the sync reading progress job 2024-02-07 11:09:30 +08:00
Hongbo Wu
95fabe1dbf keep completed jobs for up to 24 hours and failed jobs for 7 days 2024-02-06 14:21:00 +08:00