Commit graph

80 commits

Author SHA1 Message Date
Tom Rogers
4e582fb55d
Improving Self-Hosting and Removing 3rd Party dependencies. (#4513)
Some checks failed
Build Docker Images / Build docker images (push) Has been cancelled
Build Self-Hosting Docker Images / Build self-host docker images (push) Has been cancelled
Run tests / Run Codebase tests (push) Has been cancelled
Run tests / Build docker images (push) Has been cancelled
* fix: Library Header layout shift

* Bump Github Actions versions.

* Self-Hosting Changes

* Fix Minio Environment Variable

* Just make pdfs successful, due to lack of PDFHandler

* Fix issue where flag was set wrong

* Added an NGINX Example file

* Add some documentation for self-hosting via Docker Compose

* Make some adjustments to Puppeteer due to failing sites.

* adjust timings

* Add start of Mail Service

* Fix Docker Files

* More email service stuff

* Add Guide to use Zapier for Email-Importing.

* Ensure that if no env is provided it uses the old email settings

* Add some instructions for self-hosted email

* Add SNS Endpoints for Mail Watcher

* Add steps and functionality for using SES and SNS for email

* Uncomment a few jobs.

* Added option for Firefox for parser. Was having issues with Chromium on Docker.

* Add missing space.

Co-authored-by: Russ Taylor <729694+russtaylor@users.noreply.github.com>

* Fix some wording on the Guide

* update browser extension to handle self-hosted instances

* add slight documentation to options page

* Fix MV

* Do raw handlers for Medium

* Fix images in Medium

* Update self-hosting/GUIDE.md

Co-authored-by: Mike Baker <1426795+mbaker3@users.noreply.github.com>

* Update Guide with other variables

* Add The Verge to JS-less handlers

* Update regex and image-proxy

* Update self-hosting/nginx/nginx.conf

Co-authored-by: Mike Baker <1426795+mbaker3@users.noreply.github.com>

* Update regex and image-proxy

* Update self-hosting/docker-compose/docker-compose.yml

Co-authored-by: Mike Baker <1426795+mbaker3@users.noreply.github.com>

* Fix Minio for Export

* Merge to main

* Update GUIDE with newer NGINX

* Update nginx config to include api/save route

* Enable Native PDF View for PDFS

* Enable Native PDF View for PDFS

* feat:lover packages test

* feat:working build

* feat:alpine build

* docs:api dockerfile docs

* Write a PDF.js wrapper to replace pspdfkit

* Revert changes for replication, set settings to have default mode

* build folder got removed due to gitignore on pdf

* Add Box shadow to pdf pages

* Add Toggle for Progress in PDFS, enabled native viewer toggle

* Update node version to LTS

* Update node version to LTS

* Fix Linting issues

* Fix Linting issues

* Make env variable nullable

* Add touchend listener for mobile

* Make changes to PDF for mobile

* fix(android): change serverUrl to selfhosted first

* feat:2 stage alpine content fetch

* feat:separated start script

* fix:changed to node 22

* Add back youtube functionality and add guide

* trigger build

* Fix cache issue on YouTube

* Allow empty AWS_S3_ENDPOINT

* Allow empty AWS_S3_ENDPOINT

* Add GCHR for all images

* Add GCHR For self hosting.

* Add GCHR For self hosting.

* Test prebuilt.

* Test prebuilt

* Test prebuilt...

* Fix web image

* Remove Web Image (For now)

* Move docker-compose to images

* Move docker-compose files to correct locations

* Remove the need for ARGS

* Update packages, and Typescript versions

* Fix

* Fix issues with build on Web

* Correct push

* Fix Linting issues

* Fix Trace import

* Add missing types

* Fix Tasks

* Add information into guide about self-build

* Fix issues with PDF Viewer

---------

Co-authored-by: keumky2 <keumky2@woowahan.com>
Co-authored-by: William Theaker <wtheaker@nvidia.com>
Co-authored-by: Russ Taylor <729694+russtaylor@users.noreply.github.com>
Co-authored-by: David Adams <david@dadams2.com>
Co-authored-by: Mike Baker <1426795+mbaker3@users.noreply.github.com>
Co-authored-by: m1xxos <66390094+m1xxos@users.noreply.github.com>
Co-authored-by: Adil <mr.adil777@gmail.com>
2025-01-27 13:33:16 +01:00
Hongbo Wu
70d6e9dcae reduce logs 2024-08-29 18:18:17 +08:00
Hongbo Wu
706ab6075d double capacity of content-fetch worker 2024-08-23 12:23:22 +08:00
Hongbo Wu
eec2734f5a keep up to 1 hour complete tasks and up to 1 day failed tasks 2024-08-22 15:27:53 +08:00
Hongbo Wu
f0865654dc fix: allow content-fetch to still process http requests 2024-08-22 15:14:41 +08:00
Hongbo Wu
aa2caca944 do not cache content from https://jacksonh.org 2024-08-21 18:47:23 +08:00
Hongbo Wu
0366c426bc process up to 2 jobs concurrently 2024-08-21 17:58:58 +08:00
Hongbo Wu
6de285432d fix importer status not updated if failed to fetch content 2024-08-21 16:53:50 +08:00
Hongbo Wu
72d89308c5 process up to 10 jobs concurrently 2024-08-21 12:24:35 +08:00
Hongbo Wu
0fbc6d0a87 update importer to use content-fetch queue 2024-08-21 12:24:35 +08:00
Hongbo Wu
5bd272dde0 use one queue with different priority for fetching content of rss feed item or saved url 2024-08-21 12:24:35 +08:00
Hongbo Wu
d29ac109cb slow and rss queue process 5 tasks/s and normal queue process 100 tasks per second 2024-08-21 12:24:35 +08:00
Hongbo Wu
08fbb8aebf use different queues for fast,slow and rss content fetch jobs 2024-08-21 12:24:35 +08:00
Hongbo Wu
87b4ec503e enqueue content-fetch task to the queue 2024-08-21 12:24:35 +08:00
Hongbo Wu
e3eae1c96c create a worker to process content-fetch job 2024-08-21 12:24:35 +08:00
Hongbo Wu
4674321531 reduce blocking domain to 1 hour 2024-08-18 12:37:10 +08:00
Hongbo Wu
322f736fe0 stop storing original html in the database 2024-07-31 19:14:38 +08:00
Hongbo Wu
0e0c4bddac block failed domains 2024-07-24 16:55:50 +08:00
Hongbo Wu
29a5b20d2c remove scrapingbee from content-fetch 2024-07-24 12:17:13 +08:00
Hongbo Wu
c75cbb39d6 injecting webgl fingerprint 2024-07-09 14:11:31 +08:00
Hongbo Wu
dd01202374 do not cache some urls 2024-07-05 19:46:18 +08:00
Hongbo Wu
728059c6f8 do not cache some urls 2024-07-05 19:05:36 +08:00
Hongbo Wu
b38b28c75e create a browser singleton instance and checks browser existence before creating context 2024-07-04 19:12:42 +08:00
Hongbo Wu
59c826fd5e use @omnivore/utils in content-fetch 2024-07-03 21:58:22 +08:00
Hongbo Wu
f2ff4b7b0a fix: only send content_fetch_failure event to analytics 2024-05-31 12:44:01 +08:00
Hongbo Wu
fc9d5c64ec do not fail if cache missed 2024-05-17 17:27:34 +08:00
Hongbo Wu
6f2aa2e0cd add more logs 2024-05-17 17:19:55 +08:00
Hongbo Wu
52ebf466e3 get content from cache first when saving url 2024-05-17 16:46:54 +08:00
Hongbo Wu
9c3d619ad5 put locale and timezone in cache key 2024-05-17 16:22:20 +08:00
Hongbo Wu
dde9f16396 put error message in the analytic event 2024-05-17 16:16:44 +08:00
Hongbo Wu
f3ce6f4d4e catch content fetch result in redis 2024-05-17 15:55:28 +08:00
Hongbo Wu
efb9b6b139 add source to the content_fetch event 2024-05-17 14:54:46 +08:00
Hongbo Wu
9dee510be1 fix rss 2024-05-14 20:18:18 +08:00
Hongbo Wu
cce5f2463d still use redis for cache 2024-05-14 17:16:26 +08:00
Hongbo Wu
04ba62977e fix rebase conflicts 2024-05-14 17:14:41 +08:00
Hongbo Wu
e093c9e096 fix comment 2024-05-14 17:14:41 +08:00
Hongbo Wu
3e925e0193 update comment 2024-05-14 17:14:41 +08:00
Hongbo Wu
5bd157ca25 hash url as the key 2024-05-14 17:14:41 +08:00
Hongbo Wu
7a0b2f3d33 upload file only not exists 2024-05-14 17:14:41 +08:00
Hongbo Wu
9286174ec7 upload and download original content from GCS 2024-05-14 17:14:40 +08:00
Hongbo Wu
33e1c4dd00 remove flush method from analytics class 2024-05-13 19:10:14 +08:00
Hongbo Wu
7634ed667f capture total time of fetching a page 2024-05-13 17:01:52 +08:00
Hongbo Wu
f64bd4700f update analytic event details 2024-05-13 15:18:04 +08:00
Hongbo Wu
a924c8448b capture content-fetch success and error events 2024-05-13 14:55:48 +08:00
Hongbo Wu
0c0a95a79c fix newsletter dir not saved correctly 2024-04-24 21:10:13 +08:00
Hongbo Wu
824b256d20 fix memory leak from axios error 2024-04-24 15:55:54 +08:00
Hongbo Wu
7f441b4ff3 dedupe save-page job 2024-04-23 21:44:25 +08:00
Hongbo Wu
927394e07c fix: save url operation is delayed 2024-02-21 17:54:08 +08:00
Jackson Harper
856a7bc5ef Add health checks to content-fetch 2024-02-12 14:42:51 +08:00
Hongbo Wu
86f4553dd1 fix: duplicate key value violates unique constraint "library_item_pkey" 2024-02-07 20:41:34 +08:00