181 commits

Author	SHA1	Message	Date
Thomas Rogers	89e78b377d	fix(): Add network timeout to all yarn installs ... Bump image-size from 1.2.0 to 1.2.1 (#4567) Bumps [image-size](https://github.com/image-size/image-size) from 1.2.0 to 1.2.1. - [Release notes](https://github.com/image-size/image-size/releases) - [Commits](https://github.com/image-size/image-size/compare/v1.2.0...v1.2.1) --- updated-dependencies: - dependency-name: image-size dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> fix(): Add network timeout to all yarn installs fix(): remove node-buffer Bump brace-expansion from 1.1.11 to 1.1.12 in /pkg/admin (#4593) Bumps [brace-expansion](https://github.com/juliangruber/brace-expansion) from 1.1.11 to 1.1.12. - [Release notes](https://github.com/juliangruber/brace-expansion/releases) - [Commits](https://github.com/juliangruber/brace-expansion/compare/1.1.11...v1.1.12) --- updated-dependencies: - dependency-name: brace-expansion dependency-version: 1.1.12 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Bump @babel/runtime-corejs3 from 7.26.0 to 7.28.2 (#4592) Bumps [@babel/runtime-corejs3](https://github.com/babel/babel/tree/HEAD/packages/babel-runtime-corejs3) from 7.26.0 to 7.28.2. - [Release notes](https://github.com/babel/babel/releases) - [Changelog](https://github.com/babel/babel/blob/main/CHANGELOG.md) - [Commits](https://github.com/babel/babel/commits/v7.28.2/packages/babel-runtime-corejs3) --- updated-dependencies: - dependency-name: "@babel/runtime-corejs3" dependency-version: 7.28.2 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Bump next from 13.5.8 to 14.2.30 (#4591) Bumps [next](https://github.com/vercel/next.js) from 13.5.8 to 14.2.30. - [Release notes](https://github.com/vercel/next.js/releases) - [Changelog](https://github.com/vercel/next.js/blob/canary/release.js) - [Commits](https://github.com/vercel/next.js/compare/v13.5.8...v14.2.30) --- updated-dependencies: - dependency-name: next dependency-version: 14.2.30 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Bump axios from 0.27.2 to 0.30.0 (#4590) Bumps [axios](https://github.com/axios/axios) from 0.27.2 to 0.30.0. - [Release notes](https://github.com/axios/axios/releases) - [Changelog](https://github.com/axios/axios/blob/v0.30.0/CHANGELOG.md) - [Commits](https://github.com/axios/axios/compare/v0.27.2...v0.30.0) --- updated-dependencies: - dependency-name: axios dependency-version: 0.30.0 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Bump @babel/runtime from 7.14.6 to 7.28.2 in /pkg/admin (#4589) Bumps [@babel/runtime](https://github.com/babel/babel/tree/HEAD/packages/babel-runtime) from 7.14.6 to 7.28.2. - [Release notes](https://github.com/babel/babel/releases) - [Changelog](https://github.com/babel/babel/blob/main/CHANGELOG.md) - [Commits](https://github.com/babel/babel/commits/v7.28.2/packages/babel-runtime) --- updated-dependencies: - dependency-name: "@babel/runtime" dependency-version: 7.28.2 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Bump @radix-ui/react-separator from 0.1.4 to 1.1.6 (#4579) Bumps [@radix-ui/react-separator](https://github.com/radix-ui/primitives) from 0.1.4 to 1.1.6. - [Changelog](https://github.com/radix-ui/primitives/blob/main/release-process.md) - [Commits](https://github.com/radix-ui/primitives/commits) --- updated-dependencies: - dependency-name: "@radix-ui/react-separator" dependency-version: 1.1.6 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Bump http-proxy-middleware from 2.0.7 to 2.0.9 (#4574) Bumps [http-proxy-middleware](https://github.com/chimurai/http-proxy-middleware) from 2.0.7 to 2.0.9. - [Release notes](https://github.com/chimurai/http-proxy-middleware/releases) - [Changelog](https://github.com/chimurai/http-proxy-middleware/blob/v2.0.9/CHANGELOG.md) - [Commits](https://github.com/chimurai/http-proxy-middleware/compare/v2.0.7...v2.0.9) --- updated-dependencies: - dependency-name: http-proxy-middleware dependency-version: 2.0.9 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Bump @babel/helpers from 7.14.6 to 7.28.2 in /pkg/admin (#4594) Bumps [@babel/helpers](https://github.com/babel/babel/tree/HEAD/packages/babel-helpers) from 7.14.6 to 7.28.2. - [Release notes](https://github.com/babel/babel/releases) - [Changelog](https://github.com/babel/babel/blob/main/CHANGELOG.md) - [Commits](https://github.com/babel/babel/commits/v7.28.2/packages/babel-helpers) --- updated-dependencies: - dependency-name: "@babel/helpers" dependency-version: 7.28.2 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Bump pg and @types/pg (#4547) Bumps [pg](https://github.com/brianc/node-postgres/tree/HEAD/packages/pg) and [@types/pg](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/pg). These dependencies needed to be updated together. Updates `pg` from 8.13.1 to 8.13.3 - [Changelog](https://github.com/brianc/node-postgres/blob/master/CHANGELOG.md) - [Commits](https://github.com/brianc/node-postgres/commits/pg@8.13.3/packages/pg) Updates `@types/pg` from 8.11.10 to 8.11.11 - [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases) - [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/pg) --- updated-dependencies: - dependency-name: pg dependency-type: direct:production update-type: version-update:semver-patch - dependency-name: "@types/pg" dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Bump @graphql-tools/utils from 9.2.1 to 10.8.6 (#4560) Bumps [@graphql-tools/utils](https://github.com/ardatan/graphql-tools/tree/HEAD/packages/utils) from 9.2.1 to 10.8.6. - [Release notes](https://github.com/ardatan/graphql-tools/releases) - [Changelog](https://github.com/ardatan/graphql-tools/blob/master/packages/utils/CHANGELOG.md) - [Commits](https://github.com/ardatan/graphql-tools/commits/@graphql-tools/utils@10.8.6/packages/utils) --- updated-dependencies: - dependency-name: "@graphql-tools/utils" dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> chore(): update versions chore(): remove unused workload fix(): kbar dependency fix(): kbar dependency chore(): fix packages fix(): dependencies fix(): dependencies fix(): dependencies fix(): dependencies api fix(): dependencies api fix(): dependencies api chore(): fix order of github fix(): fix tsconfig for liqe	2025-07-29 19:01:37 +02:00
Tom Rogers	4e582fb55d	Improving Self-Hosting and Removing 3rd Party dependencies. (#4513) ... * fix: Library Header layout shift * Bump Github Actions versions. * Self-Hosting Changes * Fix Minio Environment Variable * Just make pdfs successful, due to lack of PDFHandler * Fix issue where flag was set wrong * Added an NGINX Example file * Add some documentation for self-hosting via Docker Compose * Make some adjustments to Puppeteer due to failing sites. * adjust timings * Add start of Mail Service * Fix Docker Files * More email service stuff * Add Guide to use Zapier for Email-Importing. * Ensure that if no env is provided it uses the old email settings * Add some instructions for self-hosted email * Add SNS Endpoints for Mail Watcher * Add steps and functionality for using SES and SNS for email * Uncomment a few jobs. * Added option for Firefox for parser. Was having issues with Chromium on Docker. * Add missing space. Co-authored-by: Russ Taylor <729694+russtaylor@users.noreply.github.com> * Fix some wording on the Guide * update browser extension to handle self-hosted instances * add slight documentation to options page * Fix MV * Do raw handlers for Medium * Fix images in Medium * Update self-hosting/GUIDE.md Co-authored-by: Mike Baker <1426795+mbaker3@users.noreply.github.com> * Update Guide with other variables * Add The Verge to JS-less handlers * Update regex and image-proxy * Update self-hosting/nginx/nginx.conf Co-authored-by: Mike Baker <1426795+mbaker3@users.noreply.github.com> * Update regex and image-proxy * Update self-hosting/docker-compose/docker-compose.yml Co-authored-by: Mike Baker <1426795+mbaker3@users.noreply.github.com> * Fix Minio for Export * Merge to main * Update GUIDE with newer NGINX * Update nginx config to include api/save route * Enable Native PDF View for PDFS * Enable Native PDF View for PDFS * feat:lover packages test * feat:working build * feat:alpine build * docs:api dockerfile docs * Write a PDF.js wrapper to replace pspdfkit * Revert changes for replication, set settings to have default mode * build folder got removed due to gitignore on pdf * Add Box shadow to pdf pages * Add Toggle for Progress in PDFS, enabled native viewer toggle * Update node version to LTS * Update node version to LTS * Fix Linting issues * Fix Linting issues * Make env variable nullable * Add touchend listener for mobile * Make changes to PDF for mobile * fix(android): change serverUrl to selfhosted first * feat:2 stage alpine content fetch * feat:separated start script * fix:changed to node 22 * Add back youtube functionality and add guide * trigger build * Fix cache issue on YouTube * Allow empty AWS_S3_ENDPOINT * Allow empty AWS_S3_ENDPOINT * Add GCHR for all images * Add GCHR For self hosting. * Add GCHR For self hosting. * Test prebuilt. * Test prebuilt * Test prebuilt... * Fix web image * Remove Web Image (For now) * Move docker-compose to images * Move docker-compose files to correct locations * Remove the need for ARGS * Update packages, and Typescript versions * Fix * Fix issues with build on Web * Correct push * Fix Linting issues * Fix Trace import * Add missing types * Fix Tasks * Add information into guide about self-build * Fix issues with PDF Viewer --------- Co-authored-by: keumky2 <keumky2@woowahan.com> Co-authored-by: William Theaker <wtheaker@nvidia.com> Co-authored-by: Russ Taylor <729694+russtaylor@users.noreply.github.com> Co-authored-by: David Adams <david@dadams2.com> Co-authored-by: Mike Baker <1426795+mbaker3@users.noreply.github.com> Co-authored-by: m1xxos <66390094+m1xxos@users.noreply.github.com> Co-authored-by: Adil <mr.adil777@gmail.com>	2025-01-27 13:33:16 +01:00
Hongbo Wu	70d6e9dcae	reduce logs	2024-08-29 18:18:17 +08:00
Hongbo Wu	706ab6075d	double capacity of content-fetch worker	2024-08-23 12:23:22 +08:00
Hongbo Wu	eec2734f5a	keep up to 1 hour complete tasks and up to 1 day failed tasks	2024-08-22 15:27:53 +08:00
Hongbo Wu	f0865654dc	fix: allow content-fetch to still process http requests	2024-08-22 15:14:41 +08:00
Hongbo Wu	aa2caca944	do not cache content from https://jacksonh.org	2024-08-21 18:47:23 +08:00
Hongbo Wu	0366c426bc	process up to 2 jobs concurrently	2024-08-21 17:58:58 +08:00
Hongbo Wu	6de285432d	fix importer status not updated if failed to fetch content	2024-08-21 16:53:50 +08:00
Hongbo Wu	72d89308c5	process up to 10 jobs concurrently	2024-08-21 12:24:35 +08:00
Hongbo Wu	0fbc6d0a87	update importer to use content-fetch queue	2024-08-21 12:24:35 +08:00
Hongbo Wu	5bd272dde0	use one queue with different priority for fetching content of rss feed item or saved url	2024-08-21 12:24:35 +08:00
Hongbo Wu	d29ac109cb	slow and rss queue process 5 tasks/s and normal queue process 100 tasks per second	2024-08-21 12:24:35 +08:00
Hongbo Wu	34edbeba56	fix dockerfile	2024-08-21 12:24:35 +08:00
Hongbo Wu	08fbb8aebf	use different queues for fast,slow and rss content fetch jobs	2024-08-21 12:24:35 +08:00
Hongbo Wu	87b4ec503e	enqueue content-fetch task to the queue	2024-08-21 12:24:35 +08:00
Hongbo Wu	e3eae1c96c	create a worker to process content-fetch job	2024-08-21 12:24:35 +08:00
Hongbo Wu	4674321531	reduce blocking domain to 1 hour	2024-08-18 12:37:10 +08:00
Hongbo Wu	322f736fe0	stop storing original html in the database	2024-07-31 19:14:38 +08:00
Hongbo Wu	0e0c4bddac	block failed domains	2024-07-24 16:55:50 +08:00
Hongbo Wu	31fe4b65a0	remove readability from content-fetch	2024-07-24 12:53:41 +08:00
Hongbo Wu	29a5b20d2c	remove scrapingbee from content-fetch	2024-07-24 12:17:13 +08:00
Hongbo Wu	75338f5927	bypass cloudflare captcha	2024-07-10 14:43:47 +08:00
Hongbo Wu	73e180f43d	add more dependencies to docker container	2024-07-09 19:16:21 +08:00
Hongbo Wu	c75cbb39d6	injecting webgl fingerprint	2024-07-09 14:11:31 +08:00
Hongbo Wu	dd01202374	do not cache some urls	2024-07-05 19:46:18 +08:00
Hongbo Wu	728059c6f8	do not cache some urls	2024-07-05 19:05:36 +08:00
Hongbo Wu	b38b28c75e	create a browser singleton instance and checks browser existence before creating context	2024-07-04 19:12:42 +08:00
Hongbo Wu	bbc7b5e600	use @omnivore/utils in import-handler	2024-07-03 22:20:27 +08:00
Hongbo Wu	59c826fd5e	use @omnivore/utils in content-fetch	2024-07-03 21:58:22 +08:00
Hongbo Wu	f2ff4b7b0a	fix: only send content_fetch_failure event to analytics	2024-05-31 12:44:01 +08:00
Hongbo Wu	fc9d5c64ec	do not fail if cache missed	2024-05-17 17:27:34 +08:00
Hongbo Wu	6f2aa2e0cd	add more logs	2024-05-17 17:19:55 +08:00
Hongbo Wu	52ebf466e3	get content from cache first when saving url	2024-05-17 16:46:54 +08:00
Hongbo Wu	9c3d619ad5	put locale and timezone in cache key	2024-05-17 16:22:20 +08:00
Hongbo Wu	dde9f16396	put error message in the analytic event	2024-05-17 16:16:44 +08:00
Hongbo Wu	f3ce6f4d4e	catch content fetch result in redis	2024-05-17 15:55:28 +08:00
Hongbo Wu	efb9b6b139	add source to the content_fetch event	2024-05-17 14:54:46 +08:00
Hongbo Wu	9dee510be1	fix rss	2024-05-14 20:18:18 +08:00
Hongbo Wu	cce5f2463d	still use redis for cache	2024-05-14 17:16:26 +08:00
Hongbo Wu	04ba62977e	fix rebase conflicts	2024-05-14 17:14:41 +08:00
Hongbo Wu	e093c9e096	fix comment	2024-05-14 17:14:41 +08:00
Hongbo Wu	3e925e0193	update comment	2024-05-14 17:14:41 +08:00
Hongbo Wu	5bd157ca25	hash url as the key	2024-05-14 17:14:41 +08:00
Hongbo Wu	7a0b2f3d33	upload file only not exists	2024-05-14 17:14:41 +08:00
Hongbo Wu	9286174ec7	upload and download original content from GCS	2024-05-14 17:14:40 +08:00
Hongbo Wu	33e1c4dd00	remove flush method from analytics class	2024-05-13 19:10:14 +08:00
Hongbo Wu	7634ed667f	capture total time of fetching a page	2024-05-13 17:01:52 +08:00
Hongbo Wu	f64bd4700f	update analytic event details	2024-05-13 15:18:04 +08:00
Hongbo Wu	a924c8448b	capture content-fetch success and error events	2024-05-13 14:55:48 +08:00