9266 commits

This branch

This branch

All branches

Author	SHA1	Message	Date
Jackson Harper	e9f22bba43	Some debug	2024-01-20 09:09:25 +08:00
Hongbo Wu	fb1d8331a4	Merge pull request #3396 from omnivore-app/fix/queue-redis-connection ... use worker redis client when connect to the queue in queue-processor	2024-01-19 21:03:20 +08:00
Hongbo Wu	780cda6e4b	use worker redis client when connect to the queue in queue-processor	2024-01-19 20:44:54 +08:00
Hongbo Wu	338733829f	Merge pull request #3394 from omnivore-app/fix/cache-content-fetched ... fix/cache content fetched	2024-01-19 18:16:05 +08:00
Hongbo Wu	6b89f4b809	graceful shutdown redis	2024-01-19 18:06:12 +08:00
Hongbo Wu	356ad38678	reduce logs	2024-01-19 18:01:09 +08:00
Hongbo Wu	4340286b89	add retry on failure in jobs	2024-01-19 17:38:25 +08:00
Jackson Harper	b8d08ae33f	Merge pull request #3393 from omnivore-app/fix/job-priority ... Fix refresh feed job priority	2024-01-19 17:28:09 +08:00
Hongbo Wu	8acdea5002	set cache if not exists	2024-01-19 17:14:49 +08:00
Jackson Harper	defb79d8c8	Use let instead of var	2024-01-19 17:12:34 +08:00
Hongbo Wu	51d5166321	get cached content from redis when saving pages	2024-01-19 17:06:37 +08:00
Jackson Harper	8ba1242ffb	Back off refresh-all-feeds if queue is very deep	2024-01-19 16:59:41 +08:00
Jackson Harper	d2491d3eb7	Reduce refresh-all-feeds priority	2024-01-19 16:59:15 +08:00
Hongbo Wu	699d38609b	cache fetch result in redis	2024-01-19 16:58:24 +08:00
Jackson Harper	e06ff24cea	Fix logging	2024-01-19 16:44:20 +08:00
Jackson Harper	02818d94a2	Adjust priorities	2024-01-19 16:37:57 +08:00
Hongbo Wu	35805e6e8f	use user provided url for feed added	2024-01-19 16:28:37 +08:00
Jackson Harper	84375b7c36	Add validation before refreshing feed URLs	2024-01-19 16:25:05 +08:00
Jackson Harper	2dbff6a6ab	Add some more debug output	2024-01-19 16:12:44 +08:00
Jackson Harper	0e590e53a6	Improve RSS priorities for refreshing feeds	2024-01-19 16:12:22 +08:00
Hongbo Wu	d135a7fe6a	Merge pull request #3391 from omnivore-app/fix/content-fetch-dockerfile ... fix: build content-fetch in dockerfile	2024-01-19 12:45:19 +08:00
Jackson Harper	69541d3e91	Merge pull request #3392 from omnivore-app/fix/backend-svc-endpoint ... Fix construction of service endpoint	2024-01-19 12:40:30 +08:00
Hongbo Wu	3d42973c5f	fix api endpint	2024-01-19 12:36:15 +08:00
Jackson Harper	f03766d05b	Fix construction of service endpoint ... NEed to fix how we construct all these	2024-01-19 12:29:50 +08:00
Hongbo Wu	7c25a628d9	fix pem format	2024-01-19 11:44:16 +08:00
Hongbo Wu	2e8f173879	create queue on start up	2024-01-19 11:26:53 +08:00
Hongbo Wu	a6d7623cea	do not close redis connection	2024-01-19 10:39:10 +08:00
Hongbo Wu	846d1e3381	fix: build content-fetch in dockerfile	2024-01-19 10:13:05 +08:00
Hongbo Wu	57aca54538	Merge pull request #3366 from omnivore-app/feature/separate-fetch-content ... feature/separate fetch content	2024-01-18 23:30:26 +08:00
Hongbo Wu	c2a084803a	validate feed url before adding	2024-01-18 21:56:19 +08:00
Hongbo Wu	d2c17f7539	graceful shutdown redis	2024-01-18 21:52:33 +08:00
Hongbo Wu	3e291d1019	fix tests	2024-01-18 21:13:32 +08:00
Hongbo Wu	d4f49dceb6	remove job id	2024-01-18 18:48:46 +08:00
Hongbo Wu	a6f0e2f2d9	add save page job processor	2024-01-18 18:48:46 +08:00
Hongbo Wu	43df5d30a8	prioritized jobs	2024-01-18 18:48:46 +08:00
Hongbo Wu	0015a946b2	create batch save page jobs after fetching content	2024-01-18 18:48:46 +08:00
Hongbo Wu	8a3b50f07f	do not create thumbnail task if the item already has a thumbnail	2024-01-18 18:48:46 +08:00
Hongbo Wu	5136874372	remove debug logs	2024-01-18 18:48:46 +08:00
Hongbo Wu	a03eee5ef7	fix dependecies	2024-01-18 18:48:46 +08:00
Hongbo Wu	d9feb740cb	convert content-fetch to typescript	2024-01-18 18:48:46 +08:00
Hongbo Wu	cd3402b98a	rewrite puppeteer in typescript	2024-01-18 18:48:46 +08:00
Hongbo Wu	51e586ed3d	separate content-fetch in puppeteer packages from saving page content	2024-01-18 18:48:46 +08:00
Hongbo Wu	c297a8dd4e	feat: group fetch content tasks by url	2024-01-18 18:48:46 +08:00
Hongbo Wu	8a91a3c629	fix importer tests	2024-01-18 18:48:46 +08:00
Jackson Harper	3f163a99c7	Merge pull request #3389 from omnivore-app/fix/refresh-feed-api-url ... Fix endpoint for internal refresh feed API call	2024-01-18 17:13:05 +08:00
Jackson Harper	3f707c06f2	Fix checks for env vars	2024-01-18 16:59:53 +08:00
Jackson Harper	3b0338967d	Fix endpoint for internal refresh feed API call ... This should eventually be replaced.	2024-01-18 16:48:45 +08:00
Jackson Harper	bffe31bcc7	Merge pull request #3359 from omnivore-app/feat/backend-bull-job-processor ... Use bullmq to process jobs	2024-01-18 15:38:30 +08:00
Jackson Harper	6193c78288	Fix typo	2024-01-18 15:37:38 +08:00
Jackson Harper	7754139207	Remove debug	2024-01-18 15:33:51 +08:00