58 commits

Author	SHA1	Message	Date
Jackson Harper	629c044273	Queue the transcript processing as a separate job ... Handle YouTube in two steps, first get metadata then get the transcript.	2024-03-14 15:09:53 +08:00
Jackson Harper	2dbd16a61e	Pull duration and description from YouTube metadata	2024-03-14 12:48:56 +08:00
Jackson Harper	f7225b298a	Rebase	2024-03-14 12:48:56 +08:00
Hongbo Wu	d41459a5cd	log unhandled error	2024-03-06 12:04:16 +08:00
Jackson Harper	bbe6204b91	summaries	2024-02-28 15:12:09 +08:00
Hongbo Wu	fb576a58a0	initial export	2024-02-22 15:20:24 +08:00
Hongbo Wu	f94267ee1a	enqueue export job	2024-02-22 13:38:51 +08:00
Hongbo Wu	ce19218e15	get queue priority from a table	2024-02-21 22:01:04 +08:00
Hongbo Wu	cd311d6454	do not update db options in queue-processor	2024-02-20 19:09:33 +08:00
Hongbo Wu	59d4d78f78	connection idle timeout in 10 seconds	2024-02-20 18:33:21 +08:00
Hongbo Wu	485caec803	append job version to the job id	2024-02-07 18:42:10 +08:00
Hongbo Wu	8c23cff17b	remove limit on the sync reading progress job	2024-02-07 11:09:30 +08:00
Hongbo Wu	95fabe1dbf	keep completed jobs for up to 24 hours and failed jobs for 7 days	2024-02-06 14:21:00 +08:00
Hongbo Wu	53f63b90d7	increase the seed valud of backoff delay to 2 seconds	2024-02-05 16:26:05 +08:00
Hongbo Wu	25e374f6ff	fix: replace webhook endpoint with a bullmq job	2024-02-02 16:18:40 +08:00
Hongbo Wu	5e49131fc2	Merge pull request #3470 from omnivore-app/fix/async-bulk-action ... async bulk action on item count > 100 and create batch jobs	2024-02-02 16:00:12 +08:00
Hongbo Wu	26b7103f0b	improve batch update	2024-02-02 13:13:18 +08:00
Hongbo Wu	1bccf33320	add job_id and parent for tracking	2024-02-02 10:50:07 +08:00
Jackson Harper	ed7ac6c1b4	Fix typo in lifecycle endpoint path	2024-02-01 20:25:52 +08:00
Hongbo Wu	702bd6c3c6	async bulk action on item count > 100 and create batch jobs	2024-02-01 17:44:30 +08:00
Jackson Harper	0e9e331e18	Implement a lifecycle hook for queue processor so it can shut down gracefully	2024-02-01 16:43:57 +08:00
Jackson Harper	1b44cb860b	Function for cron jobs dont include queue in the read position metrics since its not part of that queue	2024-02-01 10:33:51 +08:00
Jackson Harper	7d6645b87a	Rebase	2024-02-01 10:02:38 +08:00
Jackson Harper	570b7a4b78	Use a set in redis and push items to the set ... This lets us grab all available items, apply them, and then remove them from the set. Once completed the items that were applied can be removed from the set, if any new items were added during that time they wont be effected.	2024-02-01 09:59:42 +08:00
Jackson Harper	d8ea4ff1c5	Add reading position messages to exported metrics for queue processor	2024-02-01 09:59:41 +08:00
Jackson Harper	fbfa934479	Write behind reading progress as a list that can be reduced at commit or read time	2024-02-01 09:59:39 +08:00
Hongbo Wu	5e883cb2ba	running worker in the test	2024-01-31 17:45:49 +08:00
Hongbo Wu	5646413b59	add a job for updating highlight annotations in library_item	2024-01-31 14:15:33 +08:00
Hongbo Wu	5ed35bbeff	create a job for update labels in library item	2024-01-31 13:32:37 +08:00
Hongbo Wu	ddddf8234e	create a job for db trigger	2024-01-30 23:53:05 +08:00
Hongbo Wu	e03895c8b7	enqueue trigger rule job once page is created or updated	2024-01-26 17:46:17 +08:00
Hongbo Wu	be0a2e8c52	use different redis server for mq and cache	2024-01-26 13:01:53 +08:00
Hongbo Wu	e54c691c09	create a job for finding thumbnail	2024-01-26 13:01:51 +08:00
Jackson Harper	a639a2fa10	Add types to exported metrics	2024-01-25 12:37:02 +08:00
Jackson Harper	5f9be385b7	Remove unneeded escapes	2024-01-24 21:34:51 +08:00
Jackson Harper	c581372a93	Improve exported metrics naming convention	2024-01-24 21:13:13 +08:00
Jackson Harper	d24db92e03	Dont force init this queue	2024-01-24 12:40:00 +08:00
Jackson Harper	c29f31d618	Add _count to prom metrics names	2024-01-24 12:39:04 +08:00
Jackson Harper	e4109a63f0	Expose custom metrics for the queue processor	2024-01-24 12:15:38 +08:00
Jackson Harper	e407db914a	Add new job for handling upload content	2024-01-22 15:36:46 +08:00
Jackson Harper	db8284dc0b	Wait until backend queue is ready before putting jobs into it	2024-01-22 11:31:46 +08:00
Jackson Harper	f7b17cb93f	Only create queue object once in queue processor	2024-01-22 11:27:13 +08:00
Hongbo Wu	780cda6e4b	use worker redis client when connect to the queue in queue-processor	2024-01-19 20:44:54 +08:00
Hongbo Wu	338733829f	Merge pull request #3394 from omnivore-app/fix/cache-content-fetched ... fix/cache content fetched	2024-01-19 18:16:05 +08:00
Hongbo Wu	4340286b89	add retry on failure in jobs	2024-01-19 17:38:25 +08:00
Jackson Harper	8ba1242ffb	Back off refresh-all-feeds if queue is very deep	2024-01-19 16:59:41 +08:00
Jackson Harper	2dbff6a6ab	Add some more debug output	2024-01-19 16:12:44 +08:00
Hongbo Wu	a6f0e2f2d9	add save page job processor	2024-01-18 18:48:46 +08:00
Jackson Harper	fd81f65e53	Remove server.close call	2024-01-17 16:37:51 +08:00
Jackson Harper	361d57ac1d	Remove secrets loading as this is handled by k8s now. Fix redis init	2024-01-17 13:33:20 +08:00