70 commits

Author	SHA1	Message	Date
Hongbo Wu	de258ae0bf	timeout if job takes more than 10 minutes	2024-04-23 15:50:15 +08:00
Hongbo Wu	71477b1712	implement create digest job	2024-04-16 21:00:51 +08:00
Hongbo Wu	03e497b489	use a separate queue for digest job	2024-04-16 21:00:26 +08:00
Hongbo Wu	12aa7d4447	create digest job api	2024-04-16 21:00:10 +08:00
Hongbo Wu	b4911639c7	stub create digest job	2024-04-16 21:00:09 +08:00
Hongbo Wu	f66d3fa36c	migrate inbound-email tasks from pubsub to bullmq	2024-04-11 11:08:22 +08:00
Hongbo Wu	5f239d2dcb	update graceful shutdown	2024-04-11 11:08:22 +08:00
Hongbo Wu	8af2bb076c	rename send mail functions	2024-04-05 16:59:20 +08:00
Hongbo Wu	10dcb922f2	Enqueue confirmation email	2024-04-05 16:57:02 +08:00
Jackson Harper	b5bea17ffe	Export the age of the oldest job in the queue	2024-03-28 10:25:44 +08:00
Hongbo Wu	92abfed2df	replace logger.warn with warning	2024-03-27 15:41:13 +08:00
Hongbo Wu	390d12eea1	add export to integration and get task api	2024-03-21 11:14:23 +08:00
Jackson Harper	629c044273	Queue the transcript processing as a separate job ... Handle YouTube in two steps, first get metadata then get the transcript.	2024-03-14 15:09:53 +08:00
Jackson Harper	2dbd16a61e	Pull duration and description from YouTube metadata	2024-03-14 12:48:56 +08:00
Jackson Harper	f7225b298a	Rebase	2024-03-14 12:48:56 +08:00
Hongbo Wu	d41459a5cd	log unhandled error	2024-03-06 12:04:16 +08:00
Jackson Harper	bbe6204b91	summaries	2024-02-28 15:12:09 +08:00
Hongbo Wu	fb576a58a0	initial export	2024-02-22 15:20:24 +08:00
Hongbo Wu	f94267ee1a	enqueue export job	2024-02-22 13:38:51 +08:00
Hongbo Wu	ce19218e15	get queue priority from a table	2024-02-21 22:01:04 +08:00
Hongbo Wu	cd311d6454	do not update db options in queue-processor	2024-02-20 19:09:33 +08:00
Hongbo Wu	59d4d78f78	connection idle timeout in 10 seconds	2024-02-20 18:33:21 +08:00
Hongbo Wu	485caec803	append job version to the job id	2024-02-07 18:42:10 +08:00
Hongbo Wu	8c23cff17b	remove limit on the sync reading progress job	2024-02-07 11:09:30 +08:00
Hongbo Wu	95fabe1dbf	keep completed jobs for up to 24 hours and failed jobs for 7 days	2024-02-06 14:21:00 +08:00
Hongbo Wu	53f63b90d7	increase the seed valud of backoff delay to 2 seconds	2024-02-05 16:26:05 +08:00
Hongbo Wu	25e374f6ff	fix: replace webhook endpoint with a bullmq job	2024-02-02 16:18:40 +08:00
Hongbo Wu	5e49131fc2	Merge pull request #3470 from omnivore-app/fix/async-bulk-action ... async bulk action on item count > 100 and create batch jobs	2024-02-02 16:00:12 +08:00
Hongbo Wu	26b7103f0b	improve batch update	2024-02-02 13:13:18 +08:00
Hongbo Wu	1bccf33320	add job_id and parent for tracking	2024-02-02 10:50:07 +08:00
Jackson Harper	ed7ac6c1b4	Fix typo in lifecycle endpoint path	2024-02-01 20:25:52 +08:00
Hongbo Wu	702bd6c3c6	async bulk action on item count > 100 and create batch jobs	2024-02-01 17:44:30 +08:00
Jackson Harper	0e9e331e18	Implement a lifecycle hook for queue processor so it can shut down gracefully	2024-02-01 16:43:57 +08:00
Jackson Harper	1b44cb860b	Function for cron jobs dont include queue in the read position metrics since its not part of that queue	2024-02-01 10:33:51 +08:00
Jackson Harper	7d6645b87a	Rebase	2024-02-01 10:02:38 +08:00
Jackson Harper	570b7a4b78	Use a set in redis and push items to the set ... This lets us grab all available items, apply them, and then remove them from the set. Once completed the items that were applied can be removed from the set, if any new items were added during that time they wont be effected.	2024-02-01 09:59:42 +08:00
Jackson Harper	d8ea4ff1c5	Add reading position messages to exported metrics for queue processor	2024-02-01 09:59:41 +08:00
Jackson Harper	fbfa934479	Write behind reading progress as a list that can be reduced at commit or read time	2024-02-01 09:59:39 +08:00
Hongbo Wu	5e883cb2ba	running worker in the test	2024-01-31 17:45:49 +08:00
Hongbo Wu	5646413b59	add a job for updating highlight annotations in library_item	2024-01-31 14:15:33 +08:00
Hongbo Wu	5ed35bbeff	create a job for update labels in library item	2024-01-31 13:32:37 +08:00
Hongbo Wu	ddddf8234e	create a job for db trigger	2024-01-30 23:53:05 +08:00
Hongbo Wu	e03895c8b7	enqueue trigger rule job once page is created or updated	2024-01-26 17:46:17 +08:00
Hongbo Wu	be0a2e8c52	use different redis server for mq and cache	2024-01-26 13:01:53 +08:00
Hongbo Wu	e54c691c09	create a job for finding thumbnail	2024-01-26 13:01:51 +08:00
Jackson Harper	a639a2fa10	Add types to exported metrics	2024-01-25 12:37:02 +08:00
Jackson Harper	5f9be385b7	Remove unneeded escapes	2024-01-24 21:34:51 +08:00
Jackson Harper	c581372a93	Improve exported metrics naming convention	2024-01-24 21:13:13 +08:00
Jackson Harper	d24db92e03	Dont force init this queue	2024-01-24 12:40:00 +08:00
Jackson Harper	c29f31d618	Add _count to prom metrics names	2024-01-24 12:39:04 +08:00