80 commits

Author	SHA1	Message	Date
Hongbo Wu	57695c370c	monitor each job latency in prometheus	2024-06-12 13:35:04 +08:00
Hongbo Wu	f052ab55a0	monitor home feed job latency in prometheus	2024-06-12 13:01:36 +08:00
Hongbo Wu	76383d8a33	feat: generate preview content if excerpt less than 180 chars	2024-06-06 19:08:41 +08:00
Hongbo Wu	efb7160373	rename just read feed to home	2024-05-27 21:11:08 +08:00
Hongbo Wu	41b9fef316	score library item after saved	2024-05-27 11:45:51 +08:00
Hongbo Wu	a48a614676	save just read feed in redis sorted set	2024-05-24 14:52:21 +08:00
Hongbo Wu	01ebcbb16b	add bulk upload original content job	2024-05-10 14:37:05 +08:00
Hongbo Wu	14e91d338d	do not create user profile and topics	2024-04-30 12:48:19 +08:00
Hongbo Wu	2f7c18b363	add more logs	2024-04-23 15:50:15 +08:00
Hongbo Wu	1b6f4976d3	remove timeout	2024-04-23 15:50:15 +08:00
Hongbo Wu	de258ae0bf	timeout if job takes more than 10 minutes	2024-04-23 15:50:15 +08:00
Hongbo Wu	71477b1712	implement create digest job	2024-04-16 21:00:51 +08:00
Hongbo Wu	03e497b489	use a separate queue for digest job	2024-04-16 21:00:26 +08:00
Hongbo Wu	12aa7d4447	create digest job api	2024-04-16 21:00:10 +08:00
Hongbo Wu	b4911639c7	stub create digest job	2024-04-16 21:00:09 +08:00
Hongbo Wu	f66d3fa36c	migrate inbound-email tasks from pubsub to bullmq	2024-04-11 11:08:22 +08:00
Hongbo Wu	5f239d2dcb	update graceful shutdown	2024-04-11 11:08:22 +08:00
Hongbo Wu	8af2bb076c	rename send mail functions	2024-04-05 16:59:20 +08:00
Hongbo Wu	10dcb922f2	Enqueue confirmation email	2024-04-05 16:57:02 +08:00
Jackson Harper	b5bea17ffe	Export the age of the oldest job in the queue	2024-03-28 10:25:44 +08:00
Hongbo Wu	92abfed2df	replace logger.warn with warning	2024-03-27 15:41:13 +08:00
Hongbo Wu	390d12eea1	add export to integration and get task api	2024-03-21 11:14:23 +08:00
Jackson Harper	629c044273	Queue the transcript processing as a separate job ... Handle YouTube in two steps, first get metadata then get the transcript.	2024-03-14 15:09:53 +08:00
Jackson Harper	2dbd16a61e	Pull duration and description from YouTube metadata	2024-03-14 12:48:56 +08:00
Jackson Harper	f7225b298a	Rebase	2024-03-14 12:48:56 +08:00
Hongbo Wu	d41459a5cd	log unhandled error	2024-03-06 12:04:16 +08:00
Jackson Harper	bbe6204b91	summaries	2024-02-28 15:12:09 +08:00
Hongbo Wu	fb576a58a0	initial export	2024-02-22 15:20:24 +08:00
Hongbo Wu	f94267ee1a	enqueue export job	2024-02-22 13:38:51 +08:00
Hongbo Wu	ce19218e15	get queue priority from a table	2024-02-21 22:01:04 +08:00
Hongbo Wu	cd311d6454	do not update db options in queue-processor	2024-02-20 19:09:33 +08:00
Hongbo Wu	59d4d78f78	connection idle timeout in 10 seconds	2024-02-20 18:33:21 +08:00
Hongbo Wu	485caec803	append job version to the job id	2024-02-07 18:42:10 +08:00
Hongbo Wu	8c23cff17b	remove limit on the sync reading progress job	2024-02-07 11:09:30 +08:00
Hongbo Wu	95fabe1dbf	keep completed jobs for up to 24 hours and failed jobs for 7 days	2024-02-06 14:21:00 +08:00
Hongbo Wu	53f63b90d7	increase the seed valud of backoff delay to 2 seconds	2024-02-05 16:26:05 +08:00
Hongbo Wu	25e374f6ff	fix: replace webhook endpoint with a bullmq job	2024-02-02 16:18:40 +08:00
Hongbo Wu	5e49131fc2	Merge pull request #3470 from omnivore-app/fix/async-bulk-action ... async bulk action on item count > 100 and create batch jobs	2024-02-02 16:00:12 +08:00
Hongbo Wu	26b7103f0b	improve batch update	2024-02-02 13:13:18 +08:00
Hongbo Wu	1bccf33320	add job_id and parent for tracking	2024-02-02 10:50:07 +08:00
Jackson Harper	ed7ac6c1b4	Fix typo in lifecycle endpoint path	2024-02-01 20:25:52 +08:00
Hongbo Wu	702bd6c3c6	async bulk action on item count > 100 and create batch jobs	2024-02-01 17:44:30 +08:00
Jackson Harper	0e9e331e18	Implement a lifecycle hook for queue processor so it can shut down gracefully	2024-02-01 16:43:57 +08:00
Jackson Harper	1b44cb860b	Function for cron jobs dont include queue in the read position metrics since its not part of that queue	2024-02-01 10:33:51 +08:00
Jackson Harper	7d6645b87a	Rebase	2024-02-01 10:02:38 +08:00
Jackson Harper	570b7a4b78	Use a set in redis and push items to the set ... This lets us grab all available items, apply them, and then remove them from the set. Once completed the items that were applied can be removed from the set, if any new items were added during that time they wont be effected.	2024-02-01 09:59:42 +08:00
Jackson Harper	d8ea4ff1c5	Add reading position messages to exported metrics for queue processor	2024-02-01 09:59:41 +08:00
Jackson Harper	fbfa934479	Write behind reading progress as a list that can be reduced at commit or read time	2024-02-01 09:59:39 +08:00
Hongbo Wu	5e883cb2ba	running worker in the test	2024-01-31 17:45:49 +08:00
Hongbo Wu	5646413b59	add a job for updating highlight annotations in library_item	2024-01-31 14:15:33 +08:00