33 commits

Author	SHA1	Message	Date
Jackson Harper	fbfa934479	Write behind reading progress as a list that can be reduced at commit or read time	2024-02-01 09:59:39 +08:00
Hongbo Wu	5e883cb2ba	running worker in the test	2024-01-31 17:45:49 +08:00
Hongbo Wu	5646413b59	add a job for updating highlight annotations in library_item	2024-01-31 14:15:33 +08:00
Hongbo Wu	5ed35bbeff	create a job for update labels in library item	2024-01-31 13:32:37 +08:00
Hongbo Wu	ddddf8234e	create a job for db trigger	2024-01-30 23:53:05 +08:00
Hongbo Wu	e03895c8b7	enqueue trigger rule job once page is created or updated	2024-01-26 17:46:17 +08:00
Hongbo Wu	be0a2e8c52	use different redis server for mq and cache	2024-01-26 13:01:53 +08:00
Hongbo Wu	e54c691c09	create a job for finding thumbnail	2024-01-26 13:01:51 +08:00
Jackson Harper	a639a2fa10	Add types to exported metrics	2024-01-25 12:37:02 +08:00
Jackson Harper	5f9be385b7	Remove unneeded escapes	2024-01-24 21:34:51 +08:00
Jackson Harper	c581372a93	Improve exported metrics naming convention	2024-01-24 21:13:13 +08:00
Jackson Harper	d24db92e03	Dont force init this queue	2024-01-24 12:40:00 +08:00
Jackson Harper	c29f31d618	Add _count to prom metrics names	2024-01-24 12:39:04 +08:00
Jackson Harper	e4109a63f0	Expose custom metrics for the queue processor	2024-01-24 12:15:38 +08:00
Jackson Harper	e407db914a	Add new job for handling upload content	2024-01-22 15:36:46 +08:00
Jackson Harper	db8284dc0b	Wait until backend queue is ready before putting jobs into it	2024-01-22 11:31:46 +08:00
Jackson Harper	f7b17cb93f	Only create queue object once in queue processor	2024-01-22 11:27:13 +08:00
Hongbo Wu	780cda6e4b	use worker redis client when connect to the queue in queue-processor	2024-01-19 20:44:54 +08:00
Hongbo Wu	338733829f	Merge pull request #3394 from omnivore-app/fix/cache-content-fetched ... fix/cache content fetched	2024-01-19 18:16:05 +08:00
Hongbo Wu	4340286b89	add retry on failure in jobs	2024-01-19 17:38:25 +08:00
Jackson Harper	8ba1242ffb	Back off refresh-all-feeds if queue is very deep	2024-01-19 16:59:41 +08:00
Jackson Harper	2dbff6a6ab	Add some more debug output	2024-01-19 16:12:44 +08:00
Hongbo Wu	a6f0e2f2d9	add save page job processor	2024-01-18 18:48:46 +08:00
Jackson Harper	fd81f65e53	Remove server.close call	2024-01-17 16:37:51 +08:00
Jackson Harper	361d57ac1d	Remove secrets loading as this is handled by k8s now. Fix redis init	2024-01-17 13:33:20 +08:00
Jackson Harper	d1438fe800	Remove old code, clean up redis logging	2024-01-17 11:47:31 +08:00
Jackson Harper	d92ee4bcfe	Clean up redis instantiation	2024-01-17 11:03:52 +08:00
Jackson Harper	50b48a4225	Separate worker and client code for redis connections	2024-01-16 23:21:48 +08:00
Jackson Harper	7c8304ac15	Fix client imports	2024-01-16 22:08:30 +08:00
Jackson Harper	d31d309bc0	MOre on env parsing for k8s	2024-01-16 21:30:17 +08:00
Jackson Harper	5986bd0a68	Change around redis initialization so we can pull in secrets from secret manager async on startup	2024-01-16 21:30:15 +08:00
Jackson Harper	0a82e5721e	Graceful k8s shutdown	2024-01-16 21:27:22 +08:00
Jackson Harper	819052f51f	Add queue-processor endpoint to the api	2024-01-16 21:27:22 +08:00