2536 commits

Author	SHA1	Message	Date
Hongbo Wu	4eba7df84c	always upsert library items	2024-02-06 16:53:32 +08:00
Hongbo Wu	468f835f9e	Merge pull request #3502 from omnivore-app/fix/bullmq-job-retry-backoff ... increase the seed valud of backoff delay to 2 seconds	2024-02-06 15:09:40 +08:00
Hongbo Wu	95fabe1dbf	keep completed jobs for up to 24 hours and failed jobs for 7 days	2024-02-06 14:21:00 +08:00
Hongbo Wu	a939180dc9	fix: dedupe update-db job by using library_item_id as job_id	2024-02-06 14:10:53 +08:00
Hongbo Wu	3477da59bb	Merge pull request #3503 from omnivore-app/fix/force-update-reading-progress ... do not update db immediately for force update of reading progress	2024-02-06 13:12:23 +08:00
Hongbo Wu	d77fad804a	fix tests	2024-02-05 18:29:58 +08:00
Hongbo Wu	37c75865d1	do not update db immediately for force update of reading progress	2024-02-05 16:56:48 +08:00
Hongbo Wu	53f63b90d7	increase the seed valud of backoff delay to 2 seconds	2024-02-05 16:26:05 +08:00
Jackson Harper	2d6bd0779f	Dont fetch from rsshub	2024-02-03 19:08:25 +08:00
Jackson Harper	0a38cd6422	More attempts now that lock timeouts are lower	2024-02-03 19:08:13 +08:00
Jackson Harper	dde80091b4	Merge pull request #3494 from omnivore-app/fix/rss-more-content-fetch-blocks ... Add rss.arxiv and daringfireball to content fetch block lists	2024-02-03 10:57:04 +08:00
Jackson Harper	7fffa8ce75	Add some logging	2024-02-03 10:04:40 +08:00
Jackson Harper	46affc0508	Add xkcd	2024-02-03 10:02:25 +08:00
Jackson Harper	36915ec1ec	Add rss.arxiv and daringfireball to content fetch block lists	2024-02-03 10:01:18 +08:00
Jackson Harper	7937c73de1	Remove old warning thats no longer needed	2024-02-03 09:54:54 +08:00
Jackson Harper	9102fc9ae4	Merge pull request #3491 from omnivore-app/fix/dont-sync-readingtime-too-soon ... Only sync reading times if they are more than a minute old	2024-02-03 09:09:52 +08:00
Jackson Harper	a745e72ce4	Allow setting postgres extra options via envvar	2024-02-03 08:57:41 +08:00
Jackson Harper	9cc046e3a2	Only sync reading times if they are more than a minute old ... this should hopefully reduce contention with other rules running	2024-02-02 20:41:14 +08:00
Hongbo Wu	b68adc5983	fix tests	2024-02-02 17:05:09 +08:00
Hongbo Wu	df2dbb2916	fix: find library item by url not using correct index	2024-02-02 16:46:13 +08:00
Hongbo Wu	74783313da	fix query	2024-02-02 16:18:41 +08:00
Hongbo Wu	1742ffda24	change priority to 5	2024-02-02 16:18:41 +08:00
Hongbo Wu	25e374f6ff	fix: replace webhook endpoint with a bullmq job	2024-02-02 16:18:40 +08:00
Hongbo Wu	5e49131fc2	Merge pull request #3470 from omnivore-app/fix/async-bulk-action ... async bulk action on item count > 100 and create batch jobs	2024-02-02 16:00:12 +08:00
Hongbo Wu	54a4022409	reduce size	2024-02-02 15:45:11 +08:00
Hongbo Wu	e980be7771	skip test	2024-02-02 14:14:41 +08:00
Hongbo Wu	26b7103f0b	improve batch update	2024-02-02 13:13:18 +08:00
Jackson Harper	d339c5e8f1	Use the md5 index when looking up library items by original url	2024-02-02 13:08:00 +08:00
Hongbo Wu	2d1eebfad8	remove size	2024-02-02 12:21:08 +08:00
Hongbo Wu	9c28726a9b	add parent config	2024-02-02 11:49:02 +08:00
Hongbo Wu	1bccf33320	add job_id and parent for tracking	2024-02-02 10:50:07 +08:00
Jackson Harper	ed7ac6c1b4	Fix typo in lifecycle endpoint path	2024-02-01 20:25:52 +08:00
Hongbo Wu	2a0e6f8fa4	fix tests	2024-02-01 17:44:32 +08:00
Hongbo Wu	702bd6c3c6	async bulk action on item count > 100 and create batch jobs	2024-02-01 17:44:30 +08:00
Jackson Harper	7f4f7c6a80	Merge pull request #3479 from omnivore-app/fix/queue-processor-lifecycle-hook ... Implement a lifecycle hook for queue processor so it can shut down gracefully	2024-02-01 17:01:11 +08:00
Hongbo Wu	8ac696e87d	Merge pull request #3480 from omnivore-app/fix/mark-read-action ... fix: update reading progress in cache when mark as read action was triggered by a rule	2024-02-01 16:56:21 +08:00
Hongbo Wu	252443f1e4	fix: update reading progress in cache when mark as read action was triggered in a rule	2024-02-01 16:45:08 +08:00
Jackson Harper	0e9e331e18	Implement a lifecycle hook for queue processor so it can shut down gracefully	2024-02-01 16:43:57 +08:00
Jackson Harper	3feef7ca66	Use the DB stored reading progress when calculating maxes	2024-02-01 16:23:21 +08:00
Jackson Harper	f03b7182b3	Merge pull request #3460 from omnivore-app/feat/reading-progress-write-behind ... Write behind cache for reading progress	2024-02-01 14:43:45 +08:00
Hongbo Wu	abaf726044	revert the change of priority	2024-02-01 13:22:06 +08:00
Hongbo Wu	19fe60d27a	change the update-db job priority to 5 and use exponential backoff strategy for retrying	2024-02-01 13:14:01 +08:00
Hongbo Wu	e13c418389	remove the inner join with highlights table in the sql of update-label job	2024-02-01 13:09:50 +08:00
Jackson Harper	02c28be403	reduce unction is no longer async	2024-02-01 11:24:13 +08:00
Jackson Harper	792f13edfc	Remove async from non-async function	2024-02-01 10:49:48 +08:00
Jackson Harper	1b44cb860b	Function for cron jobs dont include queue in the read position metrics since its not part of that queue	2024-02-01 10:33:51 +08:00
Jackson Harper	7d6645b87a	Rebase	2024-02-01 10:02:38 +08:00
Jackson Harper	570b7a4b78	Use a set in redis and push items to the set ... This lets us grab all available items, apply them, and then remove them from the set. Once completed the items that were applied can be removed from the set, if any new items were added during that time they wont be effected.	2024-02-01 09:59:42 +08:00
Jackson Harper	0e37bfa03d	We need to update readAt because that isnt resolved from the cache	2024-02-01 09:59:42 +08:00
Jackson Harper	c47a720fe4	Do not cache on write, so the most updated item is always available on read ... This is mostly an issue on tests where all the queries can be run in a single process, but if we cache on write, it means the next read will just have the cached value, instead of the calculated maximum value.	2024-02-01 09:59:42 +08:00
Jackson Harper	bdf01ddaab	Clear cache on force, return updated read position values	2024-02-01 09:59:42 +08:00
Jackson Harper	a060aaf496	Make reading position parsing safer	2024-02-01 09:59:42 +08:00
Jackson Harper	9dc070243a	Revert temp	2024-02-01 09:59:41 +08:00
Jackson Harper	d8ea4ff1c5	Add reading position messages to exported metrics for queue processor	2024-02-01 09:59:41 +08:00
Jackson Harper	7bb3718a8f	Add job to sync read position data	2024-02-01 09:59:41 +08:00
Jackson Harper	a40f5bed55	Create a service for interacting with cached read positions	2024-02-01 09:59:41 +08:00
Jackson Harper	fbfa934479	Write behind reading progress as a list that can be reduced at commit or read time	2024-02-01 09:59:39 +08:00
Jackson Harper	c2327781e1	Remove empty ctor	2024-02-01 09:57:11 +08:00
Jackson Harper	86c80d991f	Write behing cache for reading progress	2024-02-01 09:57:11 +08:00
Hongbo Wu	c3276fdaf1	Merge pull request #3466 from omnivore-app/fix/rewrite-trigger-functions-in-db ... create a job for db trigger	2024-01-31 19:47:45 +08:00
Hongbo Wu	6d5ce451ac	fix bulk action	2024-01-31 19:24:18 +08:00
Hongbo Wu	1b8111b907	wait until all jobs are done in test	2024-01-31 19:10:57 +08:00
Hongbo Wu	5e883cb2ba	running worker in the test	2024-01-31 17:45:49 +08:00
Hongbo Wu	cf898299e4	some lint	2024-01-31 14:18:34 +08:00
Hongbo Wu	5646413b59	add a job for updating highlight annotations in library_item	2024-01-31 14:15:33 +08:00
Hongbo Wu	43855247b8	update bulk action to add labels	2024-01-31 13:56:56 +08:00
Hongbo Wu	5ed35bbeff	create a job for update labels in library item	2024-01-31 13:32:37 +08:00
Hongbo Wu	ddddf8234e	create a job for db trigger	2024-01-30 23:53:05 +08:00
Hongbo Wu	36d2c21466	remove thumbnail job on complete or failure	2024-01-30 18:24:48 +08:00
Hongbo Wu	2eedba1522	fix: assign the rule job with priority 1 and do not retry	2024-01-30 18:14:34 +08:00
Hongbo Wu	e724bbd465	fallback to worker redis client if fetched content not found in cache	2024-01-30 12:27:45 +08:00
Hongbo Wu	8d8309d3bb	fix: empty client request id in save-page job data when saving pdf in rss item	2024-01-29 21:22:26 +08:00
Hongbo Wu	5faa24556e	fix: string interpolation in logs	2024-01-29 19:11:26 +08:00
Hongbo Wu	809c572ca7	reduce the size of the log entry by truncating any string values to 500 characters	2024-01-29 14:32:45 +08:00
Hongbo Wu	d5d581fc54	remove redundant logs	2024-01-29 14:32:43 +08:00
Hongbo Wu	0181a2bfcd	Merge pull request #3446 from omnivore-app/fix/refresh-feed ... fix: add rss label and feed url to the pdf in rss feed item	2024-01-29 14:21:08 +08:00
Hongbo Wu	58d384f4f6	fix pdf saving	2024-01-29 14:11:07 +08:00
Hongbo Wu	4f061546bf	remove api call to upload file request	2024-01-29 12:55:19 +08:00
Hongbo Wu	a2bdddf8c9	fix tests	2024-01-29 12:30:31 +08:00
Hongbo Wu	738f85923c	Merge pull request #3442 from omnivore-app/fix/rule-handler ... move trigger rule into a job	2024-01-29 12:20:21 +08:00
Hongbo Wu	ac5acf7f21	fix: add rss label and feed url to the pdf in rss feed item	2024-01-26 22:34:55 +08:00
Hongbo Wu	fd80724d79	fallback to guid if link of rss item is not available	2024-01-26 22:09:53 +08:00
Hongbo Wu	30fef30384	resolve conflicts	2024-01-26 20:32:56 +08:00
Hongbo Wu	e03895c8b7	enqueue trigger rule job once page is created or updated	2024-01-26 17:46:17 +08:00
Hongbo Wu	7556c6da5f	move trigger rule into a job	2024-01-26 17:45:40 +08:00
Hongbo Wu	52419c39da	resolve conflicts	2024-01-26 13:01:53 +08:00
Hongbo Wu	7e89235806	update content-fetch	2024-01-26 13:01:53 +08:00
Hongbo Wu	be0a2e8c52	use different redis server for mq and cache	2024-01-26 13:01:53 +08:00
Hongbo Wu	4c765378e3	do not retry if find_thumbnail job fails	2024-01-26 13:01:53 +08:00
Hongbo Wu	e54c691c09	create a job for finding thumbnail	2024-01-26 13:01:51 +08:00
Jackson Harper	f3c1b9d9a0	Merge pull request #3430 from omnivore-app/fix/queue-processor-custom-metrics ... Expose custom metrics for the queue processor	2024-01-25 21:17:49 +08:00
Hongbo Wu	fd7c2ffb49	fix code blocks not formatted correctly in articles from wechat official accounts	2024-01-25 16:30:59 +08:00
Hongbo Wu	dae90aaffa	fix code blocks not formatted correctly	2024-01-25 16:30:59 +08:00
Hongbo Wu	8834efabd4	throw error if save-page job failed	2024-01-25 16:30:59 +08:00
Hongbo Wu	5e239d2568	run readability in save-page instead of puppeteer	2024-01-25 16:30:59 +08:00
Jackson Harper	a639a2fa10	Add types to exported metrics	2024-01-25 12:37:02 +08:00
Jackson Harper	5f9be385b7	Remove unneeded escapes	2024-01-24 21:34:51 +08:00
Jackson Harper	c581372a93	Improve exported metrics naming convention	2024-01-24 21:13:13 +08:00
Hongbo Wu	7bb63b4ecd	for campability with old clients, use refreshedAt to replace lastFetchedAt	2024-01-24 15:32:43 +08:00
Hongbo Wu	3f3a96297f	update entity	2024-01-24 14:55:50 +08:00
Hongbo Wu	1563b44197	save the timestamp when feed fails	2024-01-24 14:54:39 +08:00
Hongbo Wu	9e55fe0d7b	do not retry if user is deleted	2024-01-24 14:12:34 +08:00
Hongbo Wu	d357d2b0af	fix tests	2024-01-24 13:41:49 +08:00
Hongbo Wu	86c4bbabee	update logs	2024-01-24 13:30:57 +08:00
Hongbo Wu	9d378d3f46	get active user subscriptions only	2024-01-24 13:21:37 +08:00
Hongbo Wu	fd74ebffa2	validate feed item url and convert relative url to absolute url	2024-01-24 13:15:46 +08:00
Hongbo Wu	6f3fccfcd0	mark subscription as error if item was failed	2024-01-24 13:09:43 +08:00
Jackson Harper	d24db92e03	Dont force init this queue	2024-01-24 12:40:00 +08:00
Jackson Harper	c29f31d618	Add _count to prom metrics names	2024-01-24 12:39:04 +08:00
Jackson Harper	e4109a63f0	Expose custom metrics for the queue processor	2024-01-24 12:15:38 +08:00
Hongbo Wu	3cb7c38e7f	update refreshed_at when feed is processed	2024-01-24 12:12:26 +08:00
Hongbo Wu	68a4fb298d	add refreshedAt to the gql schema	2024-01-24 12:03:26 +08:00
Jackson Harper	58d41951c2	Dont not fetch content for medium.com rss feeds ... All these are blocked	2024-01-24 02:11:39 +08:00
Jackson Harper	f8264bfc10	Hard code tweet url to avoid import	2024-01-23 23:45:28 +08:00
Hongbo Wu	c1d5ec722f	do not encode url when getting file data	2024-01-23 22:06:27 +08:00
Hongbo Wu	d4e487e181	set the state to failed if content is not fetched	2024-01-23 21:50:10 +08:00
Hongbo Wu	47da6fb3dc	Merge pull request #3421 from omnivore-app/fix/title ... fix: do not throw error if title is undefined in cache	2024-01-23 19:33:22 +08:00
Hongbo Wu	5495ad782e	fix: do not throw error if title is undefined in cache	2024-01-23 19:19:46 +08:00
Hongbo Wu	8f5d4ae8a0	fix: catch rejected promise if it is in a try-catch block	2024-01-23 18:45:46 +08:00
Hongbo Wu	567ce172e6	make title optional for cache in save-page job	2024-01-23 16:55:17 +08:00
Hongbo Wu	9f7669b010	fix: do not use encoded url as cache key	2024-01-22 16:49:05 +08:00
Jackson Harper	21405e6328	Fix await	2024-01-22 15:48:15 +08:00
Jackson Harper	e407db914a	Add new job for handling upload content	2024-01-22 15:36:46 +08:00
Jackson Harper	3e0f8c884e	Merge pull request #3404 from omnivore-app/debug/extra-logging ... Add logging to save_page service	2024-01-22 13:59:19 +08:00
Jackson Harper	6dd4f49c2f	Add logging to save_page service	2024-01-22 13:32:17 +08:00
Jackson Harper	4da93cbd2d	Merge pull request #3402 from omnivore-app/fix/unique-constraint ... fix: ignore error when saving existing library item in db	2024-01-22 12:38:09 +08:00
Hongbo Wu	141036dc6c	fix tests	2024-01-22 12:18:08 +08:00
Jackson Harper	db8284dc0b	Wait until backend queue is ready before putting jobs into it	2024-01-22 11:31:46 +08:00
Jackson Harper	f7b17cb93f	Only create queue object once in queue processor	2024-01-22 11:27:13 +08:00
Jackson Harper	0f78419e1a	Remove some debug	2024-01-22 11:26:59 +08:00
Hongbo Wu	5964d827c8	fix tests	2024-01-22 11:04:11 +08:00
Jackson Harper	33c994ad5c	Remove RSS feed task handler url as this handled with jobs now	2024-01-22 11:00:55 +08:00
Hongbo Wu	a807a4a18e	fix: do not throw error when creating duplicate item in db	2024-01-22 10:45:47 +08:00
Jackson Harper	9d3a183c45	Dont map label input	2024-01-21 15:44:41 +08:00
Jackson Harper	6cedffc249	Only require original content in savePageJob	2024-01-21 15:30:06 +08:00
Jackson Harper	202d19c9d3	Fix api status mapping	2024-01-21 14:11:53 +08:00
Jackson Harper	5393743c9d	Fix imports of authorized function	2024-01-21 13:33:05 +08:00
Jackson Harper	56a8ef23b6	Remove authorized function from imported helpers ... This creates some issues with init order when using the queue processor.	2024-01-21 13:22:17 +08:00
Jackson Harper	eefceeb1ef	Move checks to have the PDF check	2024-01-21 13:19:50 +08:00
Jackson Harper	9abd0a59da	Clean up imports	2024-01-21 13:16:35 +08:00
Jackson Harper	db3d7dca36	Add missing file	2024-01-21 07:43:50 +08:00
Jackson Harper	a4e707075f	Refactor savePage a bit to use it as a service from the jobs	2024-01-20 21:07:02 +08:00
Jackson Harper	2f405ebed0	Use savePage service instead of API call	2024-01-20 14:07:22 +08:00
Jackson Harper	3a18327230	Remove unused	2024-01-20 13:43:17 +08:00
Jackson Harper	7786221fbc	Fix update_subscription awaits	2024-01-20 13:38:25 +08:00
Jackson Harper	89537c13de	Simplify function input	2024-01-20 13:32:11 +08:00
Jackson Harper	1898e32607	Use service instead of API to update subscription	2024-01-20 13:27:54 +08:00
Jackson Harper	e5dec228f3	Remove some debug	2024-01-20 12:46:22 +08:00
Jackson Harper	4eeb012b78	Pass lits of fetchContentTasks into rss handlers ... This prevents the list from growing on each run	2024-01-20 11:36:14 +08:00
Jackson Harper	2b536c6086	Add more debug	2024-01-20 10:48:57 +08:00
Jackson Harper	52050728ba	Add more debugging logs	2024-01-20 10:29:07 +08:00
Jackson Harper	f4af4593c2	Fix debug line	2024-01-20 10:04:54 +08:00
Jackson Harper	4c1a182c7f	Add a refresh context to make debugging refresh runs easier	2024-01-20 09:50:21 +08:00
Jackson Harper	2961f69f02	Add a refresh context to make debugging rss jobs easier	2024-01-20 09:47:46 +08:00
Jackson Harper	a9eeda9369	Remove non-useful logging line	2024-01-20 09:23:45 +08:00
Jackson Harper	e9f22bba43	Some debug	2024-01-20 09:09:25 +08:00
Hongbo Wu	780cda6e4b	use worker redis client when connect to the queue in queue-processor	2024-01-19 20:44:54 +08:00
Hongbo Wu	338733829f	Merge pull request #3394 from omnivore-app/fix/cache-content-fetched ... fix/cache content fetched	2024-01-19 18:16:05 +08:00
Hongbo Wu	356ad38678	reduce logs	2024-01-19 18:01:09 +08:00
Hongbo Wu	4340286b89	add retry on failure in jobs	2024-01-19 17:38:25 +08:00
Jackson Harper	defb79d8c8	Use let instead of var	2024-01-19 17:12:34 +08:00
Hongbo Wu	51d5166321	get cached content from redis when saving pages	2024-01-19 17:06:37 +08:00
Jackson Harper	8ba1242ffb	Back off refresh-all-feeds if queue is very deep	2024-01-19 16:59:41 +08:00
Jackson Harper	d2491d3eb7	Reduce refresh-all-feeds priority	2024-01-19 16:59:15 +08:00
Jackson Harper	e06ff24cea	Fix logging	2024-01-19 16:44:20 +08:00
Jackson Harper	02818d94a2	Adjust priorities	2024-01-19 16:37:57 +08:00
Hongbo Wu	35805e6e8f	use user provided url for feed added	2024-01-19 16:28:37 +08:00
Jackson Harper	84375b7c36	Add validation before refreshing feed URLs	2024-01-19 16:25:05 +08:00
Jackson Harper	2dbff6a6ab	Add some more debug output	2024-01-19 16:12:44 +08:00
Jackson Harper	0e590e53a6	Improve RSS priorities for refreshing feeds	2024-01-19 16:12:22 +08:00
Hongbo Wu	d135a7fe6a	Merge pull request #3391 from omnivore-app/fix/content-fetch-dockerfile ... fix: build content-fetch in dockerfile	2024-01-19 12:45:19 +08:00
Hongbo Wu	3d42973c5f	fix api endpint	2024-01-19 12:36:15 +08:00
Jackson Harper	f03766d05b	Fix construction of service endpoint ... NEed to fix how we construct all these	2024-01-19 12:29:50 +08:00
Hongbo Wu	c2a084803a	validate feed url before adding	2024-01-18 21:56:19 +08:00
Hongbo Wu	d2c17f7539	graceful shutdown redis	2024-01-18 21:52:33 +08:00
Hongbo Wu	3e291d1019	fix tests	2024-01-18 21:13:32 +08:00
Hongbo Wu	d4f49dceb6	remove job id	2024-01-18 18:48:46 +08:00
Hongbo Wu	a6f0e2f2d9	add save page job processor	2024-01-18 18:48:46 +08:00
Hongbo Wu	8a3b50f07f	do not create thumbnail task if the item already has a thumbnail	2024-01-18 18:48:46 +08:00
Hongbo Wu	d9feb740cb	convert content-fetch to typescript	2024-01-18 18:48:46 +08:00
Jackson Harper	3f707c06f2	Fix checks for env vars	2024-01-18 16:59:53 +08:00
Jackson Harper	3b0338967d	Fix endpoint for internal refresh feed API call ... This should eventually be replaced.	2024-01-18 16:48:45 +08:00
Jackson Harper	6193c78288	Fix typo	2024-01-18 15:37:38 +08:00
Jackson Harper	7754139207	Remove debug	2024-01-18 15:33:51 +08:00
Jackson Harper	fb28d7b3d1	Enqueue new feeds at front	2024-01-18 14:34:07 +08:00
Jackson Harper	ee142d3d7c	Handle SIGINT in server better by moving DB call to after redis shutdown	2024-01-18 12:00:13 +08:00
Jackson Harper	bb55137bb4	Disconnect one client while quitting the other	2024-01-18 11:39:41 +08:00
Jackson Harper	fa685d4a29	Tear down redis with disconnect instead of quit	2024-01-18 11:28:09 +08:00
Hongbo Wu	d558fe2b17	fix tests	2024-01-18 11:24:26 +08:00
Jackson Harper	5a7056b7ec	Dont fail whole job if we cant update a subscription group	2024-01-18 10:18:45 +08:00
Jackson Harper	65347a1996	Use quit when shutting down redis	2024-01-17 17:30:16 +08:00
Jackson Harper	fd81f65e53	Remove server.close call	2024-01-17 16:37:51 +08:00
Jackson Harper	9e6b749604	Update error message	2024-01-17 16:21:42 +08:00
Jackson Harper	e1e2f86219	Remove duplicated error	2024-01-17 16:20:42 +08:00
Jackson Harper	dc4126d52c	REvert some unneeded env changes	2024-01-17 16:15:00 +08:00
Jackson Harper	361d57ac1d	Remove secrets loading as this is handled by k8s now. Fix redis init	2024-01-17 13:33:20 +08:00
Jackson Harper	d1438fe800	Remove old code, clean up redis logging	2024-01-17 11:47:31 +08:00
Jackson Harper	4f23271f29	Remove some debug	2024-01-17 11:45:05 +08:00
Jackson Harper	7a925f3fdc	Clean up redis code, remove connection close in job	2024-01-17 11:14:59 +08:00
Jackson Harper	d92ee4bcfe	Clean up redis instantiation	2024-01-17 11:03:52 +08:00
Jackson Harper	68b7b93586	More logging	2024-01-17 10:42:38 +08:00
Jackson Harper	ec765c0f5c	Some more logging	2024-01-17 10:21:19 +08:00
Jackson Harper	0936942378	Remove connect call	2024-01-16 23:32:53 +08:00
Jackson Harper	50b48a4225	Separate worker and client code for redis connections	2024-01-16 23:21:48 +08:00
Jackson Harper	c5e4e2216b	Simplify connection options	2024-01-16 23:02:46 +08:00
Jackson Harper	49734efec7	Add logging	2024-01-16 22:43:06 +08:00
Jackson Harper	7c8304ac15	Fix client imports	2024-01-16 22:08:30 +08:00
Jackson Harper	7ee3866f34	Pull in the newer connection and retry strategy code	2024-01-16 22:04:19 +08:00
Jackson Harper	90a0455c36	Update job to use ioredis	2024-01-16 22:00:25 +08:00
Jackson Harper	fcf2d07c99	Update redis data source	2024-01-16 21:58:34 +08:00
Jackson Harper	8580398d6f	Clean up post rebase	2024-01-16 21:35:46 +08:00
Jackson Harper	ba3e5b053d	Cleanup after rebase	2024-01-16 21:31:44 +08:00
Jackson Harper	f184c8baff	Fix path	2024-01-16 21:30:17 +08:00
Jackson Harper	84195f10c2	Fix endpoint	2024-01-16 21:30:17 +08:00
Jackson Harper	4df9b70c95	Fix for detecting environment	2024-01-16 21:30:17 +08:00
Jackson Harper	e050864a91	Add more debugging	2024-01-16 21:30:17 +08:00
Jackson Harper	9633e7df79	Update envar name	2024-01-16 21:30:17 +08:00
Jackson Harper	62fdb866a8	Add some extra logging	2024-01-16 21:30:17 +08:00
Jackson Harper	26d832bb92	Trim URLs from feeds	2024-01-16 21:30:17 +08:00
Jackson Harper	652f4b6180	Use createHttpTask for save operation	2024-01-16 21:30:17 +08:00
Jackson Harper	26657c5611	Use cloud task function from backend code instead of from rss-handler	2024-01-16 21:30:17 +08:00
Jackson Harper	a8e8fe23da	Remove azure config from backend	2024-01-16 21:30:17 +08:00
Jackson Harper	69eb22ae3c	Remove some env vars that are not used	2024-01-16 21:30:17 +08:00
Jackson Harper	c7f605d00a	If redis fails assume we need to refetch the item	2024-01-16 21:30:17 +08:00
Jackson Harper	00b72f1c3a	Add lwn.net to block list	2024-01-16 21:30:17 +08:00
Jackson Harper	d31d309bc0	MOre on env parsing for k8s	2024-01-16 21:30:17 +08:00
Jackson Harper	5986bd0a68	Change around redis initialization so we can pull in secrets from secret manager async on startup	2024-01-16 21:30:15 +08:00
Jackson Harper	0a82e5721e	Graceful k8s shutdown	2024-01-16 21:27:22 +08:00
Jackson Harper	7bb6602454	Fix linting	2024-01-16 21:27:22 +08:00
Jackson Harper	7aa76b70fe	Add missing jobs	2024-01-16 21:27:22 +08:00
Jackson Harper	819052f51f	Add queue-processor endpoint to the api	2024-01-16 21:27:22 +08:00
Jackson Harper	b4914f5346	Fix creation of redis clients	2024-01-16 21:27:20 +08:00
Jackson Harper	447bba23ba	Fix init	2024-01-16 21:26:30 +08:00
Jackson Harper	20df5c6212	Parse redis cert on load	2024-01-16 21:26:28 +08:00
Jackson Harper	45fcde8ebc	Update redis config	2024-01-16 21:25:34 +08:00
Jackson Harper	b4ce07efd3	Better handling of cases when redis isnt configured	2024-01-16 21:25:34 +08:00
Jackson Harper	d29c26d93b	Some redis setup	2024-01-16 21:25:34 +08:00
Jackson Harper	fc25d6d615	Enqueue rss-refresh-feed jobs in bull	2024-01-16 21:25:31 +08:00
Hongbo Wu	150a456c35	replace redis client library with ioredis	2024-01-16 15:42:50 +08:00
Hongbo Wu	1e05f7dade	log error	2024-01-16 12:17:07 +08:00
Hongbo Wu	5141c2a042	do not retry saving following items if item is saved	2024-01-16 12:15:21 +08:00
Hongbo Wu	816019e6a3	extend ttl of the saved item cache to 26 hours	2024-01-16 11:17:21 +08:00
Hongbo Wu	06067c1d66	fix tests	2024-01-15 11:26:40 +08:00
Hongbo Wu	bd612bd310	replace username too	2024-01-15 11:23:01 +08:00
Hongbo Wu	deddb51c42	fix: change user email address after user deleted account	2024-01-12 11:45:28 +08:00
Hongbo Wu	d9d800cfc4	add index for sorting by saved_at	2024-01-10 17:09:15 +08:00
Tom Rogers	451939832f	Probably a good idea to add back the schema changes...	2024-01-09 11:11:28 +00:00
Thomas Rogers	c0373646cb	Fix Linting...	2024-01-09 11:22:32 +01:00
Hongbo Wu	1ec5b7affe	fix labels not saved when merging highlights	2024-01-05 23:22:09 +08:00
Hongbo Wu	ccd4f2d3f0	catch redis client error and continue the process	2024-01-05 15:20:42 +08:00
Hongbo Wu	32a58e49f9	make redis optional for backend	2024-01-04 21:55:41 +08:00
Hongbo Wu	bcd05cc7fc	set the url in redis when an item is inserted into db	2024-01-04 18:10:14 +08:00
Hongbo Wu	08c4624fce	save the url in redis for 8 hours so rss-feeder will not try to re-save it	2024-01-04 18:03:34 +08:00
Hongbo Wu	d1db49f9bb	Merge pull request #3300 from omnivore-app/fix/batch-prune ... fix/batch prune	2024-01-04 15:19:50 +08:00
Hongbo Wu	366c97ead7	Merge pull request #3314 from omnivore-app/fix/update-event ... skip sending item update event for reading progress update	2024-01-04 15:17:08 +08:00
Hongbo Wu	d31eb7a88b	skip sending item update event for reading progress update	2024-01-04 15:06:31 +08:00
Hongbo Wu	554c4cee12	add test cases	2024-01-03 18:26:07 +08:00
Hongbo Wu	d31c520a90	add empty trash api to prune soft deleted items in trash	2024-01-03 18:26:06 +08:00
Hongbo Wu	5eb19cda71	prune users in batch	2024-01-03 18:25:16 +08:00
Jackson Harper	a1ea20b964	Update highlights query for new users	2024-01-03 18:21:13 +08:00
Jackson Harper	0b098e640f	Merge pull request #3309 from omnivore-app/fix/includes-query ... if the query contains includes:, we want to search in all folders	2024-01-03 17:48:41 +08:00
Hongbo Wu	060690609d	Merge pull request #3264 from omnivore-app/feature/prune-following-items ... add endpoint to prune items from folder older than some days	2024-01-03 17:46:21 +08:00
Hongbo Wu	6e73b10771	if the query contains includes:, we want to search in all folders	2024-01-03 17:38:12 +08:00
Hongbo Wu	b348fae677	fix multi-select bulk action api	2023-12-29 18:42:07 +08:00
Hongbo Wu	42502f9ff1	Merge pull request #3280 from omnivore-app/fix/slow-db-query ... execute update query in batch to avoid slow bulk action query	2023-12-29 16:48:18 +08:00
Hongbo Wu	7153014a87	fix duplicate labels	2023-12-29 16:36:55 +08:00
Hongbo Wu	6e32c1247d	insert labels on name conflict do nothing	2023-12-29 16:11:11 +08:00
Hongbo Wu	943db3332e	Merge pull request #3276 from omnivore-app/fix/in-library ... by default we want to search in the inbox if no in: is specified	2023-12-29 14:51:00 +08:00
Hongbo Wu	cbb07318e5	get the unchanged result set of sub query by checking updated_at	2023-12-29 12:53:49 +08:00
Hongbo Wu	b5d84098d0	increase batch size to 100	2023-12-29 09:53:23 +08:00
Hongbo Wu	de027ee93f	use cursor and fetch instead of limit and offset because the subquery result set could change during update	2023-12-29 09:53:23 +08:00
Hongbo Wu	55518139fb	generate raw sql because postgres does not support prepared statements in DO blocks	2023-12-29 09:53:23 +08:00
Hongbo Wu	1079a19cd9	execute update query in batch to avoid slow bulk action query	2023-12-29 09:53:23 +08:00
Hongbo Wu	881626d35a	fix type of highlight is null from update since api	2023-12-28 16:55:23 +08:00
Hongbo Wu	2b886d7e68	return highlights in update since api response	2023-12-28 14:03:07 +08:00
Hongbo Wu	47aa2eaae5	by default we want to search in the inbox if no in: is specified	2023-12-22 18:01:38 +08:00
Hongbo Wu	c21ba9e11f	fix tests	2023-12-22 17:12:24 +08:00
Hongbo Wu	ccec94dae0	insert labels only if name unique	2023-12-22 15:14:07 +08:00
Hongbo Wu	c939735678	remove invalid-url error from subscribing feed	2023-12-22 09:51:27 +08:00
Hongbo Wu	f6e04daf60	return invalid url if feed url is different from user provided url	2023-12-22 09:05:43 +08:00
Hongbo Wu	696de8ea50	check both user provided url and parsed feed url when looking for existing subscription	2023-12-22 08:08:15 +08:00
Hongbo Wu	db28bbd0f9	fix invalid unknown item type	2023-12-20 11:51:19 +08:00
Hongbo Wu	0a4758105a	feat: update newsletter email api	2023-12-20 11:29:10 +08:00
Hongbo Wu	e4b6cba2ff	Merge pull request #3261 from omnivore-app/feature/add-state-for-non-fetch-content ... feature/add state for non fetch content	2023-12-19 18:30:13 +08:00
Hongbo Wu	92aa11b98e	add fetchContent api	2023-12-19 18:05:02 +08:00
Hongbo Wu	07b96c46d5	fix tests	2023-12-19 17:45:12 +08:00
Hongbo Wu	b5f82d448e	return folder when newsletter email api is called	2023-12-19 14:52:30 +08:00
Hongbo Wu	8e77c047bd	add endpoint to prune items from folder older than some days	2023-12-19 14:25:55 +08:00
Hongbo Wu	22778af285	run readability on the preview content	2023-12-19 10:41:20 +08:00
Hongbo Wu	b1aca117f4	if fetch_content is false, set the state to content_not_fetched and use preview content	2023-12-19 10:41:20 +08:00
Hongbo Wu	6f1ef075c6	add content_not_fetched state for items saved without content fetched	2023-12-19 10:41:20 +08:00
Hongbo Wu	15dc05ca1a	fix sending rss feed to inbox folder	2023-12-18 19:52:44 +08:00
Hongbo Wu	55ed5a0a80	fix typo	2023-12-18 16:20:40 +08:00
Hongbo Wu	97735d0633	make existing newsletters go to inbox folder	2023-12-18 16:08:36 +08:00
Hongbo Wu	7a2cf6c6fe	fix sort by word count not working	2023-12-18 10:40:10 +08:00
Hongbo Wu	58f16ec62c	Merge pull request #3247 from omnivore-app/feature/feeds ... feat: fetch content and folder settings for newsletters and subscriptions	2023-12-15 16:46:02 +08:00
Hongbo Wu	d579a8f907	not throw error if sort keyword not found	2023-12-15 13:31:22 +08:00
Hongbo Wu	f915c18dfa	allow plural and singular form for has/no filter	2023-12-15 13:18:25 +08:00
Hongbo Wu	42749c5a32	throw error if sort column not exists	2023-12-15 11:47:00 +08:00
Hongbo Wu	b86ee139e0	move to folder api will create a task to fetch content if content not exists	2023-12-14 19:20:03 +08:00
Hongbo Wu	83e0982457	save newsletters to the right folder get from subscription and email settings	2023-12-14 18:53:32 +08:00
Hongbo Wu	f7ec0220a8	allow setting folder when creating a new email	2023-12-14 17:48:02 +08:00
Hongbo Wu	32f99d7b07	allow updating folder and fetch content settings for subscriptions	2023-12-14 17:27:38 +08:00
Hongbo Wu	a257147fc5	fetch following item content only when required	2023-12-14 17:25:55 +08:00
Hongbo Wu	f2c468f906	allow setting fetchContent and folder when subscribe to a feed	2023-12-14 16:32:48 +08:00
Hongbo Wu	d67b974367	add fetch_content and folder column to the subscriptions table and folder column to newsletter_emails table	2023-12-14 15:41:54 +08:00
Hongbo Wu	06e17e2eb8	return source of the labels tagged to an item	2023-12-14 10:40:18 +08:00
Hongbo Wu	dcd6e8bb60	allow adding source of label when adding labels	2023-12-13 16:25:25 +08:00
Hongbo Wu	56a7ae4dc1	add source column to entity_labels table	2023-12-13 14:35:24 +08:00
Hongbo Wu	79baee56c6	Merge pull request #3242 from omnivore-app/fix/subscribe-api ... fix/subscribe api	2023-12-12 20:01:40 +08:00
Hongbo Wu	528163905d	Merge pull request #3244 from omnivore-app/improve-advanced-search ... improve advanced search	2023-12-12 19:53:29 +08:00
Hongbo Wu	9b8e4a6876	fix lint	2023-12-12 19:48:08 +08:00
Hongbo Wu	2155be4704	fix tests	2023-12-12 19:42:09 +08:00
Hongbo Wu	07868c729d	update comments	2023-12-12 18:13:38 +08:00
Hongbo Wu	a7f9214cda	ignore empty text field	2023-12-12 17:56:10 +08:00
Hongbo Wu	609c46cdc3	ignore empty field value	2023-12-12 16:35:50 +08:00
Hongbo Wu	7622d8a139	ignore field case	2023-12-12 16:17:31 +08:00
Hongbo Wu	55cc5f5769	treat unknown fields as text search	2023-12-12 16:14:51 +08:00
Hongbo Wu	a41ec3d2f6	lint	2023-12-12 15:38:39 +08:00
Hongbo Wu	e730abb9aa	fix subscribe api returns error due to createdAt being null	2023-12-12 15:17:00 +08:00
Hongbo Wu	1056971e89	copy published date to the recommended item	2023-12-12 11:17:12 +08:00
Jackson Harper	a7c74cef6f	Merge pull request #3236 from omnivore-app/scan-feed-api ... update scan feed api for the client	2023-12-11 15:35:51 +08:00
Hongbo Wu	b3d0bb9ed7	return absolute feed url	2023-12-11 14:14:54 +08:00
Hongbo Wu	9f59d09c20	validate and return rss feed if the url is an rss feed url	2023-12-11 13:46:35 +08:00
Hongbo Wu	36896a8770	max rss feed defaults to 256	2023-12-11 13:18:43 +08:00
Hongbo Wu	b135dfbc46	get max feed limit from env	2023-12-11 13:01:56 +08:00
Hongbo Wu	4d41117d8b	ignore sort:score since we always order by score when doing fulltext search	2023-12-11 11:54:56 +08:00
Hongbo Wu	0d60ece6d6	make folder optional	2023-12-08 20:03:36 +08:00
Hongbo Wu	da3fede3a4	fix saved search bug: undefined category in input	2023-12-08 19:58:41 +08:00
Hongbo Wu	5e566d450e	update bulk action api	2023-12-08 16:50:39 +08:00
Hongbo Wu	1697f11e75	wrap the where query in parenthesis because we also need to check user_id	2023-12-08 16:49:07 +08:00
Hongbo Wu	36777568ac	create/add RSS labels to the library item when feedUrl is there	2023-12-08 11:29:03 +08:00
Hongbo Wu	3759e10615	fix feed url in pdf file not saved	2023-12-08 11:29:02 +08:00
Hongbo Wu	07d8c72972	create a standalone version of liqe	2023-12-07 18:47:25 +08:00
Hongbo Wu	bf8d081161	wrap non-english words in quotes	2023-12-07 17:59:45 +08:00
Hongbo Wu	5f09165128	allow bulk action on max 100 items	2023-12-07 10:55:25 +08:00
Hongbo Wu	ed29721728	fix bulk action query limited to 1000 char	2023-12-06 22:06:47 +08:00
Hongbo Wu	d42f257f9e	fix searching for multiple labels in iOS	2023-12-06 21:09:30 +08:00
Hongbo Wu	dff2376df3	fix integration handler to use first as number	2023-12-06 14:44:07 +08:00
Hongbo Wu	4ec16a1752	fix parenthesized expression	2023-12-06 14:02:23 +08:00
Hongbo Wu	284697da5b	fix sort	2023-12-06 13:52:17 +08:00
Hongbo Wu	c8bd953cf3	capture the right value of the rss feed url in search string	2023-12-06 10:46:15 +08:00
Hongbo Wu	8579b79eb3	fix tests	2023-12-05 20:58:08 +08:00
Hongbo Wu	4dacceac01	allow use keyword	2023-12-05 20:43:23 +08:00
Hongbo Wu	29e2bf45e3	allow use:folders in search string for new clients to filter inbox items	2023-12-05 20:40:02 +08:00
Hongbo Wu	7c5f90983e	clean up	2023-12-05 13:05:31 +08:00
Hongbo Wu	e6322ffaf8	remove unused code	2023-12-05 13:04:07 +08:00
Hongbo Wu	3c67d1911c	fix tests	2023-12-05 12:31:26 +08:00
Hongbo Wu	073bd8b43a	fix in:trash	2023-12-05 10:58:20 +08:00
Hongbo Wu	88a61d962f	parse sort parameter	2023-12-05 10:58:20 +08:00
Hongbo Wu	a3d3f4daf4	feat: allow logical operator in search query	2023-12-05 10:58:18 +08:00
Hongbo Wu	120fe19dab	fix typo	2023-12-04 18:53:46 +08:00
Hongbo Wu	a116833659	feat: allow moving items to a folder in a bulk	2023-12-04 18:10:14 +08:00
Hongbo Wu	37beab64c6	Merge pull request #3198 from omnivore-app/fix/move-to-folder-api ... fix moveToFolder API error by skipping fetching content again	2023-12-04 16:10:09 +08:00
Hongbo Wu	767629b8c7	replace articleSavingRequest with success (boolean value) as a response	2023-12-04 15:54:14 +08:00
Hongbo Wu	824b8187af	use system token in integration handler	2023-12-04 15:38:08 +08:00
Hongbo Wu	bafcdb2fee	skip rate limiter for system requests	2023-12-04 14:56:26 +08:00
Hongbo Wu	e33c4ad685	fix moveToFolder API error by skipping fetching content again	2023-12-04 14:29:54 +08:00
Hongbo Wu	fe86085245	update comment	2023-12-01 12:19:05 +08:00
Hongbo Wu	706ff94cd1	allow use:folders in search string for new clients to filter inbox items	2023-12-01 12:11:09 +08:00
Hongbo Wu	bd3945fc7e	Merge pull request #3164 from omnivore-app/feature/fetch-content-for-following ... feat: fetch content for rss feed items in following folder	2023-11-28 12:07:17 +08:00
Hongbo Wu	af5040a518	saving posts from telegram channel as rss items	2023-11-24 17:23:35 +08:00
Hongbo Wu	8075cfd431	allow adding telegram channel as feed	2023-11-24 16:06:24 +08:00
Hongbo Wu	989f4f04e0	add scan feeds api to scan feeds from opml or html	2023-11-24 12:11:33 +08:00
Hongbo Wu	022d0f9e66	Merge pull request #3159 from omnivore-app/fix/nested-labels ... feat: wildcard search for labels and nested labels	2023-11-23 21:44:44 +08:00
Hongbo Wu	fd781644f1	feat: fetch content for rss feed items in following folder	2023-11-23 18:03:25 +08:00
Hongbo Wu	750b59cb8d	Merge pull request #3139 from omnivore-app/change-folder-to-state-1	2023-11-23 12:31:37 +08:00
Hongbo Wu	46f3515698	feat: wildcard search for labels and nested labels	2023-11-22 23:46:55 +08:00
Hongbo Wu	4e60396d17	save thumbnail of the following item	2023-11-20 17:28:34 +08:00
Hongbo Wu	e26ed0b6f0	add RSS label to the following feed items	2023-11-17 18:26:00 +08:00
Hongbo Wu	c5a97c7a41	change archive and trash from folder to state	2023-11-17 13:51:33 +08:00
Hongbo Wu	d5d3c72196	Merge pull request #3135 from omnivore-app/change-folder-to-state ... Change category to folder in saved search	2023-11-17 12:31:21 +08:00
Hongbo Wu	fca406bb3d	allow updating taskName	2023-11-17 11:27:30 +08:00
Hongbo Wu	24f31ce1eb	add folder to the updates api params	2023-11-17 11:27:30 +08:00
Hongbo Wu	0869cfd477	replace category with folder in the GQL type	2023-11-16 17:50:30 +08:00
Hongbo Wu	d82eda1f32	allow importing all or unarchived only pocket items	2023-11-16 16:37:14 +08:00
Hongbo Wu	25d0123007	fix tests	2023-11-15 17:41:28 +08:00
Hongbo Wu	f9b389edad	add folder to article and search item type	2023-11-15 17:25:14 +08:00
Hongbo Wu	12c18a3661	add preview content to the search API response	2023-11-15 15:53:01 +08:00
Hongbo Wu	8b92e96db1	generate slug	2023-11-15 14:27:54 +08:00
Hongbo Wu	c6807a554a	item_id and saved_at are optional in the post request	2023-11-15 12:34:35 +08:00
Hongbo Wu	c59b4ca97e	Merge pull request #3054 from omnivore-app/feature/social-feeds ... feature: following	2023-11-15 11:19:14 +08:00
Hongbo Wu	609d30adbb	compatiblity with old saved searches	2023-11-15 10:22:58 +08:00
Hongbo Wu	3d84108202	remove feeds table	2023-11-14 15:27:58 +08:00
Hongbo Wu	2ade0698c5	remove unused code	2023-11-10 23:03:01 +08:00
Hongbo Wu	59bbc5562b	fix tests	2023-11-10 22:37:14 +08:00
Hongbo Wu	adf13131db	add folder to library_item table	2023-11-10 17:34:46 +08:00
Hongbo Wu	69dc1ba47a	fix tests	2023-11-09 22:39:51 +08:00
Hongbo Wu	98cc0e2b82	upsert feed table when subscribing	2023-11-09 22:06:30 +08:00
Hongbo Wu	490b40e467	fix tests	2023-11-09 19:08:52 +08:00
Hongbo Wu	9dfc5f856d	rename variables	2023-11-09 19:00:57 +08:00
Hongbo Wu	ffac5c195d	update comment	2023-11-09 16:13:31 +08:00
Hongbo Wu	f96ba17c92	fetch rss content if required	2023-11-09 16:09:23 +08:00
Hongbo Wu	b7496db56c	add following handler to save following item	2023-11-09 15:53:44 +08:00
Hongbo Wu	cbb5a0a921	revert searchItem changes	2023-11-08 18:16:48 +08:00
Hongbo Wu	cc53992b9a	add addFollowingToLibrary API	2023-11-08 18:16:48 +08:00
Hongbo Wu	99616501cf	add saveFollowing API	2023-11-08 18:16:48 +08:00
Hongbo Wu	3ff9ce3a09	add in:following search filter	2023-11-08 18:16:48 +08:00
Hongbo Wu	98faefa644	add date variables to date search	2023-11-08 18:16:48 +08:00
Hongbo Wu	2c9ec454f5	fix tests	2023-11-08 18:16:48 +08:00
Hongbo Wu	9b0b378c39	check is_fetching_content and only fetch content if required for rss feed items	2023-11-08 18:16:48 +08:00
Hongbo Wu	679512c9d1	add columns to library_item	2023-11-08 18:16:48 +08:00
Hongbo Wu	1da8dc6545	update library_item table	2023-11-08 18:16:48 +08:00
Hongbo Wu	da0d1705b5	add following graphql api to fetch user following	2023-11-08 18:16:48 +08:00
Hongbo Wu	fc783f01ec	add feeds graphql api for searching feed by title or url	2023-11-08 18:16:48 +08:00
Hongbo Wu	878d881e05	add feed entity	2023-11-08 18:16:48 +08:00
Hongbo Wu	e282eea99f	add feeds graphql api to the schema	2023-11-08 18:16:48 +08:00
Jackson Harper	1e336dd61c	Use CLIENT_URL instead of hard coded values	2023-11-08 18:16:32 +08:00
Hongbo Wu	3e9a049306	add fields to the response of user personalization api	2023-11-07 15:39:56 +08:00
Hongbo Wu	fdad3a28af	Merge pull request #3024 from omnivore-app/integratoin ... improvement on integrations	2023-11-03 17:20:17 +08:00
Hongbo Wu	3761b396f4	remove unused code	2023-11-03 10:17:36 +08:00
Hongbo Wu	9ef608b4b6	fix: highlights not added to the content if searching for highlighted content	2023-11-03 10:11:14 +08:00
Hongbo Wu	ffad064d3a	Merge pull request #3051 from naveennamani/patch-1 ... Fixes #3050 - URLs sensitive to trailing slash	2023-11-01 15:48:04 +08:00
Hongbo Wu	e98d029250	remove unused fields in library_item entity so we exclude them from the webhook event	2023-11-01 14:56:36 +08:00
Hongbo Wu	c93fc14de8	remove user from highlight webhook event	2023-11-01 13:26:36 +08:00
Hongbo Wu	30cc952f02	do not publish events if the highlights are recommended by club users	2023-11-01 13:24:06 +08:00
Naveen Namani	ef1adfee98	Fixes #3050 - URLs sensitive to trailing slash ... By passing false to `removeTrailingSlash` for normalize-url call, the URLs sensitive to trailing slash will be parsed correctly	2023-10-30 14:31:46 +05:30
Hongbo Wu	8df6fb3ae2	import unarchived items by default	2023-10-30 16:22:09 +08:00
Hongbo Wu	f6c0b2ba34	add import_item_state to set integration api payload	2023-10-30 16:09:10 +08:00
Hongbo Wu	a528692125	add import_item_state to integration entity	2023-10-30 15:12:18 +08:00
Hongbo Wu	07eb97e7cc	remove unused code	2023-10-30 14:41:53 +08:00
Hongbo Wu	58d5eb6bc0	create an API to create a cloud task for each active integration when the cronjob triggers	2023-10-30 14:41:52 +08:00
Hongbo Wu	5edba30e23	create an integration handler for syncing with readwise	2023-10-30 14:41:52 +08:00
Hongbo Wu	47f67c237d	change source to pocket if item is imported from pocket	2023-10-30 14:41:52 +08:00
Hongbo Wu	6bdc7f01e1	Merge pull request #3027 from omnivore-app/fix/site-search ... fix: site scoped search not working for domain and hostname	2023-10-30 11:08:26 +08:00
Jackson Harper	6db23ba05c	Add intercom identity verification	2023-10-27 16:13:03 +08:00
Hongbo Wu	d480d53151	fix: site scoped search not working for domain and hostname ... * alter site_name_tsv to include hostname and domain * add test cases	2023-10-26 22:10:35 +08:00
Hongbo Wu	a6e8362401	add debugging logs	2023-10-26 12:56:46 +08:00
Hongbo Wu	cf95b6fa07	rename the endpoint to /prune	2023-10-25 14:41:49 +08:00
Hongbo Wu	dddb51f7d5	add test cases	2023-10-25 13:14:49 +08:00
Hongbo Wu	5f5423ddcb	add a pubsub service endpoint for cleanup soft deleted users	2023-10-25 13:02:59 +08:00
Hongbo Wu	c2092e0e5d	Merge pull request #2995 from omnivore-app/fix/item-id-update ... fix: library item id could be updated if a different client request id supplied in save page api payload	2023-10-24 15:24:18 +08:00
Jackson Harper	533e3554fd	Merge pull request #2997 from omnivore-app/feature/reading-progress-api ... ignore position check if force = true in reading progress api payload	2023-10-24 14:45:47 +08:00
Hongbo Wu	056bf9ebf1	fix update sql	2023-10-24 13:12:59 +08:00
Hongbo Wu	3c75a23e75	fix: showing original url instead of signed url for pdf files in search api response	2023-10-24 12:02:58 +08:00
Hongbo Wu	41866d5cb7	cont	2023-10-23 22:02:43 +08:00
Hongbo Wu	3f14ac4f9b	skip publishing event for importing	2023-10-23 21:31:38 +08:00
Hongbo Wu	79a647deb4	do not publish content and original html in the queue	2023-10-23 21:27:16 +08:00
Hongbo Wu	f46bb616fa	update read_at to now	2023-10-23 21:14:23 +08:00
Hongbo Wu	e673db423b	limit max content size to 10MB	2023-10-23 21:11:37 +08:00
Hongbo Wu	f18ea7a7ce	ignore position check if force = true in reading progress api payload	2023-10-23 20:57:20 +08:00
Hongbo Wu	0c60706503	fix: date filter in bulk action api	2023-10-23 19:34:41 +08:00
Hongbo Wu	27b352f9ae	fix: updates since api returns error for android client if invalid date is supplied	2023-10-23 19:07:47 +08:00
Hongbo Wu	e94a61a9bc	fix: library item id could be updated if a different client request id supplied in save page api payload	2023-10-23 17:44:38 +08:00
Hongbo Wu	b26e15c5d6	lint	2023-10-23 15:44:47 +08:00
Hongbo Wu	bbc46c6389	fix: when source is rss-feed or csv-import, do not parse items in the backend	2023-10-23 15:43:19 +08:00
Hongbo Wu	850569d7e4	add test cases	2023-10-23 12:24:19 +08:00
Hongbo Wu	7763a8a939	perf: limit search item results to 100 a time and fetch labels and recommendations only needed	2023-10-23 10:52:03 +08:00
Jackson Harper	1ebfb94309	Merge pull request #2977 from Rudra-Ji/main ... fix typo	2023-10-20 18:00:43 +08:00
Hongbo Wu	b2a61b2d7c	Merge pull request #2969 from omnivore-app/fix/query-group ... Group RSS subscriptions by feed URL and filter by scheduled time	2023-10-20 14:45:44 +08:00
Hongbo Wu	6618a6ed19	migrate popular reads into database	2023-10-20 14:31:16 +08:00
Rudra	23eb7ea76f	fix typo	2023-10-20 11:51:02 +05:30
Jackson Harper	a23b67b70b	Dont rate limit auth in local env so tests can run	2023-10-20 11:44:52 +08:00
Jackson Harper	b9e2f9ee53	Add a longer rate limit window on createaccount/reset password, reduce api rate limit hits	2023-10-20 11:28:16 +08:00
Hongbo Wu	4e36a5809e	fix failure of re-subscribing rss feed	2023-10-20 11:05:58 +08:00
Hongbo Wu	750ccb0c42	fix conflicts with main	2023-10-19 17:44:53 +08:00
Hongbo Wu	1880f2ace4	Update rss handler to fetch a group of rss feeds in one task	2023-10-19 17:27:55 +08:00
Hongbo Wu	f50d1c9cb9	Group RSS subscriptions by feed URL and create cloud task for the groups	2023-10-19 17:27:08 +08:00
Hongbo Wu	5283a84908	add scheduledAt to the task payload	2023-10-19 17:27:08 +08:00
Hongbo Wu	18aece84e7	add scheduledAt to subscriptions	2023-10-19 17:27:08 +08:00
Jackson Harper	c93e61ec24	Remove debug	2023-10-18 21:30:23 +08:00
Jackson Harper	78a642db6d	Update type signature	2023-10-18 19:53:26 +08:00
Jackson Harper	5c576347a8	MOre debug	2023-10-18 19:26:45 +08:00
Jackson Harper	fbaaaeca7c	Add some debugging	2023-10-18 18:48:18 +08:00
Jackson Harper	7848413645	Update schema	2023-10-18 16:34:20 +08:00
Jackson Harper	cf101c6d18	Cache and check feed checksums to reduce fetching	2023-10-18 16:26:03 +08:00
Hongbo Wu	46313b14ba	set trust proxy = true if set in env var	2023-10-18 13:06:33 +08:00
Hongbo Wu	988b20e30c	test trust proxy=true	2023-10-18 12:30:27 +08:00
Hongbo Wu	506004d410	test trust proxy=2	2023-10-18 11:58:15 +08:00
Hongbo Wu	32ba744d20	test trust proxy=1	2023-10-18 11:04:32 +08:00
Hongbo Wu	91076014cd	log req.ip	2023-10-18 11:04:32 +08:00
Hongbo Wu	0874f6ff21	log x-forwarded-for header for debugging purpose	2023-10-18 11:04:32 +08:00
Jackson Harper	71aaef3628	Update attachment URL	2023-10-18 10:43:53 +08:00
Jackson Harper	b88c64424f	Special handling of file URLs	2023-10-18 10:37:21 +08:00
Jackson Harper	78843fff14	Remove import	2023-10-18 10:27:09 +08:00
Jackson Harper	4220922ff8	Preserve original URL if possible	2023-10-18 10:23:26 +08:00
Jackson Harper	b6ce4f39b1	Fix imports	2023-10-18 10:15:49 +08:00
Jackson Harper	25c0051fd4	Only use signed URLs for PDF attachments	2023-10-18 10:14:35 +08:00
Hongbo Wu	fa2fba1a2c	Merge pull request #2942 from omnivore-app/fix/rules ... improvement on subscriptions and delete user api	2023-10-17 17:14:49 +08:00
Hongbo Wu	700fd73dbb	disallow social login if account is deleted	2023-10-17 17:07:46 +08:00
Hongbo Wu	6e79f6c67f	update sql to deduplicate rss subscriptions	2023-10-17 16:41:14 +08:00
Hongbo Wu	eb667e04b9	update reading progress in db	2023-10-17 15:09:16 +08:00
Hongbo Wu	1b2d93e118	remove duplicate rss subscriptions by user and url; create a unique constraint	2023-10-16 17:35:29 +08:00
Jackson Harper	9e8ccb35e3	Limit users to a single import a day	2023-10-16 15:48:03 +08:00
Hongbo Wu	e54c1c81a1	soft delete user by calling delete account api	2023-10-16 15:24:53 +08:00
Jackson Harper	34798f39f0	Disable JSONLD fetching which can be quite slow	2023-10-16 11:25:37 +08:00
Jackson Harper	a20b956e26	Merge pull request #2926 from omnivore-app/fix/api-site-name-query ... Add ILIKE queries to all field queries	2023-10-13 22:02:55 +08:00
Hongbo Wu	497b6e9f72	fetch highlights if necessary and before converting content to markdown	2023-10-13 18:27:57 +08:00
Jackson Harper	3f4be32df8	Merge library item imports	2023-10-13 18:26:02 +08:00
Hongbo Wu	a6c5f4e862	fix bulk action slow query	2023-10-13 13:45:09 +08:00
Hongbo Wu	4d64231abf	save content reader in the item	2023-10-12 12:59:36 +08:00
Hongbo Wu	d1472c6519	Merge pull request #2915 from omnivore-app/fix/drop-label-position-trigger ... drop position trigger on labels table and sort labels by name returned by labels api	2023-10-12 12:10:33 +08:00
Hongbo Wu	9b25a64fd8	Merge pull request #2916 from omnivore-app/fix/email-attachment ... fix email attachment not being saved as item	2023-10-12 12:09:01 +08:00
Hongbo Wu	039632556b	add more method in mock cloud storage and update test cases for email-attachment rest api	2023-10-12 11:56:54 +08:00
Hongbo Wu	1bc9912271	fix failure of saving highlight when position is null	2023-10-12 11:27:46 +08:00
Hongbo Wu	ebb45f1973	fix email attachment not being saved as item	2023-10-12 10:47:20 +08:00
Hongbo Wu	f1c4bc6a74	fix email attachment not being saved as item	2023-10-12 10:46:12 +08:00
Hongbo Wu	b0f0dba53d	drop position trigger on labels table and sort labels by name returned by labels api	2023-10-12 10:39:35 +08:00
Hongbo Wu	30ba22cbbc	add user_id to the sql query to reduce query latency	2023-10-11 16:05:35 +08:00
Hongbo Wu	a4aeb40276	fix: after successfully deleted highlights, still return error codes	2023-10-11 14:48:44 +08:00
Hongbo Wu	768845e101	update the filter value to use is:reading of the Continue Reading filter	2023-10-10 15:33:59 +08:00
Hongbo Wu	62ffaa924b	add wordsCount filter	2023-10-10 12:52:49 +08:00
Hongbo Wu	93c410a564	fix is:read filter and add is:reading filter	2023-10-10 12:21:32 +08:00
Hongbo Wu	3e91f3836b	fix tests	2023-10-10 10:58:48 +08:00
Hongbo Wu	835b61064f	allow filter by wordsCount	2023-10-10 10:58:48 +08:00
Hongbo Wu	1adbed9876	revert postgres connection timeout	2023-10-10 10:33:49 +08:00
Hongbo Wu	bab343c41f	testing	2023-10-10 10:10:45 +08:00
Hongbo Wu	ac0d295cf3	debug	2023-10-09 22:06:35 +08:00
Hongbo Wu	0b7da62aca	debug	2023-10-09 21:34:52 +08:00
Hongbo Wu	f4f38e4b31	reduce unnecessary logs	2023-10-09 17:26:03 +08:00
Hongbo Wu	78490e15f1	disable db connection timeout	2023-10-09 15:30:30 +08:00
Hongbo Wu	50901d25a3	remove elastic url from env var	2023-10-09 13:00:30 +08:00
Hongbo Wu	e424328f20	fix recommendations not fetched in search results	2023-10-05 23:00:29 +08:00
Hongbo Wu	ac576aa5c7	fix isArchived = false when archivedAt is not null	2023-10-05 17:53:57 +08:00
Hongbo Wu	39b3a1cad4	resolve conflicts	2023-10-05 14:37:21 +08:00
Hongbo Wu	463231b26b	fix trigger	2023-10-05 14:32:05 +08:00
Hongbo Wu	0e7f45a758	create a trigger to update recommender_names in library_item table	2023-10-05 14:32:05 +08:00
Hongbo Wu	f6c906201b	fetch recommendations separately too	2023-10-05 14:32:05 +08:00
Hongbo Wu	d4404b0248	fetch labels and highlights after items fetched in a search query	2023-10-05 14:32:05 +08:00
Hongbo Wu	bb59ed757c	fix word count not return	2023-10-05 14:32:05 +08:00
Hongbo Wu	cbf58614c4	use log levels in custom typeorm logger	2023-10-05 14:32:05 +08:00
Hongbo Wu	efdbc4f345	use interpolated string or dictionary parameter in logs	2023-10-05 14:32:03 +08:00
Hongbo Wu	327f6959ef	fix highlight createdByMe is wrong	2023-10-05 14:31:07 +08:00
Hongbo Wu	0ff9fdef04	add elastic scroll time in the env var	2023-10-05 14:31:07 +08:00
Hongbo Wu	0912e7cf0f	remove warn logs	2023-10-05 14:31:07 +08:00
Hongbo Wu	3b8b48bc21	fix recovered deleted items still shown in trash	2023-10-05 14:31:07 +08:00
Hongbo Wu	7bf3f1bbf4	set deletedAt to now when bulk delete	2023-10-05 14:31:07 +08:00
Hongbo Wu	47510c0785	fix recommendations not fetched in search results	2023-10-05 14:31:07 +08:00
Hongbo Wu	9c723ecbd3	fix id not set in page update pubsub message	2023-10-05 14:31:07 +08:00
Hongbo Wu	0c691257eb	fix rules permission	2023-10-05 14:31:07 +08:00
Hongbo Wu	5f6587537e	fix recommendation wildcard search	2023-10-05 14:31:07 +08:00
Hongbo Wu	bf2c54084b	fix getArticle API failed on iOS	2023-10-05 14:31:07 +08:00
Hongbo Wu	9e5196406e	revert rls on old tables	2023-10-05 14:31:07 +08:00
Hongbo Wu	2e5529fd81	allow readingProgressAnchorIndex to be null	2023-10-05 14:31:07 +08:00
Hongbo Wu	8712107a2f	remove db caching	2023-10-05 14:31:07 +08:00
Hongbo Wu	c57ead3583	add test for when rss:feed in:archive is in the query	2023-10-05 14:31:07 +08:00
Hongbo Wu	867c31a582	add test for searching in:archive label:test	2023-10-05 14:31:07 +08:00
Hongbo Wu	6cb4bb8de8	fix in:subscription search	2023-10-05 14:31:07 +08:00
Hongbo Wu	15f9017291	fix in:archive search	2023-10-05 14:31:07 +08:00
Hongbo Wu	a35e6c0147	fix sort by read time bug	2023-10-05 14:31:07 +08:00
Hongbo Wu	dc98719b85	reduce log	2023-10-05 14:31:07 +08:00
Hongbo Wu	0c5ba53c26	fix recommendations	2023-10-05 14:31:07 +08:00
Hongbo Wu	b9af543b45	fix rss feeds	2023-10-05 14:31:06 +08:00
Hongbo Wu	673ce83c33	fix searching for site	2023-10-05 14:31:06 +08:00
Hongbo Wu	4ff5484d8e	change item_type to text	2023-10-05 14:31:06 +08:00
Hongbo Wu	7c56973fad	fix update filter api	2023-10-05 14:31:06 +08:00
Hongbo Wu	e696709a69	resolve conflicts	2023-10-05 14:31:06 +08:00
Hongbo Wu	7225d861f3	use skip and take instead of limit, offset	2023-10-05 14:31:06 +08:00
Hongbo Wu	68e381deb2	revert note changes	2023-10-05 14:31:06 +08:00
Hongbo Wu	fdd771bc06	add migration script	2023-10-05 14:31:06 +08:00
Hongbo Wu	dd292879a4	sort type ahead search result by saved at desc order	2023-10-05 14:31:06 +08:00
Hongbo Wu	2b9a6b26f6	debug test error	2023-10-05 14:30:37 +08:00
Hongbo Wu	a284884398	fix rebase conflicts	2023-10-05 14:30:37 +08:00
Hongbo Wu	5a9dc344d3	fix rebase conflicts	2023-10-05 14:30:35 +08:00
Hongbo Wu	cbaa3d6900	fix newsletters test	2023-10-05 14:30:10 +08:00
Hongbo Wu	c3caf15e44	fix auth tests	2023-10-05 14:30:10 +08:00
Hongbo Wu	024d558cfe	update labels rls	2023-10-05 14:30:10 +08:00
Hongbo Wu	6672ca0109	fix labels tests	2023-10-05 14:30:10 +08:00
Hongbo Wu	4b95b732cf	fix highlight tests	2023-10-05 14:30:10 +08:00
Hongbo Wu	b036149fe2	fix feature tests	2023-10-05 14:30:10 +08:00
Hongbo Wu	d50174ef76	fix article tests	2023-10-05 14:30:10 +08:00
Hongbo Wu	669cddeec8	fix some rls issue	2023-10-05 14:30:10 +08:00
Hongbo Wu	cea5f790d8	fix some tests	2023-10-05 14:30:09 +08:00
Hongbo Wu	01c74cb7c4	fix tests for popular reads	2023-10-05 14:29:25 +08:00
Hongbo Wu	50bfc1fa74	fix tests	2023-10-05 14:29:23 +08:00
Hongbo Wu	bab957256d	fix multiple parameters in query builder	2023-10-05 14:28:52 +08:00
Hongbo Wu	e1a7a24323	improve search result	2023-10-05 14:28:52 +08:00
Hongbo Wu	1bade22076	fix file uploading	2023-10-05 14:28:52 +08:00
Hongbo Wu	e05d341952	fix file uploading	2023-10-05 14:28:52 +08:00
Hongbo Wu	86cff913fb	fix label search	2023-10-05 14:28:52 +08:00
Hongbo Wu	0747c4603c	fix bulk action	2023-10-05 14:28:52 +08:00
Hongbo Wu	afa1165a29	fix setting labels	2023-10-05 14:28:52 +08:00
Hongbo Wu	33ba17707b	fix setting labels	2023-10-05 14:28:52 +08:00
Hongbo Wu	0016b0a9a7	fix update library item	2023-10-05 14:28:52 +08:00
Hongbo Wu	d9d71aa443	fix highlights api	2023-10-05 14:28:52 +08:00
Hongbo Wu	5ed7e819e8	fix label constraint	2023-10-05 14:28:52 +08:00
Hongbo Wu	b72acf8e6f	fix permission issue	2023-10-05 14:28:52 +08:00
Hongbo Wu	4a55e801b4	fix typo	2023-10-05 14:28:50 +08:00
Hongbo Wu	3f572ea89a	fix typo	2023-10-05 14:27:52 +08:00
Hongbo Wu	65bb4aa85f	remove knex	2023-10-05 14:27:18 +08:00
Hongbo Wu	c1ad9b6f41	cont	2023-10-05 14:27:18 +08:00
Hongbo Wu	f8f3ec0c56	save claim in httpContext	2023-10-05 14:27:17 +08:00
Hongbo Wu	b1899e340d	replace library item	2023-10-05 14:25:10 +08:00
Hongbo Wu	c655f78f8c	replace highlight	2023-10-05 14:24:28 +08:00
Hongbo Wu	381e9bc173	replace database subscriber with db trigger	2023-10-05 14:24:05 +08:00
Hongbo Wu	612151c151	remove elastic from docker-compose	2023-10-05 14:24:04 +08:00
Hongbo Wu	5dce78eedb	update entity subscriber	2023-10-05 14:22:58 +08:00
Hongbo Wu	648a16615b	update search query	2023-10-05 14:22:58 +08:00
Hongbo Wu	751df68175	implement search function	2023-10-05 14:22:58 +08:00
Hongbo Wu	0b309b7a45	add recommendation db table and entity	2023-10-05 14:22:58 +08:00
Hongbo Wu	5f0e1af285	update highlight entity	2023-10-05 14:22:58 +08:00
Hongbo Wu	06000264d2	replace saveEmail	2023-10-05 14:22:58 +08:00
Hongbo Wu	fc52b522f2	fix test	2023-10-05 14:22:58 +08:00
Hongbo Wu	9370951893	create a repository directory	2023-10-05 14:22:56 +08:00
Hongbo Wu	425864ce01	remove knex	2023-10-05 14:21:56 +08:00
Hongbo Wu	928ed2e6a9	add libraryItemPreview entity class	2023-10-05 14:18:21 +08:00
Hongbo Wu	e1af53ab48	add libraryItem entity class	2023-10-05 14:18:21 +08:00
Hongbo Wu	789cb1d501	Merge pull request #2860 from omnivore-app/fix/self-hosting-login ... fix demo user not able to login when self hosting	2023-10-05 12:02:46 +08:00
Hongbo Wu	c008c3154d	make reminder task url optional env var	2023-10-05 11:36:34 +08:00
Hongbo Wu	16f72d6a74	fix labels and highlights were deleted when deleting items	2023-10-05 10:03:06 +08:00
Hongbo Wu	418f27f586	fix convert to email login does not change source field in user table	2023-10-04 16:35:50 +08:00
Jackson Harper	c41363b51c	Merge pull request #2831 from omnivore-app/feature/convert-to-email-api ... Update Email Feature	2023-10-04 14:30:35 +08:00
Hongbo Wu	3cf0c45fa9	mark unsubscribed newsletter in the db	2023-10-03 12:50:55 +08:00
Hongbo Wu	ee50529f95	add update email form on account page	2023-10-02 22:21:50 +08:00
Hongbo Wu	c40e64773a	convert to email api	2023-10-02 17:01:13 +08:00
Hongbo Wu	082e83b1ad	fix 406 not acceptable error from subscribing an rss feed	2023-10-02 10:52:50 +08:00
Hongbo Wu	010e2841da	fix getting 403 when parsing some rss url by setting a user agent in the request	2023-09-27 10:38:49 +08:00
Jackson Harper	21244a394d	Move debug directory to not interfere with usernames	2023-09-22 17:56:25 +08:00
Jackson Harper	53ac43563d	Set user properties instead of event properties on labels setter	2023-09-19 12:30:20 +08:00
Jackson Harper	6c6f27be3e	Properly tag user on labels event	2023-09-14 16:31:01 +08:00
Jackson Harper	e4b5bd6a67	Add code to join users without email on login	2023-09-14 11:24:19 +08:00
Jackson Harper	014614d097	Merge pull request #2759 from omnivore-app/feat/web-use-posthog ... Remove segment from the web	2023-09-13 14:01:08 +08:00
Jackson Harper	dfcfe64b1b	Add a create_user event	2023-09-13 13:44:26 +08:00
Jackson Harper	0946f3d3e8	Fix some console logs, better handle posthog	2023-09-13 12:52:34 +08:00
Jackson Harper	842e4bbf98	Update help links	2023-09-13 10:22:45 +08:00
Jackson Harper	293becf596	Ignore link density checks in newsletters	2023-09-12 15:53:43 +08:00
Jackson Harper	2a3214e393	Remove identify call from backend	2023-09-12 09:53:19 +08:00
Jackson Harper	a5eb527ae1	Merge pull request #2738 from omnivore-app/fix/api-enable-rules-notifications ... Remove debounce from notification rule	2023-09-07 18:07:06 +08:00
Jackson Harper	facde7d9b3	Remove import	2023-09-07 17:51:20 +08:00
Jackson Harper	e94529a96c	Merge pull request #2707 from Podginator/feat/saved-links ... Add Saved Searches, the ability to add new searches, delete searches, and reorder searches.	2023-09-07 17:27:48 +08:00
Jackson Harper	edd7246a9b	Remove debounce from notification rule	2023-09-07 17:24:49 +08:00
Jackson Harper	d0583bf271	Disable persisted queries in Apollo	2023-09-07 14:49:48 +08:00
Jackson Harper	f7e5a8cbfd	Increase unauthed rate limit	2023-09-07 14:01:18 +08:00
Thomas Rogers	79f42a5bde	Merge Main	2023-09-06 17:54:18 +02:00
Thomas Rogers	13ab13d94d	Code Review Comments	2023-09-06 17:51:42 +02:00
Jackson Harper	d2534d692b	Better preview URLs for onboarding articles	2023-09-04 18:27:56 +08:00
Thomas Rogers	faf7880fcd	Fix some bugs regarding scrolling, fix some UI Issues, and correct the name of Non-feed Filter.	2023-08-31 20:08:54 +02:00
Thomas Rogers	a58864fe2c	Merge main	2023-08-30 19:48:04 +02:00
Thomas Rogers	d433d48365	Add Filters, and editing filters.	2023-08-30 19:37:24 +02:00
Jackson Harper	79cea907a3	Fix typo with RSS typed subscription queries	2023-08-30 14:18:39 +08:00
Jackson Harper	fe3d3be7f5	Make query more clear	2023-08-29 20:17:13 +08:00
Jackson Harper	6c1484cd09	Add type filter	2023-08-29 20:06:28 +08:00
Jackson Harper	3565c7000e	Update subscriptions query	2023-08-29 17:03:49 +08:00
Jackson Harper	8851017100	Use andWhere query to not overwrite the where	2023-08-29 15:54:07 +08:00
Jackson Harper	89c17fd0ab	Only filter inactive newsletters out from the subscriptions response	2023-08-29 11:34:55 +08:00
Jackson Harper	49496f7aa2	Only return active subscriptions	2023-08-29 10:19:57 +08:00
Jackson Harper	feaa539c15	Use and where for type query. Prettier changes	2023-08-28 15:23:14 +08:00
Jackson Harper	aa0f4399aa	Remove prettier changes	2023-08-28 15:15:05 +08:00
Jackson Harper	38c9bbdcaa	Allow getting all subscritons regardless of type at the same time	2023-08-28 15:11:16 +08:00
Hongbo Wu	ec574d0ad9	use new color or the color of the last overlap highlight when merging highlights	2023-08-18 13:54:38 +08:00
Hongbo Wu	ce480806c0	add color to create/update/merge highlight api and highlight schema	2023-08-18 13:30:54 +08:00
Hongbo Wu	cd1fc5c4db	fix typo	2023-08-17 18:46:39 +08:00
Hongbo Wu	c45b760f71	fix test	2023-08-17 18:42:11 +08:00
Hongbo Wu	e54e61f3c0	reduce log size	2023-08-17 18:33:14 +08:00
Hongbo Wu	dedd76cfb7	do not publish a pubsub event if the page is imported	2023-08-17 18:24:57 +08:00
Hongbo Wu	ec88a9fe8e	we don't want to create thumbnail for imported pages	2023-08-17 18:05:51 +08:00
Hongbo Wu	0609043956	reduce retry	2023-08-17 17:59:24 +08:00
Hongbo Wu	6f315ef4d9	create a transaction for uploading files in the upload_files resolver	2023-08-16 21:19:03 +08:00
Hongbo Wu	755fb66588	Merge pull request #2651 from omnivore-app/feature/csv-importer ... feature/csv importer	2023-08-15 14:09:36 +08:00
Hongbo Wu	9ad9cea917	raise rss limit to 150	2023-08-14 22:27:43 +08:00
Hongbo Wu	5f6be169bd	add savedAt and publishedAt to saveUrl api	2023-08-14 17:10:34 +08:00
Hongbo Wu	fb845152d9	add header to the pocket data csv file	2023-08-14 13:44:30 +08:00
Hongbo Wu	93cca5040e	add comment	2023-08-11 14:41:21 +08:00
Hongbo Wu	05e73ebc9c	reduce readwise api timeout to 5 sec	2023-08-11 14:38:45 +08:00
Hongbo Wu	b18018ab2e	reduce max content size to 5MB and 10MB for both content and originalHtml	2023-08-11 14:02:29 +08:00
Hongbo Wu	792cf0b207	upgrade google cloud sdk	2023-08-11 13:55:29 +08:00
Hongbo Wu	55841e595f	fix dependecies	2023-08-11 11:12:03 +08:00
Hongbo Wu	61f553fd09	Merge pull request #2616 from omnivore-app/fix/thumbnail-delay ... fix thumbnail not pre-cached if it was already set	2023-08-10 18:45:48 +08:00
Hongbo Wu	7bd747601b	update thumbnail if failed to fetch and return url if already signed	2023-08-10 18:41:10 +08:00
Hongbo Wu	f06d2558c2	update thumbnail if failed to fetch and return url if already signed	2023-08-10 18:41:03 +08:00
Hongbo Wu	8809c9764a	max content length = 10MB	2023-08-10 18:12:29 +08:00
Hongbo Wu	fc76897df3	timeout = 5 when syncing omnivore data to readwise	2023-08-10 15:23:03 +08:00
Hongbo Wu	d4ac069503	fix webhook logs	2023-08-10 15:07:00 +08:00
Hongbo Wu	6acedb64c8	fix maximum call stack size error	2023-08-10 14:18:29 +08:00
Hongbo Wu	7506e27dd0	create or update an opt-in record with null grantedAt if exceeding max users	2023-08-10 13:47:39 +08:00
Hongbo Wu	163dbc73a6	do not send content in the thumbnail task payload	2023-08-10 11:55:59 +08:00
Hongbo Wu	51e0dfde23	set new limit = 1500	2023-08-10 09:43:40 +08:00
Hongbo Wu	3df0ffc5d3	fix existing opted in users not being able to be granted with the feature after increasing the limit	2023-08-08 22:08:37 +08:00
Hongbo Wu	3570079824	fix thumbnail not pre-cached if it was already set	2023-08-07 18:20:59 +08:00
Hongbo Wu	55e378c9b5	Merge pull request #2572 from omnivore-app/feature/pause-rss-feed ... add a button to pause rss feed	2023-08-03 13:33:11 +08:00
Hongbo Wu	f4e537a4c7	set max redirect = 5 and timeout = 30 seconds	2023-08-03 13:31:18 +08:00
Hongbo Wu	82d820ea8f	get link following the order of preference: via, self, alternate	2023-08-03 13:05:26 +08:00
Hongbo Wu	f3994a8a06	fix test	2023-08-02 22:00:09 +08:00
Hongbo Wu	b403dea247	add a button to pause rss feed	2023-08-01 22:07:53 +08:00
Jackson Harper	3d6d421e84	Merge pull request #2566 from omnivore-app/feat/web-improved-cards ... WIP: grid card design	2023-08-01 20:34:39 +08:00
Jackson Harper	c0dbbc3b86	Update icons	2023-08-01 12:31:12 +08:00
Hongbo Wu	00e7e49a4b	increase the max number of rss subscriptions to 50	2023-07-31 21:44:50 +08:00
Hongbo Wu	d0ad636b7f	use safe stringify in logs	2023-07-28 16:52:28 +08:00
Hongbo Wu	f20e12311a	check if the page is from rss feed	2023-07-28 13:53:19 +08:00
Hongbo Wu	9bc7ed92c0	we don't want to update an rss feed page if rss-feeder is tring to re-save it	2023-07-28 12:43:04 +08:00
Hongbo Wu	2e965afd69	reformat sql query in logs	2023-07-27 17:34:12 +08:00
Hongbo Wu	99a52f8d56	replace all the console logs with logger logs	2023-07-27 16:06:44 +08:00
Hongbo Wu	808f6f3d3a	remove trace	2023-07-26 22:00:37 +08:00
Jackson Harper	5bf3213e49	Dont send newsletter notifications	2023-07-26 20:53:50 +08:00
Hongbo Wu	61967fdde5	fix test	2023-07-26 18:07:25 +08:00
Hongbo Wu	ced1a711eb	fix test	2023-07-26 17:12:40 +08:00
Hongbo Wu	7b36156ce6	add a trace when log size too big	2023-07-26 17:07:24 +08:00
Hongbo Wu	8cd47eec11	run prettier	2023-07-26 16:41:55 +08:00
Hongbo Wu	07d43812a7	use winston logger instead of console log so the log will be truncated if it exceeds the max entry size	2023-07-26 16:27:18 +08:00
Hongbo Wu	66490df72f	deeply truncate log object size	2023-07-26 15:52:13 +08:00
Hongbo Wu	f0e62b1d2a	use lodash instead of standalone lodash.debounce package	2023-07-26 13:41:51 +08:00
Hongbo Wu	0a227decdd	Merge pull request #2515 from omnivore-app/fix/push-notification ... fix/push notification	2023-07-26 12:36:13 +08:00
Hongbo Wu	2841783504	limit max log entry size to 256kb	2023-07-25 23:39:03 +08:00
Hongbo Wu	f5ae16a904	disable gql request log	2023-07-25 23:03:57 +08:00
Hongbo Wu	ecd6a2922f	disable sentry error handler	2023-07-25 22:32:00 +08:00
Hongbo Wu	fd6223ee93	revert changes	2023-07-25 21:20:26 +08:00
Hongbo Wu	a3e86a9bc1	only log error.message	2023-07-25 20:52:15 +08:00
Hongbo Wu	0e38060b0c	set maxEntrySize for logger to 256KB	2023-07-25 20:07:02 +08:00
Hongbo Wu	1006cdad36	reduce error log size	2023-07-25 19:11:08 +08:00
Hongbo Wu	de647e44bd	Revert "move sentry error handler after apollo middleware" ... This reverts commit 8d8d2b592c.	2023-07-25 19:10:12 +08:00
Hongbo Wu	8d8d2b592c	move sentry error handler after apollo middleware	2023-07-25 18:18:40 +08:00
Hongbo Wu	77456373bf	return content without highlight if it takes too long to process highlighted content ... * timeout if the get text node function takes more than 10 minutes	2023-07-25 11:19:06 +08:00
Jackson Harper	fa1ff9ba17	Upgrade node to 18.16	2023-07-24 15:55:11 +08:00
Hongbo Wu	0d3b742837	limit 20k articles per pocket import	2023-07-21 18:03:57 +08:00
Hongbo Wu	e0a6a2e0a4	limit number of rss subscriptions to 20	2023-07-21 15:28:20 +08:00
Hongbo Wu	99b12c6cfb	debounce send push notificaiton within 1 minute	2023-07-20 21:48:30 +08:00
Hongbo Wu	9a224d2324	add event_types and fix the validation of form on rules page	2023-07-20 18:33:30 +08:00
Hongbo Wu	4dad4a6fa3	revert push notification	2023-07-20 17:29:26 +08:00
Hongbo Wu	6004b51956	return not found if rss feed not found	2023-07-20 14:21:36 +08:00
Hongbo Wu	7ceb96a93e	fix edit rss ui	2023-07-20 13:35:43 +08:00
Hongbo Wu	400739f61c	stop writing empty csv file if no new data retrieved from the integration	2023-07-20 10:32:46 +08:00
Hongbo Wu	6ef74a2601	disable import button while importer is running	2023-07-20 09:52:23 +08:00
Hongbo Wu	cd440a77c3	send last fetched at as a unix timestamp in ms	2023-07-19 14:09:38 +08:00
Hongbo Wu	35c074e647	create a cloud task to fetch rss feed item for the new subscription	2023-07-19 10:07:08 +08:00
Hongbo Wu	dcb306bcfc	do not include archived items in library or subscription	2023-07-18 16:45:24 +08:00
Hongbo Wu	456a73f4ab	make library an internal label	2023-07-17 18:29:35 +08:00
Hongbo Wu	44c9bfce48	Merge pull request #2494 from omnivore-app/fix/rss-feed ... fix/rss feed	2023-07-17 17:50:00 +08:00
Hongbo Wu	3cfd390346	Merge pull request #2489 from omnivore-app/more-search-scope ... add in:subscription and in:library in search scope	2023-07-17 17:28:24 +08:00
Hongbo Wu	03ce4b00d2	add authorization header in rss feed fetching request	2023-07-17 17:27:12 +08:00
Hongbo Wu	be1e8169ba	exclude items with newsletter or rss labels in library	2023-07-17 17:06:39 +08:00
Hongbo Wu	113720caf6	wrap document fragment in the <html> and <body> in order to parse the HTML correctly	2023-07-17 16:14:59 +08:00
Hongbo Wu	b3dcf0ee46	default return builder	2023-07-17 16:14:03 +08:00
Hongbo Wu	e3af3f526c	add in:subscription and in:library in search scope	2023-07-14 17:07:25 +08:00
Hongbo Wu	153662ce6a	set savedAt and publishedAt timestamp for rss item	2023-07-14 12:21:34 +08:00
Hongbo Wu	983f0dcb99	add rss to the search scope	2023-07-13 11:09:13 +08:00
Hongbo Wu	9697ae2eca	make rss an internal label	2023-07-13 10:58:37 +08:00
Hongbo Wu	87275fbb9c	use a separate queue for fetching rss feed item content	2023-07-13 10:27:15 +08:00
Hongbo Wu	3444480b95	change the rss-feed cronjob api endpoint	2023-07-12 10:53:32 +08:00
Hongbo Wu	435fa0c541	Merge pull request #2478 from omnivore-app/fix/puppeteer-timezone ... add optional locale and timezone to saveUrl request for puppeteer to use	2023-07-11 17:32:10 +08:00
Hongbo Wu	1f283e6122	add optional locale and timezone to saveUrl request for puppeteer to use	2023-07-11 16:15:32 +08:00
Hongbo Wu	b1b4a2e3fd	save rss feed url in the index	2023-07-11 13:14:43 +08:00
Hongbo Wu	c500997693	add lastFetchedAt to the cloud task payload	2023-07-11 13:05:27 +08:00
Hongbo Wu	eb9a3eddd0	skip old item	2023-07-11 12:59:55 +08:00
Hongbo Wu	44473ba089	upload feed subscriptions in cloud storage in cronjob	2023-07-11 12:22:49 +08:00
Hongbo Wu	56a27878dc	add updateSubscription api	2023-07-10 15:24:33 +08:00
Hongbo Wu	68667053c8	create an endpoint to fetch all rss feeds	2023-07-07 21:15:42 +08:00
Hongbo Wu	574636451d	update api to accommodate rss feed subscriptions	2023-07-06 19:51:30 +08:00
Hongbo Wu	7075375e2b	fix test	2023-07-06 19:51:30 +08:00
Hongbo Wu	2adf753f8f	update graphql schema	2023-07-06 19:51:30 +08:00
Hongbo Wu	a6192ed86b	add rss subscription entity class	2023-07-06 19:51:30 +08:00
Jackson Harper	0c96172053	Merge pull request #2458 from omnivore-app/feature/search-filter-cat ... feat: add category field to Filter object	2023-07-05 10:41:43 -07:00
Hongbo Wu	599d61016e	feat: add category field to Filter object	2023-07-03 19:06:45 +08:00
Hongbo Wu	e4284d215c	feat: boost sitename score by 2 times in the basic search	2023-07-03 18:09:41 +08:00
Hongbo Wu	afaaf8af60	feat: allow searching by note in the basic search	2023-07-03 18:06:54 +08:00
Hongbo Wu	a413521e3b	Merge pull request #2445 from omnivore-app/feature/create-labels ... feat: create labels if not exist when setting labels	2023-07-03 16:54:15 +08:00
Hongbo Wu	88dc514d27	feat: allow recovering a soft deleted item through updatePage api	2023-06-30 18:39:43 +08:00
Hongbo Wu	ba35fa4447	feat: show soft deleted pages within 14 days by searching in:trash	2023-06-30 18:05:41 +08:00
Hongbo Wu	6be9afb619	feat: allow searching by note:	2023-06-30 16:02:49 +08:00
Hongbo Wu	53b854b69f	fix: improve full-text search result ... * always sort by score first * boost title score by 3 times * tie breaker is 0.3	2023-06-30 15:32:09 +08:00
Hongbo Wu	13afb7febd	feat: create labels if not exist when setting labels in the highlight	2023-06-30 11:40:39 +08:00
Hongbo Wu	8d3cac989b	feat: create labels if not exist when setting labels in the page	2023-06-30 11:08:03 +08:00
Hongbo Wu	74f2dd6bf1	catch authorization error in rate limiter and 5 rpm	2023-06-28 17:04:47 +08:00
Hongbo Wu	b7c133d58e	fix: email sending and recommendation ... * use token in the custom header as the key in rate limiter	2023-06-28 17:00:11 +08:00
Hongbo Wu	b60ace4877	log unauthorized error	2023-06-28 12:22:32 +08:00
Hongbo Wu	1b069c2696	catch authorization error and return 401	2023-06-28 12:19:47 +08:00
Hongbo Wu	af7350bd31	fix: importer status update email not sent ... * create omnivore-authorization header to contain the token	2023-06-28 11:00:40 +08:00
Hongbo Wu	7c0c4eaa40	fix: type error	2023-06-27 21:05:00 +08:00
Hongbo Wu	ed8287df19	remove tracking params from tweet url	2023-06-27 17:49:31 +08:00
Hongbo Wu	bb359d9747	save normalized url	2023-06-27 16:58:57 +08:00
Hongbo Wu	f372636d6e	fix: tweet not saved correctly when using share button on iOS	2023-06-27 16:47:36 +08:00
Hongbo Wu	90444788ba	Merge pull request #2400 from omnivore-app/feat/web-multi-select-labels ... Add support for multi select Add Label	2023-06-22 09:42:59 +08:00
Jackson Harper	cceff32e5f	Return non-error if setting labels is a noop	2023-06-21 20:24:36 +08:00
Hongbo Wu	826565a2c4	temporarily skip unsubscribing by url and mark them as unsubscribed automatically	2023-06-21 12:13:22 +08:00
Hongbo Wu	4f74b32ff6	do not subscribe a newsletter if subscription already exists and is unsubscribed	2023-06-21 12:02:15 +08:00
Hongbo Wu	84788e7551	fix: mark failed to unsubscribe newsletters as unsubscribed state	2023-06-21 11:33:49 +08:00
Hongbo Wu	ea6d06f464	Merge pull request #2375 from omnivore-app/fix/pocket-importer ... fix pocket paginated api	2023-06-19 11:19:31 +08:00
Hongbo Wu	0d8a3f7346	fix pocket paginated api	2023-06-16 18:11:31 +08:00
Hongbo Wu	c8e3f79655	Merge pull request #2368 from omnivore-app/fix/api-robots-txt ... Add robots.txt to the list of reserved names	2023-06-16 16:38:06 +08:00
Jackson Harper	cf7c50468e	Use normalized URLs when querying article saving requests ... When we save pages we save the normalized URL, so if a user saves a URL like https://foo.com/ in elastic it will be saved as https://foo.com but the browser will redirect to this url: https://omnivore.app/article?url=https://foo.com/ and the URL lookup will fail.	2023-06-16 15:15:24 +08:00
Jackson Harper	5dd28de9e8	Add security.txt file too	2023-06-16 12:54:30 +08:00
Jackson Harper	52af2d8cb4	Add robots.txt to the list of reserved names	2023-06-16 12:54:30 +08:00
Hongbo Wu	9a3673e315	Merge pull request #2367 from omnivore-app/fix/refresh-index ... refresh index after altering the page state or adding/removing/updating labels/highlights	2023-06-15 12:18:53 +08:00
Hongbo Wu	ceaa7b0356	refresh index after altering the page state or adding/removing/updating labels/highlights	2023-06-15 11:37:09 +08:00
Hongbo Wu	5d2ab3f554	create another rate limiter for auth api with 4 rpm	2023-06-14 19:31:18 +08:00
Hongbo Wu	fc08a434e1	Merge pull request #2359 from omnivore-app/fix/pocket-import ... fix/pocket import	2023-06-14 15:32:10 +08:00
Jackson Harper	aaf6bb617f	Remove backend events that we dont need to watch	2023-06-14 13:19:39 +08:00
Hongbo Wu	11cf9f7c28	stringify csv before uploading	2023-06-14 12:50:06 +08:00
Hongbo Wu	d851a705a6	refresh index after archiving page	2023-06-13 12:44:32 +08:00
Hongbo Wu	771114b30d	refresh index if update is not async	2023-06-13 12:43:18 +08:00
Hongbo Wu	1a48ad03e4	use a separate queue for thumbnailer	2023-06-12 12:25:04 +08:00
Hongbo Wu	8904247620	add test	2023-06-09 18:58:41 +08:00
Hongbo Wu	b12e127d15	use gold color for the label	2023-06-09 17:36:38 +08:00
Hongbo Wu	5db4354f3f	make internal optional in label response	2023-06-09 17:33:14 +08:00
Hongbo Wu	bd4204fc42	add analytics	2023-06-09 13:14:14 +08:00
Hongbo Wu	55904ccf04	feat: add set favorite article api which adds the favorites internal label to it	2023-06-09 13:09:47 +08:00
Hongbo Wu	6116746e80	feat: make newsletters and favorites internal labels	2023-06-09 12:48:46 +08:00
Hongbo Wu	51032d2bf1	fix: test	2023-06-08 18:52:47 +08:00
Hongbo Wu	039f00e3b2	perf: add max doc and throttling to bulk api	2023-06-08 18:36:09 +08:00
Hongbo Wu	4678d6f473	check label uniqueness by name	2023-06-08 13:33:59 +08:00
Hongbo Wu	3630d743d7	add unique labels	2023-06-08 13:29:05 +08:00
Hongbo Wu	de3623d3f2	allow mark as read and add labels bulk actions	2023-06-08 13:05:30 +08:00
Hongbo Wu	1af845e4e5	feat: add bulk action api schema	2023-06-07 21:50:43 +08:00
Hongbo Wu	b698029972	store token in cookie instead of authorization header	2023-06-07 14:24:52 +08:00
Hongbo Wu	c0872939a7	Merge pull request #2311 from omnivore-app/feature/metadata ... Feature: Set thumbnail and pre-cache all the images in the content	2023-06-07 12:08:20 +08:00
Hongbo Wu	9be84fb473	create thumbnail task for newsletters too	2023-06-07 11:52:13 +08:00
Hongbo Wu	986c4eb48b	fix: parse unsubscribe mailto email address and set subject	2023-06-06 19:53:18 +08:00
Hongbo Wu	4f4f50af49	fix: redundant argument	2023-06-06 12:05:33 +08:00
Hongbo Wu	0c3fa2cdd4	feat: start a cloud task to set thumbnail and pre-cache all the images in the content	2023-06-06 11:56:31 +08:00
Hongbo Wu	e2315b5662	feat: add thumbnail handler	2023-06-06 11:02:46 +08:00
Hongbo Wu	c66d13177f	Merge pull request #2295 from omnivore-app/fix/twitter-handler-timeout ... Feature: importer metrics collector and fix puppeteer timeout issue	2023-06-05 17:17:50 +08:00