2536 commits

Author	SHA1	Message	Date
Hongbo Wu	6618a6ed19	migrate popular reads into database	2023-10-20 14:31:16 +08:00
Rudra	23eb7ea76f	fix typo	2023-10-20 11:51:02 +05:30
Jackson Harper	a23b67b70b	Dont rate limit auth in local env so tests can run	2023-10-20 11:44:52 +08:00
Jackson Harper	b9e2f9ee53	Add a longer rate limit window on createaccount/reset password, reduce api rate limit hits	2023-10-20 11:28:16 +08:00
Hongbo Wu	4e36a5809e	fix failure of re-subscribing rss feed	2023-10-20 11:05:58 +08:00
Hongbo Wu	750ccb0c42	fix conflicts with main	2023-10-19 17:44:53 +08:00
Hongbo Wu	1880f2ace4	Update rss handler to fetch a group of rss feeds in one task	2023-10-19 17:27:55 +08:00
Hongbo Wu	f50d1c9cb9	Group RSS subscriptions by feed URL and create cloud task for the groups	2023-10-19 17:27:08 +08:00
Hongbo Wu	5283a84908	add scheduledAt to the task payload	2023-10-19 17:27:08 +08:00
Hongbo Wu	18aece84e7	add scheduledAt to subscriptions	2023-10-19 17:27:08 +08:00
Jackson Harper	c93e61ec24	Remove debug	2023-10-18 21:30:23 +08:00
Jackson Harper	78a642db6d	Update type signature	2023-10-18 19:53:26 +08:00
Jackson Harper	5c576347a8	MOre debug	2023-10-18 19:26:45 +08:00
Jackson Harper	fbaaaeca7c	Add some debugging	2023-10-18 18:48:18 +08:00
Jackson Harper	7848413645	Update schema	2023-10-18 16:34:20 +08:00
Jackson Harper	cf101c6d18	Cache and check feed checksums to reduce fetching	2023-10-18 16:26:03 +08:00
Hongbo Wu	46313b14ba	set trust proxy = true if set in env var	2023-10-18 13:06:33 +08:00
Hongbo Wu	988b20e30c	test trust proxy=true	2023-10-18 12:30:27 +08:00
Hongbo Wu	506004d410	test trust proxy=2	2023-10-18 11:58:15 +08:00
Hongbo Wu	32ba744d20	test trust proxy=1	2023-10-18 11:04:32 +08:00
Hongbo Wu	91076014cd	log req.ip	2023-10-18 11:04:32 +08:00
Hongbo Wu	0874f6ff21	log x-forwarded-for header for debugging purpose	2023-10-18 11:04:32 +08:00
Jackson Harper	71aaef3628	Update attachment URL	2023-10-18 10:43:53 +08:00
Jackson Harper	b88c64424f	Special handling of file URLs	2023-10-18 10:37:21 +08:00
Jackson Harper	78843fff14	Remove import	2023-10-18 10:27:09 +08:00
Jackson Harper	4220922ff8	Preserve original URL if possible	2023-10-18 10:23:26 +08:00
Jackson Harper	b6ce4f39b1	Fix imports	2023-10-18 10:15:49 +08:00
Jackson Harper	25c0051fd4	Only use signed URLs for PDF attachments	2023-10-18 10:14:35 +08:00
Hongbo Wu	fa2fba1a2c	Merge pull request #2942 from omnivore-app/fix/rules ... improvement on subscriptions and delete user api	2023-10-17 17:14:49 +08:00
Hongbo Wu	700fd73dbb	disallow social login if account is deleted	2023-10-17 17:07:46 +08:00
Hongbo Wu	6e79f6c67f	update sql to deduplicate rss subscriptions	2023-10-17 16:41:14 +08:00
Hongbo Wu	eb667e04b9	update reading progress in db	2023-10-17 15:09:16 +08:00
Hongbo Wu	1b2d93e118	remove duplicate rss subscriptions by user and url; create a unique constraint	2023-10-16 17:35:29 +08:00
Jackson Harper	9e8ccb35e3	Limit users to a single import a day	2023-10-16 15:48:03 +08:00
Hongbo Wu	e54c1c81a1	soft delete user by calling delete account api	2023-10-16 15:24:53 +08:00
Jackson Harper	34798f39f0	Disable JSONLD fetching which can be quite slow	2023-10-16 11:25:37 +08:00
Jackson Harper	a20b956e26	Merge pull request #2926 from omnivore-app/fix/api-site-name-query ... Add ILIKE queries to all field queries	2023-10-13 22:02:55 +08:00
Hongbo Wu	497b6e9f72	fetch highlights if necessary and before converting content to markdown	2023-10-13 18:27:57 +08:00
Jackson Harper	3f4be32df8	Merge library item imports	2023-10-13 18:26:02 +08:00
Hongbo Wu	a6c5f4e862	fix bulk action slow query	2023-10-13 13:45:09 +08:00
Hongbo Wu	4d64231abf	save content reader in the item	2023-10-12 12:59:36 +08:00
Hongbo Wu	d1472c6519	Merge pull request #2915 from omnivore-app/fix/drop-label-position-trigger ... drop position trigger on labels table and sort labels by name returned by labels api	2023-10-12 12:10:33 +08:00
Hongbo Wu	9b25a64fd8	Merge pull request #2916 from omnivore-app/fix/email-attachment ... fix email attachment not being saved as item	2023-10-12 12:09:01 +08:00
Hongbo Wu	039632556b	add more method in mock cloud storage and update test cases for email-attachment rest api	2023-10-12 11:56:54 +08:00
Hongbo Wu	1bc9912271	fix failure of saving highlight when position is null	2023-10-12 11:27:46 +08:00
Hongbo Wu	ebb45f1973	fix email attachment not being saved as item	2023-10-12 10:47:20 +08:00
Hongbo Wu	f1c4bc6a74	fix email attachment not being saved as item	2023-10-12 10:46:12 +08:00
Hongbo Wu	b0f0dba53d	drop position trigger on labels table and sort labels by name returned by labels api	2023-10-12 10:39:35 +08:00
Hongbo Wu	30ba22cbbc	add user_id to the sql query to reduce query latency	2023-10-11 16:05:35 +08:00
Hongbo Wu	a4aeb40276	fix: after successfully deleted highlights, still return error codes	2023-10-11 14:48:44 +08:00
Hongbo Wu	768845e101	update the filter value to use is:reading of the Continue Reading filter	2023-10-10 15:33:59 +08:00
Hongbo Wu	62ffaa924b	add wordsCount filter	2023-10-10 12:52:49 +08:00
Hongbo Wu	93c410a564	fix is:read filter and add is:reading filter	2023-10-10 12:21:32 +08:00
Hongbo Wu	3e91f3836b	fix tests	2023-10-10 10:58:48 +08:00
Hongbo Wu	835b61064f	allow filter by wordsCount	2023-10-10 10:58:48 +08:00
Hongbo Wu	1adbed9876	revert postgres connection timeout	2023-10-10 10:33:49 +08:00
Hongbo Wu	bab343c41f	testing	2023-10-10 10:10:45 +08:00
Hongbo Wu	ac0d295cf3	debug	2023-10-09 22:06:35 +08:00
Hongbo Wu	0b7da62aca	debug	2023-10-09 21:34:52 +08:00
Hongbo Wu	f4f38e4b31	reduce unnecessary logs	2023-10-09 17:26:03 +08:00
Hongbo Wu	78490e15f1	disable db connection timeout	2023-10-09 15:30:30 +08:00
Hongbo Wu	50901d25a3	remove elastic url from env var	2023-10-09 13:00:30 +08:00
Hongbo Wu	e424328f20	fix recommendations not fetched in search results	2023-10-05 23:00:29 +08:00
Hongbo Wu	ac576aa5c7	fix isArchived = false when archivedAt is not null	2023-10-05 17:53:57 +08:00
Hongbo Wu	39b3a1cad4	resolve conflicts	2023-10-05 14:37:21 +08:00
Hongbo Wu	463231b26b	fix trigger	2023-10-05 14:32:05 +08:00
Hongbo Wu	0e7f45a758	create a trigger to update recommender_names in library_item table	2023-10-05 14:32:05 +08:00
Hongbo Wu	f6c906201b	fetch recommendations separately too	2023-10-05 14:32:05 +08:00
Hongbo Wu	d4404b0248	fetch labels and highlights after items fetched in a search query	2023-10-05 14:32:05 +08:00
Hongbo Wu	bb59ed757c	fix word count not return	2023-10-05 14:32:05 +08:00
Hongbo Wu	cbf58614c4	use log levels in custom typeorm logger	2023-10-05 14:32:05 +08:00
Hongbo Wu	efdbc4f345	use interpolated string or dictionary parameter in logs	2023-10-05 14:32:03 +08:00
Hongbo Wu	327f6959ef	fix highlight createdByMe is wrong	2023-10-05 14:31:07 +08:00
Hongbo Wu	0ff9fdef04	add elastic scroll time in the env var	2023-10-05 14:31:07 +08:00
Hongbo Wu	0912e7cf0f	remove warn logs	2023-10-05 14:31:07 +08:00
Hongbo Wu	3b8b48bc21	fix recovered deleted items still shown in trash	2023-10-05 14:31:07 +08:00
Hongbo Wu	7bf3f1bbf4	set deletedAt to now when bulk delete	2023-10-05 14:31:07 +08:00
Hongbo Wu	47510c0785	fix recommendations not fetched in search results	2023-10-05 14:31:07 +08:00
Hongbo Wu	9c723ecbd3	fix id not set in page update pubsub message	2023-10-05 14:31:07 +08:00
Hongbo Wu	0c691257eb	fix rules permission	2023-10-05 14:31:07 +08:00
Hongbo Wu	5f6587537e	fix recommendation wildcard search	2023-10-05 14:31:07 +08:00
Hongbo Wu	bf2c54084b	fix getArticle API failed on iOS	2023-10-05 14:31:07 +08:00
Hongbo Wu	9e5196406e	revert rls on old tables	2023-10-05 14:31:07 +08:00
Hongbo Wu	2e5529fd81	allow readingProgressAnchorIndex to be null	2023-10-05 14:31:07 +08:00
Hongbo Wu	8712107a2f	remove db caching	2023-10-05 14:31:07 +08:00
Hongbo Wu	c57ead3583	add test for when rss:feed in:archive is in the query	2023-10-05 14:31:07 +08:00
Hongbo Wu	867c31a582	add test for searching in:archive label:test	2023-10-05 14:31:07 +08:00
Hongbo Wu	6cb4bb8de8	fix in:subscription search	2023-10-05 14:31:07 +08:00
Hongbo Wu	15f9017291	fix in:archive search	2023-10-05 14:31:07 +08:00
Hongbo Wu	a35e6c0147	fix sort by read time bug	2023-10-05 14:31:07 +08:00
Hongbo Wu	dc98719b85	reduce log	2023-10-05 14:31:07 +08:00
Hongbo Wu	0c5ba53c26	fix recommendations	2023-10-05 14:31:07 +08:00
Hongbo Wu	b9af543b45	fix rss feeds	2023-10-05 14:31:06 +08:00
Hongbo Wu	673ce83c33	fix searching for site	2023-10-05 14:31:06 +08:00
Hongbo Wu	4ff5484d8e	change item_type to text	2023-10-05 14:31:06 +08:00
Hongbo Wu	7c56973fad	fix update filter api	2023-10-05 14:31:06 +08:00
Hongbo Wu	e696709a69	resolve conflicts	2023-10-05 14:31:06 +08:00
Hongbo Wu	7225d861f3	use skip and take instead of limit, offset	2023-10-05 14:31:06 +08:00
Hongbo Wu	68e381deb2	revert note changes	2023-10-05 14:31:06 +08:00
Hongbo Wu	fdd771bc06	add migration script	2023-10-05 14:31:06 +08:00
Hongbo Wu	dd292879a4	sort type ahead search result by saved at desc order	2023-10-05 14:31:06 +08:00
Hongbo Wu	2b9a6b26f6	debug test error	2023-10-05 14:30:37 +08:00
Hongbo Wu	a284884398	fix rebase conflicts	2023-10-05 14:30:37 +08:00
Hongbo Wu	5a9dc344d3	fix rebase conflicts	2023-10-05 14:30:35 +08:00
Hongbo Wu	cbaa3d6900	fix newsletters test	2023-10-05 14:30:10 +08:00
Hongbo Wu	c3caf15e44	fix auth tests	2023-10-05 14:30:10 +08:00
Hongbo Wu	024d558cfe	update labels rls	2023-10-05 14:30:10 +08:00
Hongbo Wu	6672ca0109	fix labels tests	2023-10-05 14:30:10 +08:00
Hongbo Wu	4b95b732cf	fix highlight tests	2023-10-05 14:30:10 +08:00
Hongbo Wu	b036149fe2	fix feature tests	2023-10-05 14:30:10 +08:00
Hongbo Wu	d50174ef76	fix article tests	2023-10-05 14:30:10 +08:00
Hongbo Wu	669cddeec8	fix some rls issue	2023-10-05 14:30:10 +08:00
Hongbo Wu	cea5f790d8	fix some tests	2023-10-05 14:30:09 +08:00
Hongbo Wu	01c74cb7c4	fix tests for popular reads	2023-10-05 14:29:25 +08:00
Hongbo Wu	50bfc1fa74	fix tests	2023-10-05 14:29:23 +08:00
Hongbo Wu	bab957256d	fix multiple parameters in query builder	2023-10-05 14:28:52 +08:00
Hongbo Wu	e1a7a24323	improve search result	2023-10-05 14:28:52 +08:00
Hongbo Wu	1bade22076	fix file uploading	2023-10-05 14:28:52 +08:00
Hongbo Wu	e05d341952	fix file uploading	2023-10-05 14:28:52 +08:00
Hongbo Wu	86cff913fb	fix label search	2023-10-05 14:28:52 +08:00
Hongbo Wu	0747c4603c	fix bulk action	2023-10-05 14:28:52 +08:00
Hongbo Wu	afa1165a29	fix setting labels	2023-10-05 14:28:52 +08:00
Hongbo Wu	33ba17707b	fix setting labels	2023-10-05 14:28:52 +08:00
Hongbo Wu	0016b0a9a7	fix update library item	2023-10-05 14:28:52 +08:00
Hongbo Wu	d9d71aa443	fix highlights api	2023-10-05 14:28:52 +08:00
Hongbo Wu	5ed7e819e8	fix label constraint	2023-10-05 14:28:52 +08:00
Hongbo Wu	b72acf8e6f	fix permission issue	2023-10-05 14:28:52 +08:00
Hongbo Wu	4a55e801b4	fix typo	2023-10-05 14:28:50 +08:00
Hongbo Wu	3f572ea89a	fix typo	2023-10-05 14:27:52 +08:00
Hongbo Wu	65bb4aa85f	remove knex	2023-10-05 14:27:18 +08:00
Hongbo Wu	c1ad9b6f41	cont	2023-10-05 14:27:18 +08:00
Hongbo Wu	f8f3ec0c56	save claim in httpContext	2023-10-05 14:27:17 +08:00
Hongbo Wu	b1899e340d	replace library item	2023-10-05 14:25:10 +08:00
Hongbo Wu	c655f78f8c	replace highlight	2023-10-05 14:24:28 +08:00
Hongbo Wu	381e9bc173	replace database subscriber with db trigger	2023-10-05 14:24:05 +08:00
Hongbo Wu	612151c151	remove elastic from docker-compose	2023-10-05 14:24:04 +08:00
Hongbo Wu	5dce78eedb	update entity subscriber	2023-10-05 14:22:58 +08:00
Hongbo Wu	648a16615b	update search query	2023-10-05 14:22:58 +08:00
Hongbo Wu	751df68175	implement search function	2023-10-05 14:22:58 +08:00
Hongbo Wu	0b309b7a45	add recommendation db table and entity	2023-10-05 14:22:58 +08:00
Hongbo Wu	5f0e1af285	update highlight entity	2023-10-05 14:22:58 +08:00
Hongbo Wu	06000264d2	replace saveEmail	2023-10-05 14:22:58 +08:00
Hongbo Wu	fc52b522f2	fix test	2023-10-05 14:22:58 +08:00
Hongbo Wu	9370951893	create a repository directory	2023-10-05 14:22:56 +08:00
Hongbo Wu	425864ce01	remove knex	2023-10-05 14:21:56 +08:00
Hongbo Wu	928ed2e6a9	add libraryItemPreview entity class	2023-10-05 14:18:21 +08:00
Hongbo Wu	e1af53ab48	add libraryItem entity class	2023-10-05 14:18:21 +08:00
Hongbo Wu	789cb1d501	Merge pull request #2860 from omnivore-app/fix/self-hosting-login ... fix demo user not able to login when self hosting	2023-10-05 12:02:46 +08:00
Hongbo Wu	c008c3154d	make reminder task url optional env var	2023-10-05 11:36:34 +08:00
Hongbo Wu	16f72d6a74	fix labels and highlights were deleted when deleting items	2023-10-05 10:03:06 +08:00
Hongbo Wu	418f27f586	fix convert to email login does not change source field in user table	2023-10-04 16:35:50 +08:00
Jackson Harper	c41363b51c	Merge pull request #2831 from omnivore-app/feature/convert-to-email-api ... Update Email Feature	2023-10-04 14:30:35 +08:00
Hongbo Wu	3cf0c45fa9	mark unsubscribed newsletter in the db	2023-10-03 12:50:55 +08:00
Hongbo Wu	ee50529f95	add update email form on account page	2023-10-02 22:21:50 +08:00
Hongbo Wu	c40e64773a	convert to email api	2023-10-02 17:01:13 +08:00
Hongbo Wu	082e83b1ad	fix 406 not acceptable error from subscribing an rss feed	2023-10-02 10:52:50 +08:00
Hongbo Wu	010e2841da	fix getting 403 when parsing some rss url by setting a user agent in the request	2023-09-27 10:38:49 +08:00
Jackson Harper	21244a394d	Move debug directory to not interfere with usernames	2023-09-22 17:56:25 +08:00
Jackson Harper	53ac43563d	Set user properties instead of event properties on labels setter	2023-09-19 12:30:20 +08:00
Jackson Harper	6c6f27be3e	Properly tag user on labels event	2023-09-14 16:31:01 +08:00
Jackson Harper	e4b5bd6a67	Add code to join users without email on login	2023-09-14 11:24:19 +08:00
Jackson Harper	014614d097	Merge pull request #2759 from omnivore-app/feat/web-use-posthog ... Remove segment from the web	2023-09-13 14:01:08 +08:00
Jackson Harper	dfcfe64b1b	Add a create_user event	2023-09-13 13:44:26 +08:00
Jackson Harper	0946f3d3e8	Fix some console logs, better handle posthog	2023-09-13 12:52:34 +08:00
Jackson Harper	842e4bbf98	Update help links	2023-09-13 10:22:45 +08:00
Jackson Harper	293becf596	Ignore link density checks in newsletters	2023-09-12 15:53:43 +08:00
Jackson Harper	2a3214e393	Remove identify call from backend	2023-09-12 09:53:19 +08:00
Jackson Harper	a5eb527ae1	Merge pull request #2738 from omnivore-app/fix/api-enable-rules-notifications ... Remove debounce from notification rule	2023-09-07 18:07:06 +08:00
Jackson Harper	facde7d9b3	Remove import	2023-09-07 17:51:20 +08:00
Jackson Harper	e94529a96c	Merge pull request #2707 from Podginator/feat/saved-links ... Add Saved Searches, the ability to add new searches, delete searches, and reorder searches.	2023-09-07 17:27:48 +08:00
Jackson Harper	edd7246a9b	Remove debounce from notification rule	2023-09-07 17:24:49 +08:00
Jackson Harper	d0583bf271	Disable persisted queries in Apollo	2023-09-07 14:49:48 +08:00
Jackson Harper	f7e5a8cbfd	Increase unauthed rate limit	2023-09-07 14:01:18 +08:00
Thomas Rogers	79f42a5bde	Merge Main	2023-09-06 17:54:18 +02:00
Thomas Rogers	13ab13d94d	Code Review Comments	2023-09-06 17:51:42 +02:00
Jackson Harper	d2534d692b	Better preview URLs for onboarding articles	2023-09-04 18:27:56 +08:00
Thomas Rogers	faf7880fcd	Fix some bugs regarding scrolling, fix some UI Issues, and correct the name of Non-feed Filter.	2023-08-31 20:08:54 +02:00
Thomas Rogers	a58864fe2c	Merge main	2023-08-30 19:48:04 +02:00
Thomas Rogers	d433d48365	Add Filters, and editing filters.	2023-08-30 19:37:24 +02:00
Jackson Harper	79cea907a3	Fix typo with RSS typed subscription queries	2023-08-30 14:18:39 +08:00
Jackson Harper	fe3d3be7f5	Make query more clear	2023-08-29 20:17:13 +08:00
Jackson Harper	6c1484cd09	Add type filter	2023-08-29 20:06:28 +08:00
Jackson Harper	3565c7000e	Update subscriptions query	2023-08-29 17:03:49 +08:00
Jackson Harper	8851017100	Use andWhere query to not overwrite the where	2023-08-29 15:54:07 +08:00
Jackson Harper	89c17fd0ab	Only filter inactive newsletters out from the subscriptions response	2023-08-29 11:34:55 +08:00
Jackson Harper	49496f7aa2	Only return active subscriptions	2023-08-29 10:19:57 +08:00
Jackson Harper	feaa539c15	Use and where for type query. Prettier changes	2023-08-28 15:23:14 +08:00
Jackson Harper	aa0f4399aa	Remove prettier changes	2023-08-28 15:15:05 +08:00
Jackson Harper	38c9bbdcaa	Allow getting all subscritons regardless of type at the same time	2023-08-28 15:11:16 +08:00
Hongbo Wu	ec574d0ad9	use new color or the color of the last overlap highlight when merging highlights	2023-08-18 13:54:38 +08:00
Hongbo Wu	ce480806c0	add color to create/update/merge highlight api and highlight schema	2023-08-18 13:30:54 +08:00
Hongbo Wu	cd1fc5c4db	fix typo	2023-08-17 18:46:39 +08:00
Hongbo Wu	c45b760f71	fix test	2023-08-17 18:42:11 +08:00
Hongbo Wu	e54e61f3c0	reduce log size	2023-08-17 18:33:14 +08:00
Hongbo Wu	dedd76cfb7	do not publish a pubsub event if the page is imported	2023-08-17 18:24:57 +08:00
Hongbo Wu	ec88a9fe8e	we don't want to create thumbnail for imported pages	2023-08-17 18:05:51 +08:00
Hongbo Wu	0609043956	reduce retry	2023-08-17 17:59:24 +08:00
Hongbo Wu	6f315ef4d9	create a transaction for uploading files in the upload_files resolver	2023-08-16 21:19:03 +08:00
Hongbo Wu	755fb66588	Merge pull request #2651 from omnivore-app/feature/csv-importer ... feature/csv importer	2023-08-15 14:09:36 +08:00
Hongbo Wu	9ad9cea917	raise rss limit to 150	2023-08-14 22:27:43 +08:00
Hongbo Wu	5f6be169bd	add savedAt and publishedAt to saveUrl api	2023-08-14 17:10:34 +08:00
Hongbo Wu	fb845152d9	add header to the pocket data csv file	2023-08-14 13:44:30 +08:00
Hongbo Wu	93cca5040e	add comment	2023-08-11 14:41:21 +08:00
Hongbo Wu	05e73ebc9c	reduce readwise api timeout to 5 sec	2023-08-11 14:38:45 +08:00
Hongbo Wu	b18018ab2e	reduce max content size to 5MB and 10MB for both content and originalHtml	2023-08-11 14:02:29 +08:00
Hongbo Wu	792cf0b207	upgrade google cloud sdk	2023-08-11 13:55:29 +08:00
Hongbo Wu	55841e595f	fix dependecies	2023-08-11 11:12:03 +08:00
Hongbo Wu	61f553fd09	Merge pull request #2616 from omnivore-app/fix/thumbnail-delay ... fix thumbnail not pre-cached if it was already set	2023-08-10 18:45:48 +08:00
Hongbo Wu	7bd747601b	update thumbnail if failed to fetch and return url if already signed	2023-08-10 18:41:10 +08:00
Hongbo Wu	f06d2558c2	update thumbnail if failed to fetch and return url if already signed	2023-08-10 18:41:03 +08:00
Hongbo Wu	8809c9764a	max content length = 10MB	2023-08-10 18:12:29 +08:00
Hongbo Wu	fc76897df3	timeout = 5 when syncing omnivore data to readwise	2023-08-10 15:23:03 +08:00
Hongbo Wu	d4ac069503	fix webhook logs	2023-08-10 15:07:00 +08:00
Hongbo Wu	6acedb64c8	fix maximum call stack size error	2023-08-10 14:18:29 +08:00
Hongbo Wu	7506e27dd0	create or update an opt-in record with null grantedAt if exceeding max users	2023-08-10 13:47:39 +08:00
Hongbo Wu	163dbc73a6	do not send content in the thumbnail task payload	2023-08-10 11:55:59 +08:00
Hongbo Wu	51e0dfde23	set new limit = 1500	2023-08-10 09:43:40 +08:00
Hongbo Wu	3df0ffc5d3	fix existing opted in users not being able to be granted with the feature after increasing the limit	2023-08-08 22:08:37 +08:00
Hongbo Wu	3570079824	fix thumbnail not pre-cached if it was already set	2023-08-07 18:20:59 +08:00
Hongbo Wu	55e378c9b5	Merge pull request #2572 from omnivore-app/feature/pause-rss-feed ... add a button to pause rss feed	2023-08-03 13:33:11 +08:00
Hongbo Wu	f4e537a4c7	set max redirect = 5 and timeout = 30 seconds	2023-08-03 13:31:18 +08:00
Hongbo Wu	82d820ea8f	get link following the order of preference: via, self, alternate	2023-08-03 13:05:26 +08:00
Hongbo Wu	f3994a8a06	fix test	2023-08-02 22:00:09 +08:00
Hongbo Wu	b403dea247	add a button to pause rss feed	2023-08-01 22:07:53 +08:00
Jackson Harper	3d6d421e84	Merge pull request #2566 from omnivore-app/feat/web-improved-cards ... WIP: grid card design	2023-08-01 20:34:39 +08:00
Jackson Harper	c0dbbc3b86	Update icons	2023-08-01 12:31:12 +08:00
Hongbo Wu	00e7e49a4b	increase the max number of rss subscriptions to 50	2023-07-31 21:44:50 +08:00
Hongbo Wu	d0ad636b7f	use safe stringify in logs	2023-07-28 16:52:28 +08:00
Hongbo Wu	f20e12311a	check if the page is from rss feed	2023-07-28 13:53:19 +08:00
Hongbo Wu	9bc7ed92c0	we don't want to update an rss feed page if rss-feeder is tring to re-save it	2023-07-28 12:43:04 +08:00
Hongbo Wu	2e965afd69	reformat sql query in logs	2023-07-27 17:34:12 +08:00
Hongbo Wu	99a52f8d56	replace all the console logs with logger logs	2023-07-27 16:06:44 +08:00
Hongbo Wu	808f6f3d3a	remove trace	2023-07-26 22:00:37 +08:00
Jackson Harper	5bf3213e49	Dont send newsletter notifications	2023-07-26 20:53:50 +08:00
Hongbo Wu	61967fdde5	fix test	2023-07-26 18:07:25 +08:00
Hongbo Wu	ced1a711eb	fix test	2023-07-26 17:12:40 +08:00
Hongbo Wu	7b36156ce6	add a trace when log size too big	2023-07-26 17:07:24 +08:00
Hongbo Wu	8cd47eec11	run prettier	2023-07-26 16:41:55 +08:00
Hongbo Wu	07d43812a7	use winston logger instead of console log so the log will be truncated if it exceeds the max entry size	2023-07-26 16:27:18 +08:00
Hongbo Wu	66490df72f	deeply truncate log object size	2023-07-26 15:52:13 +08:00
Hongbo Wu	f0e62b1d2a	use lodash instead of standalone lodash.debounce package	2023-07-26 13:41:51 +08:00
Hongbo Wu	0a227decdd	Merge pull request #2515 from omnivore-app/fix/push-notification ... fix/push notification	2023-07-26 12:36:13 +08:00
Hongbo Wu	2841783504	limit max log entry size to 256kb	2023-07-25 23:39:03 +08:00
Hongbo Wu	f5ae16a904	disable gql request log	2023-07-25 23:03:57 +08:00
Hongbo Wu	ecd6a2922f	disable sentry error handler	2023-07-25 22:32:00 +08:00
Hongbo Wu	fd6223ee93	revert changes	2023-07-25 21:20:26 +08:00
Hongbo Wu	a3e86a9bc1	only log error.message	2023-07-25 20:52:15 +08:00
Hongbo Wu	0e38060b0c	set maxEntrySize for logger to 256KB	2023-07-25 20:07:02 +08:00
Hongbo Wu	1006cdad36	reduce error log size	2023-07-25 19:11:08 +08:00
Hongbo Wu	de647e44bd	Revert "move sentry error handler after apollo middleware" ... This reverts commit 8d8d2b592c.	2023-07-25 19:10:12 +08:00
Hongbo Wu	8d8d2b592c	move sentry error handler after apollo middleware	2023-07-25 18:18:40 +08:00
Hongbo Wu	77456373bf	return content without highlight if it takes too long to process highlighted content ... * timeout if the get text node function takes more than 10 minutes	2023-07-25 11:19:06 +08:00
Jackson Harper	fa1ff9ba17	Upgrade node to 18.16	2023-07-24 15:55:11 +08:00
Hongbo Wu	0d3b742837	limit 20k articles per pocket import	2023-07-21 18:03:57 +08:00
Hongbo Wu	e0a6a2e0a4	limit number of rss subscriptions to 20	2023-07-21 15:28:20 +08:00
Hongbo Wu	99b12c6cfb	debounce send push notificaiton within 1 minute	2023-07-20 21:48:30 +08:00
Hongbo Wu	9a224d2324	add event_types and fix the validation of form on rules page	2023-07-20 18:33:30 +08:00
Hongbo Wu	4dad4a6fa3	revert push notification	2023-07-20 17:29:26 +08:00
Hongbo Wu	6004b51956	return not found if rss feed not found	2023-07-20 14:21:36 +08:00
Hongbo Wu	7ceb96a93e	fix edit rss ui	2023-07-20 13:35:43 +08:00
Hongbo Wu	400739f61c	stop writing empty csv file if no new data retrieved from the integration	2023-07-20 10:32:46 +08:00
Hongbo Wu	6ef74a2601	disable import button while importer is running	2023-07-20 09:52:23 +08:00
Hongbo Wu	cd440a77c3	send last fetched at as a unix timestamp in ms	2023-07-19 14:09:38 +08:00
Hongbo Wu	35c074e647	create a cloud task to fetch rss feed item for the new subscription	2023-07-19 10:07:08 +08:00
Hongbo Wu	dcb306bcfc	do not include archived items in library or subscription	2023-07-18 16:45:24 +08:00
Hongbo Wu	456a73f4ab	make library an internal label	2023-07-17 18:29:35 +08:00
Hongbo Wu	44c9bfce48	Merge pull request #2494 from omnivore-app/fix/rss-feed ... fix/rss feed	2023-07-17 17:50:00 +08:00
Hongbo Wu	3cfd390346	Merge pull request #2489 from omnivore-app/more-search-scope ... add in:subscription and in:library in search scope	2023-07-17 17:28:24 +08:00
Hongbo Wu	03ce4b00d2	add authorization header in rss feed fetching request	2023-07-17 17:27:12 +08:00
Hongbo Wu	be1e8169ba	exclude items with newsletter or rss labels in library	2023-07-17 17:06:39 +08:00
Hongbo Wu	113720caf6	wrap document fragment in the <html> and <body> in order to parse the HTML correctly	2023-07-17 16:14:59 +08:00
Hongbo Wu	b3dcf0ee46	default return builder	2023-07-17 16:14:03 +08:00
Hongbo Wu	e3af3f526c	add in:subscription and in:library in search scope	2023-07-14 17:07:25 +08:00
Hongbo Wu	153662ce6a	set savedAt and publishedAt timestamp for rss item	2023-07-14 12:21:34 +08:00
Hongbo Wu	983f0dcb99	add rss to the search scope	2023-07-13 11:09:13 +08:00
Hongbo Wu	9697ae2eca	make rss an internal label	2023-07-13 10:58:37 +08:00
Hongbo Wu	87275fbb9c	use a separate queue for fetching rss feed item content	2023-07-13 10:27:15 +08:00
Hongbo Wu	3444480b95	change the rss-feed cronjob api endpoint	2023-07-12 10:53:32 +08:00
Hongbo Wu	435fa0c541	Merge pull request #2478 from omnivore-app/fix/puppeteer-timezone ... add optional locale and timezone to saveUrl request for puppeteer to use	2023-07-11 17:32:10 +08:00
Hongbo Wu	1f283e6122	add optional locale and timezone to saveUrl request for puppeteer to use	2023-07-11 16:15:32 +08:00
Hongbo Wu	b1b4a2e3fd	save rss feed url in the index	2023-07-11 13:14:43 +08:00
Hongbo Wu	c500997693	add lastFetchedAt to the cloud task payload	2023-07-11 13:05:27 +08:00
Hongbo Wu	eb9a3eddd0	skip old item	2023-07-11 12:59:55 +08:00
Hongbo Wu	44473ba089	upload feed subscriptions in cloud storage in cronjob	2023-07-11 12:22:49 +08:00
Hongbo Wu	56a27878dc	add updateSubscription api	2023-07-10 15:24:33 +08:00
Hongbo Wu	68667053c8	create an endpoint to fetch all rss feeds	2023-07-07 21:15:42 +08:00
Hongbo Wu	574636451d	update api to accommodate rss feed subscriptions	2023-07-06 19:51:30 +08:00
Hongbo Wu	7075375e2b	fix test	2023-07-06 19:51:30 +08:00
Hongbo Wu	2adf753f8f	update graphql schema	2023-07-06 19:51:30 +08:00
Hongbo Wu	a6192ed86b	add rss subscription entity class	2023-07-06 19:51:30 +08:00
Jackson Harper	0c96172053	Merge pull request #2458 from omnivore-app/feature/search-filter-cat ... feat: add category field to Filter object	2023-07-05 10:41:43 -07:00
Hongbo Wu	599d61016e	feat: add category field to Filter object	2023-07-03 19:06:45 +08:00
Hongbo Wu	e4284d215c	feat: boost sitename score by 2 times in the basic search	2023-07-03 18:09:41 +08:00
Hongbo Wu	afaaf8af60	feat: allow searching by note in the basic search	2023-07-03 18:06:54 +08:00
Hongbo Wu	a413521e3b	Merge pull request #2445 from omnivore-app/feature/create-labels ... feat: create labels if not exist when setting labels	2023-07-03 16:54:15 +08:00
Hongbo Wu	88dc514d27	feat: allow recovering a soft deleted item through updatePage api	2023-06-30 18:39:43 +08:00
Hongbo Wu	ba35fa4447	feat: show soft deleted pages within 14 days by searching in:trash	2023-06-30 18:05:41 +08:00
Hongbo Wu	6be9afb619	feat: allow searching by note:	2023-06-30 16:02:49 +08:00
Hongbo Wu	53b854b69f	fix: improve full-text search result ... * always sort by score first * boost title score by 3 times * tie breaker is 0.3	2023-06-30 15:32:09 +08:00
Hongbo Wu	13afb7febd	feat: create labels if not exist when setting labels in the highlight	2023-06-30 11:40:39 +08:00
Hongbo Wu	8d3cac989b	feat: create labels if not exist when setting labels in the page	2023-06-30 11:08:03 +08:00
Hongbo Wu	74f2dd6bf1	catch authorization error in rate limiter and 5 rpm	2023-06-28 17:04:47 +08:00
Hongbo Wu	b7c133d58e	fix: email sending and recommendation ... * use token in the custom header as the key in rate limiter	2023-06-28 17:00:11 +08:00
Hongbo Wu	b60ace4877	log unauthorized error	2023-06-28 12:22:32 +08:00
Hongbo Wu	1b069c2696	catch authorization error and return 401	2023-06-28 12:19:47 +08:00
Hongbo Wu	af7350bd31	fix: importer status update email not sent ... * create omnivore-authorization header to contain the token	2023-06-28 11:00:40 +08:00
Hongbo Wu	7c0c4eaa40	fix: type error	2023-06-27 21:05:00 +08:00
Hongbo Wu	ed8287df19	remove tracking params from tweet url	2023-06-27 17:49:31 +08:00
Hongbo Wu	bb359d9747	save normalized url	2023-06-27 16:58:57 +08:00
Hongbo Wu	f372636d6e	fix: tweet not saved correctly when using share button on iOS	2023-06-27 16:47:36 +08:00
Hongbo Wu	90444788ba	Merge pull request #2400 from omnivore-app/feat/web-multi-select-labels ... Add support for multi select Add Label	2023-06-22 09:42:59 +08:00
Jackson Harper	cceff32e5f	Return non-error if setting labels is a noop	2023-06-21 20:24:36 +08:00
Hongbo Wu	826565a2c4	temporarily skip unsubscribing by url and mark them as unsubscribed automatically	2023-06-21 12:13:22 +08:00
Hongbo Wu	4f74b32ff6	do not subscribe a newsletter if subscription already exists and is unsubscribed	2023-06-21 12:02:15 +08:00
Hongbo Wu	84788e7551	fix: mark failed to unsubscribe newsletters as unsubscribed state	2023-06-21 11:33:49 +08:00
Hongbo Wu	ea6d06f464	Merge pull request #2375 from omnivore-app/fix/pocket-importer ... fix pocket paginated api	2023-06-19 11:19:31 +08:00
Hongbo Wu	0d8a3f7346	fix pocket paginated api	2023-06-16 18:11:31 +08:00
Hongbo Wu	c8e3f79655	Merge pull request #2368 from omnivore-app/fix/api-robots-txt ... Add robots.txt to the list of reserved names	2023-06-16 16:38:06 +08:00
Jackson Harper	cf7c50468e	Use normalized URLs when querying article saving requests ... When we save pages we save the normalized URL, so if a user saves a URL like https://foo.com/ in elastic it will be saved as https://foo.com but the browser will redirect to this url: https://omnivore.app/article?url=https://foo.com/ and the URL lookup will fail.	2023-06-16 15:15:24 +08:00
Jackson Harper	5dd28de9e8	Add security.txt file too	2023-06-16 12:54:30 +08:00
Jackson Harper	52af2d8cb4	Add robots.txt to the list of reserved names	2023-06-16 12:54:30 +08:00
Hongbo Wu	9a3673e315	Merge pull request #2367 from omnivore-app/fix/refresh-index ... refresh index after altering the page state or adding/removing/updating labels/highlights	2023-06-15 12:18:53 +08:00
Hongbo Wu	ceaa7b0356	refresh index after altering the page state or adding/removing/updating labels/highlights	2023-06-15 11:37:09 +08:00
Hongbo Wu	5d2ab3f554	create another rate limiter for auth api with 4 rpm	2023-06-14 19:31:18 +08:00
Hongbo Wu	fc08a434e1	Merge pull request #2359 from omnivore-app/fix/pocket-import ... fix/pocket import	2023-06-14 15:32:10 +08:00
Jackson Harper	aaf6bb617f	Remove backend events that we dont need to watch	2023-06-14 13:19:39 +08:00
Hongbo Wu	11cf9f7c28	stringify csv before uploading	2023-06-14 12:50:06 +08:00
Hongbo Wu	d851a705a6	refresh index after archiving page	2023-06-13 12:44:32 +08:00
Hongbo Wu	771114b30d	refresh index if update is not async	2023-06-13 12:43:18 +08:00
Hongbo Wu	1a48ad03e4	use a separate queue for thumbnailer	2023-06-12 12:25:04 +08:00
Hongbo Wu	8904247620	add test	2023-06-09 18:58:41 +08:00
Hongbo Wu	b12e127d15	use gold color for the label	2023-06-09 17:36:38 +08:00
Hongbo Wu	5db4354f3f	make internal optional in label response	2023-06-09 17:33:14 +08:00
Hongbo Wu	bd4204fc42	add analytics	2023-06-09 13:14:14 +08:00
Hongbo Wu	55904ccf04	feat: add set favorite article api which adds the favorites internal label to it	2023-06-09 13:09:47 +08:00
Hongbo Wu	6116746e80	feat: make newsletters and favorites internal labels	2023-06-09 12:48:46 +08:00
Hongbo Wu	51032d2bf1	fix: test	2023-06-08 18:52:47 +08:00
Hongbo Wu	039f00e3b2	perf: add max doc and throttling to bulk api	2023-06-08 18:36:09 +08:00
Hongbo Wu	4678d6f473	check label uniqueness by name	2023-06-08 13:33:59 +08:00
Hongbo Wu	3630d743d7	add unique labels	2023-06-08 13:29:05 +08:00
Hongbo Wu	de3623d3f2	allow mark as read and add labels bulk actions	2023-06-08 13:05:30 +08:00
Hongbo Wu	1af845e4e5	feat: add bulk action api schema	2023-06-07 21:50:43 +08:00
Hongbo Wu	b698029972	store token in cookie instead of authorization header	2023-06-07 14:24:52 +08:00
Hongbo Wu	c0872939a7	Merge pull request #2311 from omnivore-app/feature/metadata ... Feature: Set thumbnail and pre-cache all the images in the content	2023-06-07 12:08:20 +08:00
Hongbo Wu	9be84fb473	create thumbnail task for newsletters too	2023-06-07 11:52:13 +08:00
Hongbo Wu	986c4eb48b	fix: parse unsubscribe mailto email address and set subject	2023-06-06 19:53:18 +08:00
Hongbo Wu	4f4f50af49	fix: redundant argument	2023-06-06 12:05:33 +08:00
Hongbo Wu	0c3fa2cdd4	feat: start a cloud task to set thumbnail and pre-cache all the images in the content	2023-06-06 11:56:31 +08:00
Hongbo Wu	e2315b5662	feat: add thumbnail handler	2023-06-06 11:02:46 +08:00
Hongbo Wu	c66d13177f	Merge pull request #2295 from omnivore-app/fix/twitter-handler-timeout ... Feature: importer metrics collector and fix puppeteer timeout issue	2023-06-05 17:17:50 +08:00
Jackson Harper	e53cc28683	Fix issue where we return EPUB content reader for items marked as book in metadata ... If we parse a page with <meta property="og:type" content="book"> in it now, we incorrectly set this to ContentReader.EPUB, which then causes syncing issues on iOS which doesn't have an EPUB reader type defined in its GraphQL schema yet.	2023-06-05 14:44:51 +08:00
Hongbo Wu	a439aac7e7	feat: update api to include preview image	2023-06-05 12:38:30 +08:00
Hongbo Wu	d69e732579	fix: dedupe url if failed to save	2023-06-01 16:11:35 +08:00
Hongbo Wu	15b41560de	fix: improve the speed of generating highlighted markdown content	2023-05-26 15:51:54 +08:00
Hongbo Wu	15b977f7c6	fix: set server timeout = 10 mins	2023-05-26 15:00:46 +08:00
Jackson Harper	751aec2676	Use number instead of function	2023-05-19 18:10:28 +08:00
Jackson Harper	f671b3625c	Change rate limiting rules	2023-05-19 18:05:57 +08:00
Hongbo Wu	4cc9547935	fix: fallback to html if failed to convert html to markdown	2023-05-19 16:55:50 +08:00
Hongbo Wu	87f55eabf6	fix: sentry report on node is null	2023-05-19 10:54:43 +08:00
Hongbo Wu	331ddf69d4	remove debugging log	2023-05-18 21:12:44 +08:00
Hongbo Wu	0adab7ad23	fix: sentry report on unable to find highlight	2023-05-18 21:07:51 +08:00
Hongbo Wu	48554be76d	fix: wrap link with ==	2023-05-18 21:04:27 +08:00
Hongbo Wu	2bf932cbdc	fix: &npsp; breaks highlight	2023-05-18 16:50:49 +08:00
Jackson Harper	058ccf3006	Merge pull request #2183 from omnivore-app/fix/unauthed-rate-limit ... Reduce the rate limit for unauthenticated users	2023-05-18 15:15:18 +08:00
Jackson Harper	1d37f04808	Properly call getClaimsByToken	2023-05-16 17:14:38 +08:00
Hongbo Wu	e14080a05f	Set labels = undefined if pocket tags is undefined	2023-05-16 11:04:22 +08:00
Hongbo Wu	62ce7ff713	Merge pull request #2200 from omnivore-app/fix/pocket-import ... Pass pocket auth token in query string	2023-05-15 21:53:15 +08:00
Hongbo Wu	175b82485f	Fix typo	2023-05-15 17:39:29 +08:00
Hongbo Wu	c2e62155cc	Pass pocket auth token in query string	2023-05-15 16:56:27 +08:00
Hongbo Wu	e3d8fae6c5	Remove debug log	2023-05-12 19:28:19 +08:00
Hongbo Wu	791d11f581	Allow content to be converted into highlightedMarkdown format	2023-05-12 19:27:04 +08:00
Jackson Harper	c49ca0149f	Set max to 10/minute for unauthenticated requests	2023-05-12 17:11:20 +08:00
Hongbo Wu	565c932a2d	Add archivedAt to the search item in the search api response	2023-05-12 12:22:11 +08:00
Jackson Harper	cbdd4f0a71	Merge pull request #2184 from omnivore-app/fix/rm-auth-cookie-on-reset ... Clear auth cookie when password reset is triggered	2023-05-11 17:33:16 +08:00
Jackson Harper	8d9f54c62c	Fix linting	2023-05-11 13:36:27 +08:00
Jackson Harper	78681df4ab	Use clearCookie, rm the pending auth cookie too	2023-05-11 13:32:43 +08:00
Jackson Harper	4329c28904	On password reset clear the auth cookie	2023-05-11 13:29:35 +08:00
Jackson Harper	88aa7aad4c	Reduce the rate limit for unauthenticated users	2023-05-11 13:23:43 +08:00
Jackson Harper	f8df71508b	Enforce max lengths on signup credentials	2023-05-10 23:09:31 +08:00
Hongbo Wu	a4d385bc83	add sorting by wordsCount	2023-05-03 08:36:33 +08:00
Hongbo Wu	31cbcc7c7f	validate sign up request body in mobile sign-up router	2023-04-27 12:18:32 +08:00
Hongbo Wu	0e8887a1d9	trim user email before saving in db	2023-04-27 11:39:34 +08:00
Hongbo Wu	673c43aee0	trim user email in mobile sign-in router	2023-04-27 11:37:27 +08:00
Hongbo Wu	fb4c8cb1ed	trim user email in mobile sign-up router	2023-04-27 11:33:03 +08:00
Hongbo Wu	692fd7b959	Fix save url error when uid is not in the apollo context	2023-04-25 17:01:25 +08:00
Hongbo Wu	3841734deb	Fix forwarded email subject being treated as an URL, for example "Fwd: subject"	2023-04-24 21:16:44 +08:00
Hongbo Wu	b891e547c9	Make content optional	2023-04-24 21:03:39 +08:00
Hongbo Wu	e512ba0bad	Fix tests	2023-04-24 18:04:18 +08:00
Hongbo Wu	69968e5288	Save url if the email subject is a parsable url	2023-04-24 12:54:23 +08:00
Hongbo Wu	bce7afde50	Eagerly load profile of the user	2023-04-24 12:53:40 +08:00
Hongbo Wu	b57ccd0616	Allow wildcards in query filter	2023-04-21 17:39:04 +08:00
Hongbo Wu	b2c115f9d9	Allow wildcards in label: filter	2023-04-21 17:10:46 +08:00
Hongbo Wu	130677b50c	Add contentReader to the typeAheadSearchResult	2023-04-20 10:00:56 +08:00
Hongbo Wu	5ce4bf7171	Compare labels with lower cased	2023-04-20 09:38:04 +08:00
Hongbo Wu	62b772eeca	Allow saving epub from email attachments	2023-04-19 11:30:49 +08:00
Jackson Harper	3c12162866	Fix typo in content type check	2023-04-18 11:22:44 +08:00
Jackson Harper	69df32d428	Implement a reader for epubs	2023-04-18 09:00:48 +08:00
Hongbo Wu	e6690709c3	Replace express-http-context with express-http-context2	2023-04-12 17:50:13 +08:00
Hongbo Wu	014dc773e3	Fix labels not saved correctly by saveUrl	2023-04-12 17:04:59 +08:00
Hongbo Wu	874247661b	resolve conflicts	2023-04-12 17:04:59 +08:00
Hongbo Wu	c07ada0218	resolve conflicts	2023-04-12 17:04:59 +08:00
Hongbo Wu	42dc3f7330	Fix some external requests not being mocked	2023-04-12 17:04:59 +08:00
Hongbo Wu	9857af27e2	Add labels and state to createArticle API	2023-04-12 17:04:59 +08:00
Hongbo Wu	abd42f7064	Add labels and state to saveUrl API	2023-04-12 16:50:11 +08:00
Hongbo Wu	ce273b5172	Add test case for archives page and saves labels in savePage API	2023-04-12 16:50:11 +08:00
Hongbo Wu	87d3f5b3d8	Add labels and state to savePage API	2023-04-12 16:50:11 +08:00
Hongbo Wu	fcf08fc364	Add state and labels to savePageInput	2023-04-12 16:50:11 +08:00
Hongbo Wu	4b578bbebb	Process labels and state in the csv file	2023-04-12 16:50:11 +08:00
Hongbo Wu	9a46e935d9	Add [] to the labels in csv	2023-04-12 16:50:11 +08:00
Hongbo Wu	06b6583f29	Mock cloud storage bucket	2023-04-12 16:50:11 +08:00
Hongbo Wu	76b63efdc1	Fix tests	2023-04-12 16:50:11 +08:00
Hongbo Wu	10c01c12f2	Add test case for integration import task handler	2023-04-12 16:50:10 +08:00
Hongbo Wu	56fd3d7617	Create a router to handle importing from integrations cloud task	2023-04-12 16:50:10 +08:00
Hongbo Wu	503ec3d413	Update task name	2023-04-12 16:50:10 +08:00
Hongbo Wu	9ff0711a73	Remove wrong comments	2023-04-12 16:50:10 +08:00
Hongbo Wu	c8ec84562b	Create an import from integration cloud task	2023-04-12 16:50:10 +08:00
Hongbo Wu	b62b769604	Paginate pocket api	2023-04-12 16:50:10 +08:00
Hongbo Wu	1c5991aded	import from integration api return a boolean value	2023-04-12 16:50:10 +08:00
Hongbo Wu	c78fe9f87d	Paginate pocket retrieve api response data	2023-04-12 16:50:10 +08:00
Hongbo Wu	737957ff82	Add import from pocket to web	2023-04-12 16:50:10 +08:00
Hongbo Wu	2e5e3bd19e	Add headers	2023-04-12 16:50:10 +08:00
Hongbo Wu	59621768c8	Get request token from pocket	2023-04-12 16:50:10 +08:00
Hongbo Wu	ae2f695df8	Get access token from pocket	2023-04-12 16:50:10 +08:00
Hongbo Wu	9c7f6fefca	Add PocketIntegration service to the list	2023-04-12 16:50:10 +08:00
Hongbo Wu	c3bfb593c6	Fix regression	2023-04-12 16:50:10 +08:00
Hongbo Wu	02247bc0a6	Add import from integration api implementation	2023-04-12 16:50:10 +08:00
Hongbo Wu	e2d050288e	write the list of urls to a csv file and upload it to gcs	2023-04-12 16:50:10 +08:00
Hongbo Wu	94d8903ec1	Add pocket integration support	2023-04-12 16:50:10 +08:00
Hongbo Wu	0bf4119c92	Refactor with factor pattern	2023-04-12 16:50:10 +08:00
Hongbo Wu	4ce4cd0a62	Add a type for export or import to the integration table	2023-04-12 16:50:10 +08:00
Hongbo Wu	9056318667	Change type field in integration table from enum to text type	2023-04-12 16:50:10 +08:00
Hongbo Wu	b43f6904f4	Merge pull request #2021 from omnivore-app/fix/subscriptions ... creates or updates subscription only if their is a valid unsubscribe link	2023-04-12 13:17:34 +08:00
Hongbo Wu	80d0e7c944	Move pageType to function_resolver	2023-04-12 12:37:22 +08:00
Hongbo Wu	158a75807c	Use camel case	2023-04-12 12:27:05 +08:00
Hongbo Wu	4969391d4c	Replace picture with image	2023-04-12 12:15:53 +08:00
Hongbo Wu	c5eb223c07	Add tweet, video, picture pageType	2023-04-12 11:53:23 +08:00
Hongbo Wu	f28912728a	creates or updates subscription only if their is a valid unsubscribe link	2023-04-11 22:39:47 +08:00
Hongbo Wu	83cd0ac880	Return error if query exceeds 255 char	2023-04-06 12:03:26 +08:00
Hongbo Wu	3d599ad7ab	Remove sanitize directive on query argument because we will change the argument type if doing so	2023-04-06 11:56:19 +08:00
Hongbo Wu	c596e88f46	Set search query max size to 255 char	2023-04-05 12:20:52 +08:00
Hongbo Wu	e5ffd3f5bd	Stop inserting search history to the DB in the search API	2023-04-05 11:45:39 +08:00
Hongbo Wu	819d275497	Add wordsCount if undefined when creating pages in elastic	2023-04-04 12:52:46 +08:00
Hongbo Wu	32d31af847	Merge pull request #1970 from omnivore-app/double-quote-size-limit ... Double max length of quote in highlights	2023-04-04 11:44:28 +08:00
Hongbo Wu	c52d2f0f56	Fix pageType to be null in updatesSince api response	2023-04-04 11:00:13 +08:00
Hongbo Wu	8d79f858c3	Double max length of quote in highlights	2023-04-03 18:46:59 +08:00
Hongbo Wu	53a2df5dd2	Fix start_date or end_date is undefined in elasticsearch search query	2023-03-27 18:45:24 +08:00
Hongbo Wu	dd24cfbeec	Remove async	2023-03-23 10:45:39 +08:00
Hongbo Wu	abb8df26e5	Set HIGHLIGHT as default highlight type in every highlight	2023-03-23 10:44:30 +08:00
Hongbo Wu	aae16d7a4d	Set HIGHLIGHT as default highlight type	2023-03-22 22:02:03 +08:00
Hongbo Wu	b3a965781c	resolve conflicts	2023-03-22 10:40:03 +08:00
Hongbo Wu	ee151637e3	Only send event if highlight updated	2023-03-22 10:37:07 +08:00
Hongbo Wu	9df1546323	Fix tests	2023-03-22 10:37:07 +08:00
Hongbo Wu	b0e91cbbbb	Change GraphQL APIs	2023-03-22 10:36:38 +08:00
Hongbo Wu	71826148fd	Add type to the highlight in the elasticsearch and three types are HIGHLIGHTS, REDACTION and NOTE	2023-03-22 10:34:40 +08:00
Hongbo Wu	013b460e39	Check the permission to access page in elasticsearch	2023-03-21 15:56:30 +08:00
Hongbo Wu	e84b517493	Do not update createdAt for existing page	2023-03-21 13:57:57 +08:00
Hongbo Wu	8148453503	When saving an existing page, update the content	2023-03-21 13:44:00 +08:00
Hongbo Wu	bc9ba1d976	Reset state to processing when creating articles	2023-03-21 13:05:00 +08:00
Hongbo Wu	5af9d9266c	Do not fetch deleted article in saving request	2023-03-21 13:03:53 +08:00
Hongbo Wu	f053b60cbe	Allow array values in page filters	2023-03-21 13:03:23 +08:00
Hongbo Wu	f93148fcdd	Add BAD_DATA to the saving request error code	2023-03-21 13:02:38 +08:00
Jackson Harper	265c050f5c	Return readerURL on save URL requests	2023-03-20 17:28:55 +08:00
Hongbo Wu	c315e8be04	Return article's url in saveUrl api	2023-03-20 16:49:18 +08:00
Hongbo Wu	fb945f2791	Return url in save API	2023-03-20 16:48:50 +08:00
Hongbo Wu	f348ec7100	Add id to saving request and make both id and url optional	2023-03-17 17:56:38 +08:00
Hongbo Wu	f9396450a5	Get saving request by url	2023-03-17 11:37:59 +08:00
Hongbo Wu	fc662dd2d5	Return existing slug if page exists	2023-03-17 11:37:59 +08:00
Hongbo Wu	86ae533b66	Merge pull request #1915 from omnivore-app/allow-scoping-searches-to-site ... Allow scoping searches to site:	2023-03-17 09:59:21 +08:00
Hongbo Wu	adae20e7e1	Use elasticsearch body builder	2023-03-15 19:15:40 +08:00
Hongbo Wu	45eed5c40d	Must match siteName	2023-03-15 13:32:04 +08:00
Hongbo Wu	7d2cb07af0	eslint	2023-03-14 22:24:17 +08:00
Hongbo Wu	e5f1f25a4d	Allow scoping searches to site:	2023-03-14 18:26:45 +08:00
Hongbo Wu	a8682b8cf6	Save html along with highlight and return	2023-03-14 15:55:32 +08:00
Hongbo Wu	bf6ae38fe1	Add html to the highlight document in elasticsearch	2023-03-14 15:20:09 +08:00
Hongbo Wu	c3f2262b6b	Update test	2023-03-14 14:52:42 +08:00
Hongbo Wu	ee2487ff01	Case insensitive	2023-03-14 14:42:56 +08:00
Hongbo Wu	461b348dc0	Use lower function in the query	2023-03-14 14:42:00 +08:00
Hongbo Wu	7dd32c3e87	Trim whitespace text from user email address	2023-03-14 12:57:23 +08:00
Hongbo Wu	a0ec7bc89d	Reset readingProgressAnchorIndex to zero if the input is zero	2023-03-11 10:16:47 +08:00
Hongbo Wu	98d5f92ad4	Call youtube data api to fetch published date	2023-03-10 17:38:03 +08:00
Hongbo Wu	a0aab94e8b	Add site_name and published date to saved twitter	2023-03-10 16:42:40 +08:00
Hongbo Wu	5c3dd904a5	do not use highlightPositionAnchorIndex as the location	2023-03-10 15:54:51 +08:00
Hongbo Wu	12160cc9dc	filter out highlights with no quote	2023-03-10 15:43:06 +08:00
Hongbo Wu	45a857152b	Add minLength = 1 for quote in highlight	2023-03-10 15:40:31 +08:00
Hongbo Wu	2e197644d4	Add debug logs	2023-03-10 15:17:04 +08:00
Hongbo Wu	45b2fa6329	fix not setting zero	2023-03-10 12:46:41 +08:00
Hongbo Wu	bba0f69f62	Add top percentage to the reading position in API	2023-03-10 12:38:25 +08:00
Hongbo Wu	590a1cb8ad	Only publish labels to add to the pubsub queue	2023-03-09 21:46:33 +08:00
Hongbo Wu	104da191d2	Allow webhooks with same event types	2023-03-09 15:00:06 +08:00
Jackson Harper	58156b72ff	Update yarn.lock	2023-03-09 10:44:42 +08:00
Jackson Harper	d0e4c01b45	Allow passing mode to backend to better handle mode changes in client	2023-03-09 10:15:14 +08:00
Hongbo Wu	a1e3ac8a07	Add quote to the updateHighlightInput of the updateHighlight API	2023-03-08 15:37:24 +08:00
Hongbo Wu	87fd49540a	Merge pull request #1697 from omnivore-app/dependabot/npm_and_yarn/knex-2.4.2 ... Bump knex from 0.21.12 to 2.4.2	2023-03-06 09:45:50 +08:00
Hongbo Wu	0a350bef6d	Fix issues with knex version bump	2023-03-05 21:36:46 +08:00
Hongbo Wu	bd12ecd7ec	Add publishedAt to the updatePageInput	2023-03-02 13:40:59 +08:00
Hongbo Wu	65c244edd7	Merge pull request #1819 from omnivore-app/feature/search-filter-with-no-as-keyword	2023-02-21 12:23:22 +08:00
Hongbo Wu	214766dda6	Add no: keyword to the search filter to filter items without label or highlight	2023-02-17 12:48:02 +08:00
Hongbo Wu	153ca6995b	Refactor exclude labels query	2023-02-17 12:13:20 +08:00
Hongbo Wu	7f761049d7	Preparse the newsletter content	2023-02-17 09:49:19 +08:00
Hongbo Wu	cc8b1cefdb	Preserve <pre> elements with prism- class and identity them as code blocks	2023-02-14 12:33:59 +08:00
Hongbo Wu	d816ee9563	Allow image and media to be loaded in puppeteer	2023-02-13 16:50:39 +08:00
Hongbo Wu	0359cefb4f	Merge pull request #1794 from omnivore-app/add-domdistiller-test-page ... Add dom-distiller generated test pages	2023-02-10 10:42:59 +08:00
Hongbo Wu	c285a017c3	Increase view.html iframe height	2023-02-09 22:20:58 +08:00
Hongbo Wu	761f2ff8f2	Add JWT token to the request	2023-02-09 18:05:04 +08:00
Hongbo Wu	a91cf9ef63	Add an option for the original html to be parsed by distiller	2023-02-09 17:44:02 +08:00
Jackson Harper	b0ccfc213d	Operate on pages in processing state during bulk operations	2023-02-09 12:28:06 +08:00
Jackson Harper	e6833fddff	Add to address on email not forwarded logs	2023-02-06 11:13:28 +08:00
Jackson Harper	1e63899748	Add additional logging for mobile auth login errors	2023-02-06 11:11:28 +08:00
Hongbo Wu	1458395630	Merge pull request #1759 from omnivore-app/archive-all-api ... bulk action api	2023-02-02 22:14:33 +08:00
Hongbo Wu	bb5c104a42	Change archiveAll to bulkAction which allows deletion too	2023-02-02 21:26:37 +08:00
Hongbo Wu	7255061e36	Unescape HTML entities in annotation when creating highlight	2023-02-02 11:32:24 +08:00
Hongbo Wu	3b888bbb08	Make Async call to archive all	2023-02-01 18:50:10 +08:00
Hongbo Wu	03ab96b7f8	Add archiveAll API	2023-02-01 18:05:16 +08:00
Hongbo Wu	dd6c8cf6dc	Make subject optional too	2023-02-01 13:53:27 +08:00
Hongbo Wu	ac6f5b497c	Accept null value for text in an email	2023-02-01 13:44:27 +08:00
Hongbo Wu	83c8166b3b	Unescape HTML entities in annotation to avoid showing escaped characters	2023-02-01 12:13:00 +08:00
Jackson Harper	749df941f8	Use ilike for recent email query	2023-01-28 16:18:06 +08:00
Jackson Harper	7476878254	Add more logging for recent email lookups	2023-01-28 15:05:45 +08:00
Jackson Harper	6a50e85759	Save the parsed email address as the to field in received_emails ... The unparsed addresses can look something like this: "jacksonh-XXXX@inbox.omnivore.app" <jacksonh-XXXX@inbox-demo.omnivore.app> but we want to join on this field to the NewsletterEmail.address field which will look like `jacksonh-XXXX@inbox.omnivore.app`	2023-01-28 13:53:23 +08:00
Jackson Harper	15782fc5ce	Add some analytics when creating reports	2023-01-26 19:32:41 +08:00
Jackson Harper	506568e0c2	Merge pull request #1667 from omnivore-app/recent-emails ... recent emails	2023-01-25 14:22:00 +08:00
Hongbo Wu	a682d87aa6	Add tests for recent emails resolver	2023-01-19 12:11:34 +08:00
Hongbo Wu	83c62c65a3	Add test for saving received email	2023-01-19 10:44:08 +08:00
Hongbo Wu	318a6557a5	Email us if a recent email marked as library item	2023-01-19 09:55:49 +08:00
Hongbo Wu	7dbf8662fc	Fix test	2023-01-19 09:38:21 +08:00
Hongbo Wu	8f9c317dd1	Save received email before publishing data	2023-01-18 22:35:07 +08:00
Jackson Harper	08048405a1	Correct typo in doc about labels	2023-01-18 22:07:39 +08:00
Hongbo Wu	36d2ac041a	Fix error of getting subscriptions without newsletter email by inner join them	2023-01-18 18:42:10 +08:00
Hongbo Wu	d60890d154	Add pubsub to save received email	2023-01-18 17:13:11 +08:00
Hongbo Wu	60e107964f	Fix typo	2023-01-18 15:00:34 +08:00
Hongbo Wu	96f727a165	Add mark a recent email as library item api	2023-01-18 15:00:34 +08:00
Hongbo Wu	62a3b48e00	Add list recent emails api	2023-01-18 15:00:34 +08:00
Hongbo Wu	3f373ab3b6	Save received email	2023-01-18 15:00:34 +08:00
Hongbo Wu	2da4ca5da8	Add saveReceivedEmail service	2023-01-18 15:00:34 +08:00
Hongbo Wu	b7d732df7c	Add received_email entity	2023-01-18 15:00:34 +08:00
Hongbo Wu	f01ab3805a	Left join active subscriptions of a newsletter email to get the right count	2023-01-18 14:37:54 +08:00
Hongbo Wu	e94f6de466	Delete subscription if unsub	2023-01-18 14:36:23 +08:00
Jackson Harper	786ad35da9	Merge pull request #1635 from omnivore-app/fix/content-display-reports ... Fix creation of content display report	2023-01-13 11:37:22 +08:00
Hongbo Wu	6572f718c9	Merge pull request #1648 from omnivore-app/fix/puppeteer-parse ... fix/puppeteer parse	2023-01-13 10:59:09 +08:00
Hongbo Wu	c647aabeaa	Save newsletter_email_id in subscription instead of the whole entity	2023-01-12 22:15:42 +08:00
Hongbo Wu	7bfd36e308	Fix readability not run in puppeteer-parse	2023-01-12 18:33:47 +08:00
Jackson Harper	824899760a	Update some style, add Matter archive importer tool	2023-01-12 14:00:46 +08:00
Jackson Harper	337b723d24	Fix creation of content display report	2023-01-11 10:51:46 +08:00
Hongbo Wu	43e813a73d	Add unique constraint on user_id and name to subscription entity	2023-01-09 15:12:26 +08:00
Hongbo Wu	892a9ef30d	Add subscriptionCount and createdAt in NewsletterEmail type	2023-01-09 12:24:45 +08:00
Hongbo Wu	3c8416ca1b	Merge pull request #1620 from omnivore-app/fix/sign-up ... Log email content if Sendgrid api key not set	2023-01-06 15:39:33 +08:00
Hongbo Wu	9762b14ebc	Check if the env is local	2023-01-06 15:28:45 +08:00
Jackson Harper	3520428ff0	Get the auth token from cookie in the user_router ... This is called through Cloud Task so .authorization is actually Cloud Task's auth token.	2023-01-06 10:29:36 +08:00
Hongbo Wu	d86a6038ea	Log email content if Sendgrid api key not set	2023-01-06 09:54:02 +08:00
Jackson Harper	88b6fcfb5d	Use cookie auth instead of header as GCP uses the Authorization header	2023-01-05 20:17:43 +08:00
Hongbo Wu	24763669b3	Merge pull request #1616 from omnivore-app/html-to-markdown ... html to markdown	2023-01-05 19:30:19 +08:00
Hongbo Wu	e28c9b10bb	Fetch markdown content if specified in search API	2023-01-05 17:17:09 +08:00
Jackson Harper	b0938ea8e2	Merge pull request #1612 from omnivore-app/feat/email-after-import ... Send email to user after a bulk import completes	2023-01-05 13:39:49 +08:00
Hongbo Wu	7b749f974d	Convert HTML to markdown if specified	2023-01-05 12:12:48 +08:00
Jackson Harper	89454d8b59	Merge pull request #1610 from omnivore-app/fix/reduce-logging ... Remove some places we are logging too much data causing sentry failures	2023-01-05 11:19:30 +08:00
Jackson Harper	cfa3b5de89	Remove unneeded linting rules	2023-01-05 10:25:10 +08:00
Jackson Harper	7b2bc21b16	Add cors config, handle exception response	2023-01-04 21:45:41 +08:00
Jackson Harper	d89a307aca	Send email to user after a bulk import completes	2023-01-04 18:15:26 +08:00
Jackson Harper	2390f2d716	Cast data to NewsletterMessage	2023-01-04 16:24:41 +08:00
Jackson Harper	e3ed7d0250	More logging changes	2023-01-04 11:23:13 +08:00
Jackson Harper	b438d42f38	Remove some unneeded console.log calls	2023-01-04 11:18:37 +08:00
Jackson Harper	8edd94bdb5	Remove some places we are logging too much data causing sentry failures	2023-01-04 11:07:54 +08:00
Jackson Harper	f939786c2a	Reserve some names for tools and landing pages	2023-01-03 20:55:30 +08:00
Jackson Harper	0edd91057e	Update user interface for file import tool	2023-01-03 17:49:07 +08:00
Jackson Harper	9314c3d8f1	Add uploadImportFile API method ... Add uploadImportFile API method Fix prefix, counting max files uploaded Add resolver types Basic web ui for the uploader interface Allow selecting type when uploading import files	2023-01-03 10:01:59 +08:00
Hongbo Wu	2c39916441	Add wordsCount to article schema	2023-01-02 16:49:11 +08:00
Hongbo Wu	e0b9a6689a	Not raise error when there is conflict of setting labels in highlights	2022-12-30 11:22:41 +08:00
Hongbo Wu	e0437008d7	Fix not updating/deleting the right label in the highlight	2022-12-29 22:40:31 +08:00
Hongbo Wu	264ebef46e	Fix not setting labels to the correct highlight in a page	2022-12-29 21:34:45 +08:00
Hongbo Wu	7c39db207b	Replace createArticle with savePage in puppeteer-parse service	2022-12-28 10:15:05 +08:00
Hongbo Wu	e22d209721	Fix test	2022-12-28 09:16:44 +08:00
Hongbo Wu	e9b31e375f	Update savePage API to accept parsed content in the param	2022-12-27 21:59:31 +08:00
Hongbo Wu	302ea2023f	Add obsidian to corsConfig	2022-12-23 10:29:43 +08:00
Jackson Harper	34029bee7e	Update getting started guide, improve section on newsletter forwards, fix link to Logseq guide	2022-12-14 17:59:32 +08:00
Hongbo Wu	83aa2093eb	Still return admins if not allowed to see members	2022-12-13 19:05:07 +08:00
Hongbo Wu	943b20f21e	Only recommend page if the user is allowed to post	2022-12-13 19:00:49 +08:00
Hongbo Wu	5ef4e8479d	Create one rule for all pages in the group to have the label and another rule for the recommended page to send a notification	2022-12-13 16:35:39 +08:00
Hongbo Wu	a340699240	add comment	2022-12-13 12:15:41 +08:00
Hongbo Wu	918ada2100	only recommend highlights created by the user	2022-12-13 12:12:20 +08:00
Hongbo Wu	25842270d7	Update the rule to only trigger on page created event	2022-12-13 12:04:17 +08:00
Hongbo Wu	4247835dc5	Merge pull request #1537 from omnivore-app/group-rules ... group rules	2022-12-13 09:54:20 +08:00
Hongbo Wu	f5650acb69	Return canPost and canSeeMembers and topics and description in type group	2022-12-12 22:57:58 +08:00
Hongbo Wu	54ceeb326a	Add canPost and canSeeMembers to type Group	2022-12-12 22:40:19 +08:00
Hongbo Wu	3e79b86179	Allow search recommendations with wildcard	2022-12-12 22:09:25 +08:00
Hongbo Wu	1338b1fc10	Remove subscription from shared page	2022-12-12 17:53:58 +08:00
Hongbo Wu	70ed3dccb7	Add analytic events for create/join/leave group actions	2022-12-12 17:49:00 +08:00
Hongbo Wu	5650971042	Do not save recent searches for rule based queries	2022-12-12 17:40:40 +08:00
Hongbo Wu	940c449880	revert for update page	2022-12-12 10:17:34 +08:00
Hongbo Wu	3347e81e39	Fix rule action not triggered due to the page index not being refreshed after page creation	2022-12-10 13:27:32 +08:00
Hongbo Wu	845235268d	Merge pull request #1525 from omnivore-app/recommend-api ... Add recommendation to the original page too	2022-12-09 21:40:26 +08:00
Hongbo Wu	f4fa9cb15b	do the same for merge highlight	2022-12-09 15:29:15 +08:00
Hongbo Wu	5fc18770ad	Upsize max highlight char length to 6000	2022-12-09 15:24:10 +08:00
Hongbo Wu	181b401030	Update new recommendation notification message	2022-12-09 10:56:08 +08:00
Hongbo Wu	1f54db79fb	Add recommendation to the original page too	2022-12-09 10:10:02 +08:00
Hongbo Wu	b8e7b15805	Fix generating colors with less than 6 digits	2022-12-08 19:01:54 +08:00
Hongbo Wu	04cb360c2f	Fix some issue	2022-12-08 18:50:36 +08:00
Hongbo Wu	b942303b18	Create a rule after creating or joining a group	2022-12-08 18:35:22 +08:00
Hongbo Wu	9f260c1442	Create a label for new group if not exists	2022-12-08 18:20:58 +08:00
Hongbo Wu	84d8a9bb67	lint	2022-12-08 16:36:17 +08:00
Hongbo Wu	ff3e8aa29e	Add delete permission to group and membership tables	2022-12-08 16:29:46 +08:00
Hongbo Wu	6830da995f	Add leaveGroup API	2022-12-08 15:36:28 +08:00
Hongbo Wu	4c92af1856	Allow recommended With Highlights	2022-12-08 12:51:01 +08:00
Hongbo Wu	ef13cbf39a	Recommend highlights	2022-12-08 12:33:05 +08:00
Hongbo Wu	62a3f6d1d8	Add recommendHighlights API	2022-12-08 11:37:35 +08:00
Jackson Harper	b150083358	Remove debug	2022-12-07 11:53:48 +08:00
Jackson Harper	533a4ae65c	Update recommendedBy query	2022-12-06 21:23:19 +08:00
Jackson Harper	a8df439297	Rename recommendedBy to recommendations	2022-12-06 21:18:21 +08:00
Jackson Harper	0892d9e7e6	Add user and note to Recommendation type	2022-12-06 17:44:53 +08:00
Jackson Harper	8ac1afe277	Use cookie auth instead of header to not interfere with GCPs auth mechanisms	2022-12-05 18:41:08 +08:00
Jackson Harper	e555cc733a	More descriptive error messages	2022-12-05 18:40:33 +08:00
Hongbo Wu	c49db571cd	Add search by recommendedBy group name	2022-12-05 14:53:29 +08:00
Hongbo Wu	bf18c3d3c7	Add recommendedBy in search result	2022-12-05 14:27:46 +08:00
Hongbo Wu	e6b56e6fba	Rename groups service	2022-12-05 14:12:04 +08:00
Hongbo Wu	714f0f1f63	Add joinGroup API	2022-12-05 12:47:38 +08:00
Hongbo Wu	27d9e8464d	Add unique constraint on user_id and group_id to group_membership table	2022-12-05 12:45:54 +08:00
Hongbo Wu	6b65c386a0	Add authentication to the request header	2022-12-02 21:15:33 +08:00
Hongbo Wu	7707f100f2	Add recommend gql api	2022-12-02 19:45:26 +08:00
Hongbo Wu	0e3d1ced46	Add recommend rest api to handler tasks	2022-12-02 19:10:41 +08:00
Hongbo Wu	5fd137d23e	Add recommendation cloud task	2022-12-02 18:49:13 +08:00
Hongbo Wu	a7d0525d2e	Add recommendPage helper function	2022-12-02 18:26:16 +08:00
Hongbo Wu	b19b04bbe6	Add recommendedBy nested field in page index	2022-12-02 15:24:37 +08:00
Hongbo Wu	93c6c4202a	Max number of group per user is 3	2022-12-02 12:36:18 +08:00
Hongbo Wu	da3d9cf76b	Add groups api	2022-12-02 12:32:26 +08:00
Hongbo Wu	7285ebb940	Add createGroup API	2022-12-01 18:41:00 +08:00
Hongbo Wu	7f04a381a1	Add is_admin field to group_membership table	2022-12-01 17:47:12 +08:00
Hongbo Wu	7c8b057f85	Merge pull request #1486 from omnivore-app/fix/site-name-extraction ... Fix site_name not saved when a page is saved by extensions	2022-12-01 11:28:51 +08:00
Hongbo Wu	0693a6c2ff	Merge pull request #1487 from omnivore-app/saved-filters-api ... saved filters api	2022-12-01 11:25:50 +08:00
Hongbo Wu	0eebfe923a	Add position in filter table and move filter api	2022-12-01 10:26:32 +08:00
Hongbo Wu	0487d8325c	Add saved filter api	2022-11-30 21:41:59 +08:00
Hongbo Wu	1d514a0530	Update table name	2022-11-30 20:39:09 +08:00
Hongbo Wu	4d845ef413	Add search_filter table and entity	2022-11-30 20:38:30 +08:00
Hongbo Wu	bb839960f1	Fix site_name not saved when a page is saved by extensions	2022-11-30 17:46:59 +08:00
Hongbo Wu	86dccc0d44	Update savedAt in update page api	2022-11-30 13:17:50 +08:00
Hongbo Wu	28025db959	Add labels in highlight type in api response	2022-11-25 17:12:47 +08:00
Jackson Harper	e4e21ddc3a	Merge pull request #1462 from omnivore-app/add-order-in-updatesince-api ... Add sort in updatesince api and default sorting by updatedAt in descending order	2022-11-25 13:18:05 +08:00
Hongbo Wu	64fd0cfdc6	Add sort in updatesince api and default sorting by updatedAt in descending order	2022-11-25 11:44:55 +08:00
Hongbo Wu	319c0c4bcd	Allow changing byline in update api	2022-11-24 16:25:27 +08:00
Hongbo Wu	eb345ba393	Merge pull request #1449 from omnivore-app/more-filters-in-rules ... Use search API to filter page in rule engine	2022-11-24 15:30:18 +08:00
Hongbo Wu	3f24d9a99e	Add includes filter to search by ids in search api	2022-11-24 14:10:13 +08:00
Hongbo Wu	05ecac7c52	Fix tests	2022-11-24 08:59:21 +08:00
Hongbo Wu	f473c18939	Add deleteRule api	2022-11-23 09:52:15 +08:00
Hongbo Wu	ba730bdc3e	Use default application credential	2022-11-22 13:00:01 +08:00
Hongbo Wu	2ae3226bdb	Add sendNotification api	2022-11-22 12:48:17 +08:00
Hongbo Wu	9344fae1b9	Add title and image url in notification	2022-11-22 10:50:38 +08:00
Hongbo Wu	6b772c9248	Disable push notification reminders	2022-11-22 10:20:52 +08:00
Jackson Harper	080a1982b2	Remove unused import	2022-11-21 22:39:10 +08:00
Jackson Harper	1e4658b771	Consistent function declaration syntax	2022-11-21 22:39:10 +08:00
Jackson Harper	c0d5d12ec0	Add missing file	2022-11-21 22:39:10 +08:00
Jackson Harper	a3f477f57a	Linting fix	2022-11-21 22:39:10 +08:00
Jackson Harper	9741f6b12d	Add highlights to newly saved pages if they contain highlight markers	2022-11-21 22:39:10 +08:00
Hongbo Wu	f358094bbf	Add get device tokens api	2022-11-21 17:03:12 +08:00
Hongbo Wu	88416bc178	Add get rules api	2022-11-21 15:03:15 +08:00
Hongbo Wu	dc92fe9c52	Create rule handler cloud run	2022-11-19 11:15:26 +08:00
Hongbo Wu	075c1b4d14	Fix send notification failure	2022-11-18 17:43:21 +08:00
Hongbo Wu	ecc217deef	Fix test	2022-11-18 16:25:22 +08:00
Hongbo Wu	06c30e4f40	Add notification cloud run	2022-11-18 16:13:49 +08:00
Hongbo Wu	e53d5036dc	Add notification endpoint	2022-11-18 14:33:27 +08:00
Hongbo Wu	fbe4c471cd	Add pubsub subscription to trigger rule actions	2022-11-18 14:00:10 +08:00
Hongbo Wu	50fe92ac85	Add setRule resolver	2022-11-17 18:53:05 +08:00
Hongbo Wu	845e932d93	Store actions as a json in db	2022-11-17 16:21:43 +08:00
Hongbo Wu	23c527c345	Add rule entity	2022-11-17 15:04:44 +08:00
Hongbo Wu	60027dd670	Add rules table migration sql	2022-11-17 14:50:57 +08:00
Hongbo Wu	71348ab492	Remove trigger type in gql schema	2022-11-17 14:50:57 +08:00
Hongbo Wu	775180d785	Add rules api gql schema	2022-11-17 14:50:57 +08:00
Hongbo Wu	6a9a5c3870	Merge pull request #1408 from omnivore-app/fix/use-must-clause ... Replace filter clause with must clause in the search query in elastic	2022-11-14 11:32:33 +08:00
Hongbo Wu	3f6a3b36ff	Replace default voice to Larry for pre-generating	2022-11-11 18:34:44 +08:00
Hongbo Wu	2301f44848	Replace filter clause with must clause in the search query in elastic	2022-11-11 18:31:47 +08:00
Hongbo Wu	a78074b0da	Merge pull request #1406 from omnivore-app/set-voice-in-user-personalization ... set voice in user personalization	2022-11-11 16:57:49 +08:00
Hongbo Wu	9872355056	Allow apollo sandbox access in local env only	2022-11-11 16:18:11 +08:00
Hongbo Wu	917396c976	Upsert user personalization in db	2022-11-11 16:11:09 +08:00
Hongbo Wu	67b26be547	Allow apollo sandbox access	2022-11-11 15:24:54 +08:00
Jackson Harper	83dbb54896	Allow filtering by author	2022-11-11 15:01:14 +08:00
Hongbo Wu	477b5d7d24	Fix typeorm issue	2022-11-11 14:53:46 +08:00
Hongbo Wu	ac21e4e6c8	Set voice and secondary voice in user personalization	2022-11-11 14:08:28 +08:00
Hongbo Wu	ce18d33ae1	Change queue name	2022-11-11 10:14:46 +08:00
Hongbo Wu	d58fb63c87	Pre-fetch realistic voice only if user is opted in	2022-11-10 18:49:52 +08:00
Hongbo Wu	b0938ee96c	Pre-fetch realistic voice only if user is opted in	2022-11-10 18:48:00 +08:00
Hongbo Wu	69d6b60adc	Extend expiration of the jwt token to 1 year	2022-11-10 17:42:25 +08:00
Hongbo Wu	399e31534d	Update jwt token	2022-11-09 22:48:33 +08:00
Hongbo Wu	c89ffdeaf9	Fix some tests	2022-11-09 18:14:15 +08:00
Hongbo Wu	63de1c3359	Add optInFeature API	2022-11-09 17:35:54 +08:00
Hongbo Wu	98f59c50f0	Remove token in features table	2022-11-09 11:32:08 +08:00
Hongbo Wu	c2f4ba4a57	Add features entity	2022-11-08 18:14:23 +08:00
Hongbo Wu	398e242131	Add features table	2022-11-08 17:30:03 +08:00
Hongbo Wu	f2a69d06c4	Change default rate to 1.1	2022-11-08 16:53:35 +08:00
Hongbo Wu	ddd1e84686	Add language and rate in the cloud task params	2022-11-08 16:22:23 +08:00
Hongbo Wu	769e8c5f96	Add default language and rate	2022-11-08 15:41:20 +08:00
Hongbo Wu	3ffd7739bb	Default voice is Larry	2022-11-08 14:23:48 +08:00
Hongbo Wu	44987072ee	Get voice from user personalization	2022-11-08 13:08:05 +08:00
Hongbo Wu	271280f4fc	Fix not correctly getting audio file	2022-11-08 12:56:52 +08:00
Hongbo Wu	e62765c3e6	Save steaming data in GCS	2022-11-08 11:24:47 +08:00
Hongbo Wu	1af5af0e97	Get realistic voice id from user personalization table	2022-11-08 10:21:27 +08:00
Hongbo Wu	2e47b0879c	Convert HTML to utterances	2022-11-07 19:30:20 +08:00
Hongbo Wu	d5583455ca	Forward text if html is not found in the email	2022-11-01 12:47:23 +08:00
Hongbo Wu	028b212ecb	Fix readwise api error when image_url is empty by removing image_url from api payload	2022-10-31 12:45:49 +08:00
Hongbo Wu	e87b81c5c8	Use order instead of page to indicate highlight location in Readwise	2022-10-28 17:44:14 +08:00
Hongbo Wu	1c31e5fc15	Skip sync with Readwise if no highlights	2022-10-28 15:05:16 +08:00
Hongbo Wu	bad6904909	Merge pull request #1355 from omnivore-app/fix/sync-integration ... Use the correct data format in the pub/sub queue	2022-10-27 14:15:01 +08:00
Hongbo Wu	3446bc3555	Use the correct data format in the pub/sub queue	2022-10-27 14:01:32 +08:00
Hongbo Wu	f0dc9b7d13	Fix tests	2022-10-27 12:45:32 +08:00
Hongbo Wu	2169ade48f	Persist integration in db before creating cloud task to sync with integrations	2022-10-27 11:32:56 +08:00
Hongbo Wu	614c6a2b95	Merge pull request #1347 from omnivore-app/fix/puppeteer-timeout ... Use auth token to identify client request in rate limiter	2022-10-25 17:21:13 +08:00
Hongbo Wu	57ba9afbfa	Remove debugging	2022-10-25 13:17:57 +08:00
Hongbo Wu	c9ea622461	Use auth token to identify client request in rate limiter	2022-10-25 12:42:32 +08:00
Hongbo Wu	5e68e30d5b	Add highlightPositionPercent and highlightPositionAnchorIndex in highlight api	2022-10-25 10:24:30 +08:00
Hongbo Wu	a2f31c943f	Fix tests	2022-10-20 11:31:39 +08:00
Hongbo Wu	e1de55bea0	Delete highlights, labels and other metadata when soft delete pages	2022-10-20 11:16:49 +08:00
Jackson Harper	d5c7aa2b8f	Fix linting	2022-10-19 17:36:58 +08:00
Jackson Harper	0ad8558e92	Add the iOS Saving Links article as a default read for new Omnivore users on iOS ... This also re-orders things a bit, making sure the getting started guide is always the topmost in a user's library.	2022-10-19 17:22:30 +08:00
Jackson Harper	d34e5aff04	Change the order of the new user welcome articles ... In the UI it will appear: - Getting Started - Power of Read it Later - Organize with Labels - (optional) iOS/Android/Web	2022-10-18 17:35:00 +08:00
Hongbo Wu	a7e92addb0	Create 128 * 128 proxy image for the site icon	2022-10-18 15:32:30 +08:00
Hongbo Wu	22e92256eb	Fetch subscription's favicon if the site icon is base64 encoded image	2022-10-18 15:32:30 +08:00
Jackson Harper	5312c31ea7	Dont crash if a popular read is not available on signup	2022-10-17 20:45:04 +08:00
Jackson Harper	08a1d4af91	Update the Getting started guide for new users	2022-10-17 17:11:25 +08:00
Hongbo Wu	496b28c9ee	Return deleted article in the updatesSince API response	2022-10-14 13:50:53 +08:00
Hongbo Wu	14c87a1240	Merge pull request #1298 from omnivore-app/feature/add-popular-read-when-signup ... feature/add popular read when signup	2022-10-13 13:13:13 +08:00
Hongbo Wu	19d0d314db	Fix tests	2022-10-12 18:05:17 +08:00
Hongbo Wu	7a48cccac9	Save client in request context	2022-10-12 14:30:39 +08:00
Hongbo Wu	9496827c6b	Add popular reads to new users based on client	2022-10-12 13:59:08 +08:00
Hongbo Wu	4b7950458c	Remove Redis client in api	2022-10-12 13:40:16 +08:00
Hongbo Wu	bccb949fce	Save advanced search terms in search history too	2022-10-12 13:35:01 +08:00
Hongbo Wu	d4f32f7bdd	Save search term in search history and update timestamp	2022-10-12 13:27:19 +08:00
Hongbo Wu	8f96c167f8	Save search term in search history	2022-10-12 12:30:22 +08:00
Hongbo Wu	1725f96d82	Add recentSearches API implementation and test	2022-10-12 12:20:48 +08:00
Hongbo Wu	fd1d43c104	Add search history repo methods	2022-10-12 10:49:07 +08:00
Hongbo Wu	ff12675713	Make user_id and term unique	2022-10-12 10:37:29 +08:00
Hongbo Wu	ab86450864	Rename X-Device to X-OmnivoreClient in the request headers	2022-10-12 10:27:39 +08:00
Hongbo Wu	2ba9247a98	Device info expires in 10 mins	2022-10-11 19:03:33 +08:00
Hongbo Wu	2210a38982	Set device of the request in Redis	2022-10-11 18:47:19 +08:00
Hongbo Wu	af9147d9d3	Add iOS popular reads to new iOS users	2022-10-11 16:07:49 +08:00
Hongbo Wu	606183829a	Add popular reads to new users	2022-10-11 15:44:53 +08:00
Hongbo Wu	a5c0ef19ec	Add recentSearches API GQL schema	2022-10-11 14:53:48 +08:00
Hongbo Wu	4e14518189	Replace keyword with term	2022-10-11 11:26:18 +08:00
Hongbo Wu	cd13e5874a	Create search_history table and entity class	2022-10-10 17:44:55 +08:00
Hongbo Wu	5d70d04154	If the saved tweet is a reply, we need to get the referenced tweet	2022-10-10 15:12:41 +08:00
Hongbo Wu	49ed8e279b	Calls preParseContent function in content-handler package before parsing content	2022-10-07 16:56:38 +08:00
Hongbo Wu	679977805f	Remove custom content-handler in packages/api	2022-10-07 16:56:38 +08:00
Hongbo Wu	8942398092	Change GET to HEAD	2022-09-30 12:52:09 +08:00
Hongbo Wu	9841ce7f8e	Remove parsing newsletter emails from forwarded emails	2022-09-30 12:51:23 +08:00
Hongbo Wu	b00a516737	Add other newsletter handlers	2022-09-30 12:51:23 +08:00
Hongbo Wu	57ca5ed6f8	Merge pull request #1246 from omnivore-app/feature/subscription-icon ... feature/subscription icon	2022-09-28 22:10:23 +08:00
Hongbo Wu	2f6b26d21d	Use GET request as some host do not allow HEAD requests	2022-09-28 20:18:40 +08:00
Hongbo Wu	a55ad135fd	Set 5s for the HEAD request timeout	2022-09-28 19:26:42 +08:00
Hongbo Wu	d6974f9e5f	Remove unused icon	2022-09-28 19:14:14 +08:00
Hongbo Wu	a6795b380a	Fetch favicon from url	2022-09-28 19:02:29 +08:00
Hongbo Wu	4aceac5f04	Revert "Get favicon from URL if not found from doc" ... This reverts commit 7306d65642.	2022-09-28 18:25:11 +08:00
Hongbo Wu	7306d65642	Get favicon from URL if not found from doc	2022-09-28 17:18:00 +08:00
Hongbo Wu	ee3fec9d96	improve text to speech dependency (#1250) ... * Generate .d.ts files from TypeScript for text-to-speech project * Remove module export for htmlToSpeechFile * Include test	2022-09-28 11:48:06 +08:00
Hongbo Wu	66d37e31b1	Use siteIcon	2022-09-27 21:41:01 +08:00
Hongbo Wu	6220d5d4af	Return subscription icon proxy url in the list subscriptions api response	2022-09-27 15:35:06 +08:00
Hongbo Wu	d42d4ace98	Use parsed image as subscription icon	2022-09-27 15:21:18 +08:00
Hongbo Wu	5d62d92974	Save icon in the subscription table	2022-09-27 12:12:47 +08:00
Hongbo Wu	94719a5cce	Add icon to the subscription entity	2022-09-27 12:00:13 +08:00
Hongbo Wu	710243e94e	Allow language in the query to override the page language in the speech file	2022-09-23 14:19:29 +08:00
Hongbo Wu	0239103fc4	Remove the table with text content less than 20 because these are ads	2022-09-22 22:16:03 +08:00
Hongbo Wu	56e3ccadf0	Add test case for Axios AM newsletter	2022-09-21 12:16:13 +08:00
Hongbo Wu	2c3b8535d8	Fix one morning brew newsletter parsing error by only keeping market info table in the original newsletter	2022-09-20 17:12:13 +08:00
Hongbo Wu	b995dc0500	Stop throwing error and return empty speech if there is no content in the article	2022-09-19 17:36:58 +08:00
Hongbo Wu	b03f7ebeb8	Add handler to pre-process morning brew newsletters	2022-09-15 16:21:09 +08:00
Hongbo Wu	d795fef8a7	Add handler to pre-process morning brew newsletters	2022-09-15 16:20:57 +08:00
Hongbo Wu	fa731acd74	Create new newsletter if old one is deleted	2022-09-14 19:14:58 +08:00
Hongbo Wu	e86a107a9a	Bump up image size to 160*160	2022-09-12 17:05:31 +08:00
Hongbo Wu	a01a10e3f5	Put title the first utterance	2022-09-09 12:10:29 +08:00
Hongbo Wu	06218ef5e0	Return image proxy url in Article function_resolver	2022-09-08 17:55:09 +08:00
Hongbo Wu	6350a3d53e	Make idx string type	2022-09-08 11:33:29 +08:00
Hongbo Wu	8e8a6dccd0	Change rate to string type	2022-09-07 22:55:07 +08:00
Hongbo Wu	9a62a3c31e	Add pageId to speech-file	2022-09-07 22:51:44 +08:00
Hongbo Wu	3cf3d40482	Replace speech-file with speech in the route	2022-09-07 22:48:40 +08:00
Hongbo Wu	aeb7e47d68	Make voice, secondaryVoice and priority query params	2022-09-07 18:42:46 +08:00
Hongbo Wu	f1b379b96e	Convert html to speech file and return	2022-09-07 18:00:32 +08:00
Hongbo Wu	96cbdc9f30	Replace wordsCount with wordCount	2022-09-07 15:16:16 +08:00
Hongbo Wu	5781d385ad	Return utterances in the API if output format is ssml	2022-09-07 15:16:16 +08:00
Hongbo Wu	d4dcec75d5	Return 0 for wordsCount if error	2022-09-07 15:16:16 +08:00
Hongbo Wu	97e560c443	Update wordsCount in page	2022-09-07 15:16:16 +08:00
Hongbo Wu	795ca47414	Save wordsCount when page is created	2022-09-06 10:19:58 +08:00
Hongbo Wu	d00391b593	Add wordCount to Page doc in elastic	2022-09-06 10:03:36 +08:00
Hongbo Wu	9826262ef4	Return an array of SSMLs instead of ssmlItems	2022-09-02 15:33:00 +08:00
Hongbo Wu	3953a9098e	Send SSML items in the synthesis API	2022-09-02 15:33:00 +08:00
Hongbo Wu	7353c328d9	Remove unused text-to-speech code	2022-08-29 22:29:40 +08:00
Hongbo Wu	d085c86bb6	Enqueue text to speech tasks	2022-08-29 22:23:07 +08:00
Hongbo Wu	8c8734d153	Re-enable auto synthesis in the backend	2022-08-29 22:09:09 +08:00
Hongbo Wu	a94dca8932	Prioritize synthesis over playing audio over pre-fetching	2022-08-29 15:45:21 +08:00
Hongbo Wu	4672b520e9	Merge pull request #1148 from omnivore-app/feature/separate-queues-for-synthesis ... feature/separate queues for synthesis	2022-08-29 13:43:53 +08:00
Hongbo Wu	a19ec8efea	Hardcode queue name	2022-08-29 13:20:39 +08:00
Satindar Dhillon	dfea982df7	Merge pull request #1146 from omnivore-app/feature/android-apple-login-redirect ... Android Apple Login Endpoint	2022-08-28 21:17:32 -07:00
Hongbo Wu	2e4cb7f4b0	Fix using wrong taskHandlerUrl for low priority task	2022-08-29 12:01:17 +08:00
Hongbo Wu	4af340f30e	Add the priority to request param and use the correct queue accordingly	2022-08-29 11:49:10 +08:00
Hongbo Wu	f903a92245	Use text-to-speech queue	2022-08-29 11:47:42 +08:00
Hongbo Wu	e976c9e169	Add text-to-speech queue name and location to env	2022-08-29 11:47:01 +08:00
Satindar Dhillon	3f6ade9614	verify apple token with both audience names used for the app	2022-08-28 09:55:41 -07:00
Satindar Dhillon	a720d5a5d5	set cors config for android web redirect	2022-08-28 09:48:39 -07:00
Satindar Dhillon	3d16b1d0b4	add redirect uri for android apple login	2022-08-28 09:36:30 -07:00
Hongbo Wu	7a9eca2622	Update text-to-speech endpoint	2022-08-26 17:53:26 +08:00
Hongbo Wu	5201f5d360	Add RLS to speech table	2022-08-26 16:56:33 +08:00
Hongbo Wu	b2d49d1fbf	Add dotenv	2022-08-26 16:30:09 +08:00
Hongbo Wu	5a1dbb594e	Call backend api to update speech in db	2022-08-26 15:41:06 +08:00
Jackson Harper	933fa9f142	Use a promise	2022-08-25 20:07:13 +08:00
Jackson Harper	82fdad3c80	Disable the automatic speech trigger	2022-08-25 19:54:39 +08:00
Hongbo Wu	b337735c9d	Do not throw error if failed to synthesize self-closing tags elements	2022-08-25 18:52:10 +08:00
Hongbo Wu	3945c2f887	Prevent synthesizing PDF	2022-08-25 17:01:29 +08:00
Hongbo Wu	3cea785336	Fix page id not get in auto-synthesize subscription	2022-08-25 16:09:35 +08:00
Hongbo Wu	7f83045c0c	Fix a bug	2022-08-25 16:09:35 +08:00
Hongbo Wu	99a628f03f	Fix tests	2022-08-25 16:09:35 +08:00
Hongbo Wu	a477560626	Add a feature flag for text to speech beta test and if turned on, automatically synthesize new articles	2022-08-25 16:09:35 +08:00
Hongbo Wu	00d1696f23	use complimentary voice for blockquote, hardcoded for now	2022-08-25 16:09:35 +08:00
Hongbo Wu	4f8d71b7cc	Add a pubsub subscription listening on the new article to automatically synthesize article	2022-08-25 16:09:35 +08:00
Hongbo Wu	9b0093e3d2	Add function to test if a page should be transcribed	2022-08-25 16:09:35 +08:00
Hongbo Wu	5622154e6c	Update listenedAt in the API	2022-08-25 16:09:35 +08:00
Satindar Dhillon	2a273ef504	specify multiple aud values when verifying google tokens	2022-08-24 21:00:15 -07:00
Satindar Dhillon	ba1c9c970b	use correct aud value when verifying google token from android	2022-08-23 09:26:47 -07:00
Hongbo Wu	45c1ba2d25	Update log	2022-08-22 16:45:10 +08:00
Hongbo Wu	5cfe0a14e9	Authorize user to get or create speech in the API	2022-08-22 16:44:05 +08:00
Hongbo Wu	cfdc236297	Validate token and authenticate user before setting Speech state to FAILED	2022-08-22 16:24:39 +08:00
Hongbo Wu	463f423cab	Enqueue TTS task in the get speech of article API if speech not found	2022-08-22 16:13:34 +08:00
Hongbo Wu	efce6dad94	Stop queueing transcription after calling createArticle API	2022-08-22 16:12:27 +08:00
Hongbo Wu	476e43a1f8	Set speech state to FAILED if encountered error during transcription	2022-08-22 16:11:49 +08:00
Hongbo Wu	4852777f66	Throw error if synthesizing was cancelled	2022-08-22 16:10:49 +08:00
Hongbo Wu	2298635e56	Add function to set speech to failed state	2022-08-22 16:09:56 +08:00
Hongbo Wu	4ce7352013	Merge pull request #1108 from omnivore-app/fix/failed-to-convert-speech ... Fix some articles not converted to speech correctly	2022-08-22 12:58:04 +08:00
Jackson Harper	965ffb3013	Merge pull request #1065 from omnivore-app/feature/android-login ... Android Login	2022-08-22 12:41:04 +08:00
Hongbo Wu	85d8816e1e	Replace li element with a sentence element when converting HTML to SSML	2022-08-22 12:39:06 +08:00
Satindar Dhillon	1115cf34d9	update google login client id for android	2022-08-19 19:39:41 -07:00
Satindar Dhillon	33a4ee39dc	check for android source in api mobile sign up route	2022-08-19 18:53:51 -07:00
Hongbo Wu	dba749c506	Fix some articles not converted to speech correctly	2022-08-20 00:11:46 +08:00
Hongbo Wu	0cf689ce21	Fix not getting revue newsletter url by checking all the hyper links in the table tr td	2022-08-19 17:32:29 +08:00
Hongbo Wu	0a75c52a8c	Fix getting script_exception when updating labels in highlights by checking null before updating	2022-08-19 14:36:58 +08:00
Hongbo Wu	9906e2153b	Save correct mp3 and speech marks file name in the database	2022-08-19 10:04:13 +08:00
Satindar Dhillon	1b46739a0d	add sign in option for android client on server	2022-08-18 18:42:18 -07:00
Hongbo Wu	a1171c7952	Catch authentication error	2022-08-18 19:24:38 +08:00
Hongbo Wu	576318fcbe	Add authentication to speech svc endpoint	2022-08-18 19:24:38 +08:00
Hongbo Wu	401633eda6	Return signed url	2022-08-18 19:24:38 +08:00
Hongbo Wu	249a5cda4a	Add bucket, audioFileName, speechMarksFileName, state to speech table	2022-08-18 19:24:38 +08:00
Hongbo Wu	4cf5b934eb	Add validation to request params	2022-08-18 19:24:38 +08:00
Hongbo Wu	29061cf7b1	Remove &nbsp in SSML	2022-08-18 19:24:38 +08:00
Hongbo Wu	a4a8fa9241	Replace emphasis elements in HTML	2022-08-18 19:24:38 +08:00
Hongbo Wu	c79651202d	Add synthesize SSML to speech and test	2022-08-18 19:24:38 +08:00
Hongbo Wu	dee94f7c93	Fix a bug for converting text to speech for articles with less than 5000 characters	2022-08-18 19:24:38 +08:00
Hongbo Wu	19b13d164d	Fix a bug not getting user personalization	2022-08-18 19:24:38 +08:00
Hongbo Wu	83c1930378	Split text into chunks of 5000 characters in paragraph	2022-08-18 19:24:38 +08:00
Hongbo Wu	c9d26acb25	Fix typo	2022-08-18 19:24:38 +08:00
Hongbo Wu	02ba753375	Upload speech marks file to gcs too	2022-08-18 19:24:38 +08:00
Hongbo Wu	c482ad320b	Add duration in logging	2022-08-18 19:24:38 +08:00
Hongbo Wu	b0bf4fc5ce	Create speech while requesting	2022-08-18 19:24:38 +08:00
Hongbo Wu	301746d494	Add /:id/mp3 router to redirect request to speech MP3 URL	2022-08-18 19:24:38 +08:00
Hongbo Wu	c74f2bc3fd	Enqueue text to speech task after creating article	2022-08-18 19:24:38 +08:00
Hongbo Wu	447e413605	Add function to parse HTML to SSML	2022-08-18 19:24:38 +08:00
Hongbo Wu	355e33b648	Test Mandarin	2022-08-18 19:24:38 +08:00
Hongbo Wu	e086a14ac2	Convert time to ms	2022-08-18 19:24:38 +08:00
Hongbo Wu	77570aa5ab	Synthesize text to speech with azure API	2022-08-18 19:24:36 +08:00
Hongbo Wu	94f9dd9e6e	Enqueue text to speech tasks	2022-08-18 19:23:41 +08:00
Hongbo Wu	4b42d013ca	Convert text to OGG file	2022-08-18 19:23:41 +08:00
Hongbo Wu	0419472c2e	Upload audio file with public access right	2022-08-18 19:23:41 +08:00
Hongbo Wu	dd332f5ee6	Add tests	2022-08-18 19:23:41 +08:00
Hongbo Wu	bca0f5ed59	Create Amazon Polly text to speech integration	2022-08-18 19:23:41 +08:00
Hongbo Wu	427cecf6b9	Update syncedAt after syncing	2022-08-09 09:51:31 +08:00
Hongbo Wu	134482a261	Add integration tests for integration router	2022-08-08 23:11:35 +08:00
Hongbo Wu	7ac45ca748	Add deleteIntegration API and integration tests	2022-08-08 18:34:13 +08:00
Hongbo Wu	4aa62d6031	Add list integrations API and integration tests	2022-08-08 18:12:44 +08:00
Hongbo Wu	0cf3f58258	Add integration tests for setIntegration API	2022-08-08 17:09:23 +08:00
Hongbo Wu	8070273dad	Fix tests	2022-08-05 16:10:33 +08:00
Hongbo Wu	22ce59a198	Enqueue or delete sync task when create or disable integration	2022-08-05 16:09:26 +08:00
Hongbo Wu	5a8184da3a	Add taskName in integrations table	2022-08-05 16:09:26 +08:00
Hongbo Wu	eb2ee5ab55	Add task to handle sync with integrations	2022-08-05 16:09:26 +08:00
Hongbo Wu	c0c58bf751	Only sync pages that were updated after syncedAt	2022-08-05 16:09:26 +08:00
Hongbo Wu	8e9d1fcc4f	Update integration syncedAt if successful	2022-08-05 16:09:26 +08:00
Hongbo Wu	bd73da0571	Add syncedAt field in integrations table	2022-08-05 16:09:26 +08:00
Hongbo Wu	8aa7ccdd1e	Add INTEGRATION_TASK_HANDLER_URL in env	2022-08-05 16:09:26 +08:00
Hongbo Wu	971bbfda7f	Add integrations service router to sync pages with integrations	2022-08-05 16:09:26 +08:00
Hongbo Wu	24caac2ed5	Update integration updatedAt after syncing	2022-08-05 16:09:26 +08:00
Hongbo Wu	8f50285ad5	Allow search page by highlight or label id	2022-08-05 16:09:24 +08:00
Hongbo Wu	fa97c9511c	Add Readwise create highlight API integration	2022-08-05 16:08:20 +08:00
Hongbo Wu	73bf695cf3	Add function to get highlight start location	2022-08-05 16:08:20 +08:00
Hongbo Wu	b8dbcdb2bd	Add createHighlight API integration for Readwise	2022-08-05 16:08:20 +08:00
Hongbo Wu	dd2db71876	Add createIntegration API implementation	2022-08-05 16:08:20 +08:00
Hongbo Wu	e66d047d51	Add createIntegration API to the GQL schema	2022-08-05 16:08:20 +08:00
Hongbo Wu	db27ac7d59	Add integration entity	2022-08-05 16:08:20 +08:00
Hongbo Wu	977bd1deda	Fix tests	2022-08-05 11:44:02 +08:00
Hongbo Wu	7c73982fe3	Add test for moving to the top	2022-08-04 21:39:45 +08:00
Hongbo Wu	a95f532e44	Add moveLabel api implementation	2022-08-04 21:38:36 +08:00
Hongbo Wu	b12607887e	Add moveLable api in gql Schema	2022-08-04 21:38:05 +08:00
Hongbo Wu	31d8f72840	Sort labels by position order	2022-08-04 21:37:32 +08:00
Hongbo Wu	5e2e2a2151	Merge pull request #1044 from omnivore-app/fix/email-forwarding ... Fix not getting correct sender email address from forwarding email by Gmail rules	2022-08-04 21:37:13 +08:00
Hongbo Wu	92d3ab0195	Merge pull request #1035 from omnivore-app/feature/sync-api ... feature/sync api	2022-08-04 20:33:05 +08:00
Hongbo Wu	70deb348cf	Fix not getting correct sender email address from forwarding email by Gmail rules	2022-08-04 18:50:23 +08:00
Satindar Dhillon	c076dd0f72	add mobile email sign up endpoint	2022-08-01 21:54:10 -07:00
Satindar Dhillon	c01d9a7021	server returns a pendingEmailVerification boolean in auth payload	2022-08-01 21:54:10 -07:00
Satindar Dhillon	6a100051ac	add mobile email sign in route to api	2022-08-01 21:54:10 -07:00
Hongbo Wu	336d06375a	Add updatesSince API resolver	2022-07-29 18:11:18 +08:00
Hongbo Wu	7ce183d6f8	Add soft delete of pages by updating page state to Deleted	2022-07-29 18:10:05 +08:00
Hongbo Wu	83f988bb35	Soft delete articles	2022-07-29 15:53:13 +08:00
Hongbo Wu	07c6214e88	Add DELETED state to the article	2022-07-29 14:08:35 +08:00
Hongbo Wu	b976b6098d	Add updatesSince API in GQL schema	2022-07-29 12:11:48 +08:00
Jackson Harper	6d19c8c9bc	Remove unused import	2022-07-28 11:43:16 -07:00
Jackson Harper	1525742b4e	Linting fixes	2022-07-28 11:23:27 -07:00
Jackson Harper	565da42b46	When parsing newsletters with no url dont fetch the generated url	2022-07-28 11:05:17 -07:00
Hongbo Wu	3a120b8f47	Return empty name if name not found in from header	2022-07-28 10:37:28 +08:00
Hongbo Wu	c99c1db57e	Add support to the case when from address is in Name <address> format	2022-07-28 10:26:15 +08:00
Hongbo Wu	0bdab51bbd	Merge pull request #1021 from omnivore-app/feature/save-from-email ... Save articles by sending emails to omnivore inbox	2022-07-27 20:10:31 +08:00
Hongbo Wu	6f11ccacb1	Save article from forwarding emails	2022-07-27 12:15:28 +08:00
Hongbo Wu	d184ca8d04	Add function isProbablyArticle to test if a forwarded email contains an article to save	2022-07-27 12:15:26 +08:00
Jackson Harper	779ff1f020	Reset password is a directory now	2022-07-26 15:57:40 -07:00
Jackson Harper	65689ca538	Remove extra /auth in paths	2022-07-26 15:57:18 -07:00
Jackson Harper	3c442c06ba	Set auth token cookie on success login	2022-07-26 15:56:40 -07:00
Jackson Harper	6cad41ec1e	Linting fixes	2022-07-26 15:00:31 -07:00
Jackson Harper	769c9b23b0	Use SSO redirection to set both client and server cookies with email login	2022-07-26 14:48:24 -07:00
Jackson Harper	e939f12eb1	Remove unneeded branches	2022-07-26 14:28:39 -07:00
Jackson Harper	074ec781a2	Set auth cookie after reseting password	2022-07-26 12:46:13 -07:00
Jackson Harper	dc268a2216	Handle resetting passwords	2022-07-26 12:33:56 -07:00
Jackson Harper	acd36fe9af	Move auth routes into /auth folder	2022-07-26 10:31:45 -07:00
Hongbo Wu	072b2dd9c5	Revert "Add Set-Cookie header" ... This reverts commit cb23613cdc.	2022-07-26 14:12:37 +08:00
Jackson Harper	2d9f4a0194	Redirect to the verify-email informational page after sign up	2022-07-25 23:09:01 -07:00
Hongbo Wu	cb23613cdc	Add Set-Cookie header	2022-07-26 14:06:11 +08:00
Hongbo Wu	dca9066f19	Continue fix RLS issue of updating user table	2022-07-26 13:41:02 +08:00
Hongbo Wu	29ed803ae5	Fix RLS issue of updating user table	2022-07-26 13:31:36 +08:00
Hongbo Wu	82da9e6ede	use sendgrid template for confirmation and password-reset email	2022-07-25 15:14:01 -07:00
Hongbo Wu	8d5da15064	use template id stored in env	2022-07-25 15:14:01 -07:00
Hongbo Wu	fe180c5b50	add confirm-email and password-reset email template id in env	2022-07-25 15:14:01 -07:00
Hongbo Wu	bd77a7f8ee	add test for reset password	2022-07-25 15:14:01 -07:00
Hongbo Wu	6699ec834d	add router handler for reset password	2022-07-25 15:14:01 -07:00
Hongbo Wu	286d167769	add router handler for email reset password	2022-07-25 15:14:01 -07:00
Hongbo Wu	bab96aaa1e	remove membership from user	2022-07-25 15:14:01 -07:00
Hongbo Wu	ea9d98aa95	make confirm-email a post api	2022-07-25 15:14:01 -07:00
Hongbo Wu	4c75260229	replace hard-coded email address with env var for the remaining files	2022-07-25 15:14:01 -07:00
Hongbo Wu	a87f75f319	add test for confirm-email router	2022-07-25 15:14:01 -07:00
Hongbo Wu	e4ea49c05f	add confirm-email router	2022-07-25 15:14:01 -07:00
Hongbo Wu	9ea9bd9ea1	send confirmation email to pending user when login	2022-07-25 15:14:01 -07:00
Hongbo Wu	abb1a414c1	remove login gql resolver	2022-07-25 15:13:58 -07:00
Hongbo Wu	f7053622a5	remove signup gql resolver	2022-07-25 15:13:30 -07:00
Hongbo Wu	80639311d7	improve confirmation email format	2022-07-25 15:12:56 -07:00
Hongbo Wu	068684d16b	add test for failing to send confirmation email when signup	2022-07-25 15:12:56 -07:00
Hongbo Wu	d284a3d302	fake sendMail in sign up test	2022-07-25 15:12:56 -07:00
Hongbo Wu	01c4bbad00	delete pending user if failed to send confirmation email	2022-07-25 15:12:56 -07:00
Hongbo Wu	b0a0020fc7	fix return false when sending failed	2022-07-25 15:12:56 -07:00
Hongbo Wu	4e11000ac0	add return type	2022-07-25 15:12:56 -07:00
Hongbo Wu	d187c6c993	use general sender for confirmation email	2022-07-25 15:12:55 -07:00
Hongbo Wu	d108806234	make sender address nullable env var	2022-07-25 15:12:55 -07:00
Hongbo Wu	b98f7c6ba2	reject with unknow reason if failed to send confirmation email	2022-07-25 15:12:55 -07:00
Hongbo Wu	1ea34bde62	send confirmation email to pending user	2022-07-25 15:12:55 -07:00
Hongbo Wu	7293469468	use sender email address in env	2022-07-25 15:12:55 -07:00
Hongbo Wu	bd5b289168	add verification token generation	2022-07-25 15:12:55 -07:00
Hongbo Wu	1346140a20	add sender email address in env	2022-07-25 15:12:55 -07:00
Hongbo Wu	ba73eec9e5	create pending user first	2022-07-25 15:12:55 -07:00
Hongbo Wu	35e060890e	add status to user entity	2022-07-25 15:12:55 -07:00
Jackson Harper	c113c94567	Add param validation so we dont get undefined values passed to GQL	2022-07-21 17:43:15 -07:00
Jackson Harper	ff629b2fed	Merge pull request #992 from omnivore-app/fix/email-urls ... Use email-signup instead of email-registration as path to match naming	2022-07-19 16:02:00 -07:00
Jackson Harper	e0a4478400	Merge pull request #954 from omnivore-app/feat/extension-commands ... Extension: support right click saveurl and shortcut key to save current tab	2022-07-19 15:37:18 -07:00
Jackson Harper	d6a6b1de66	Use email-signup instead of email-registration as path to match naming	2022-07-19 13:02:28 -07:00
Hongbo Wu	65f28fe708	add revue.email to the revue newsletter attribute selectors	2022-07-18 10:44:39 +08:00
Hongbo Wu	345c020dea	fix not saving loqseq weekly newsletter due to no url	2022-07-17 12:36:52 +08:00
Jackson Harper	4b8749ed43	Remove debug line	2022-07-16 12:40:54 -07:00
Jackson Harper	0bcb8a23c5	Use new API methods for saving	2022-07-16 12:17:58 -07:00
Jackson Harper	cb506fcf93	Remove extra quote from report string	2022-07-14 20:33:06 -07:00
Hongbo Wu	edd44fcafc	fix get article test	2022-07-14 17:56:55 +08:00
Hongbo Wu	404798a01e	use correct index name	2022-07-14 17:08:40 +08:00
Hongbo Wu	4eb55846fd	create elastic index if not exists in testing	2022-07-14 16:53:23 +08:00
Hongbo Wu	5a8e2c40c7	fix delete users db permission error	2022-07-13 16:08:46 +08:00
Hongbo Wu	098771f566	Merge pull request #955 from omnivore-app/fix/duplicate-save ... check if page saved by adding links before creating by extension	2022-07-13 11:57:20 +08:00
Hongbo Wu	01e95c3598	check if page saved by adding links before creating by extension	2022-07-13 11:34:20 +08:00
Hongbo Wu	cb1d64cd82	add siteName in api response	2022-07-13 10:57:54 +08:00
Hongbo Wu	970609c3b8	change size to first in query param	2022-07-13 10:46:03 +08:00
Hongbo Wu	931c519d44	typeahead searchs by siteName	2022-07-13 10:37:06 +08:00
Hongbo Wu	b68282fe97	rename typeahead to typeaheadSearch	2022-07-12 09:36:00 -07:00
Hongbo Wu	3a1f38a6f7	add typeahead api implementation	2022-07-12 09:36:00 -07:00
Hongbo Wu	dc9523e522	add searchAsYouType in elastic	2022-07-12 09:36:00 -07:00
Hongbo Wu	31ac511121	add typeahead gql schema	2022-07-12 09:36:00 -07:00
Jackson Harper	f7a023939d	Merge pull request #940 from omnivore-app/feature/allow-fetching-original	2022-07-11 22:08:27 -07:00
Hongbo Wu	c4a599d2ba	support newsletters hosted on convertkit.com	2022-07-12 11:47:38 +08:00
Jackson Harper	27e39b396d	Improve formatting for content display report emails ... With the previous formatting the `:` could be appended to the end of URLs.	2022-07-11 11:06:17 -07:00
Jackson Harper	e153966eee	Linting	2022-07-09 15:38:09 -07:00
Jackson Harper	9feeb80cf3	Allow fetching original content ... This is useful when debugging	2022-07-09 14:35:14 -07:00
Hongbo Wu	4fe81eaf92	Merge pull request #849 from omnivore-app/feature/delete-account ... Delete account API	2022-07-09 08:54:57 +08:00
Jackson Harper	e63324c802	Merge pull request #934 from omnivore-app/fix/unsubscribe-email-not-saved ... fix/unsubscribe email not saved	2022-07-08 13:47:16 -07:00
Jackson Harper	46d96b8f91	Merge pull request #932 from omnivore-app/newsletter-revue ... Add support for newsletters hosted on getrevue.co	2022-07-08 13:36:10 -07:00
Hongbo Wu	de803ebbd5	fix auth error when deleting user in db	2022-07-08 11:59:55 +08:00
Hongbo Wu	ef0c2434fb	Save unsubscribe email address or http url of newsletter emails	2022-07-07 17:14:12 +08:00
Hongbo Wu	322e135ead	delete this user pages in elastic	2022-07-07 12:32:55 +08:00
Hongbo Wu	41f43b54ab	Publish delete event	2022-07-07 12:31:46 +08:00
Hongbo Wu	efd47f3f83	add function to delete pages by param	2022-07-07 11:07:00 +08:00
Hongbo Wu	b898f55bb9	add delete account result in resolvers	2022-07-07 10:56:20 +08:00
Satindar Dhillon	e5d6386b96	return UserNotFound error if user does not exist in delete account trx	2022-07-06 09:49:01 -07:00
Satindar Dhillon	ddba50dee6	fix delete user resolver and map it to deleteAccount mutation	2022-07-06 09:38:24 -07:00
Satindar Dhillon	da76ae7aab	attempt to write resolver functions	2022-07-06 09:38:24 -07:00
Satindar Dhillon	02411347a7	regenerate graphql files	2022-07-06 09:38:24 -07:00
Satindar Dhillon	d595bfb7f1	add delete account mutation to schema.ts	2022-07-06 09:38:24 -07:00
Hongbo Wu	9599edb9fb	Add support for newsletters hosted on getrevue.co	2022-07-06 22:06:51 +08:00
Hongbo Wu	2e77241044	Check and get Revue newsletter url from email	2022-07-06 22:06:21 +08:00
Hongbo Wu	e5518e9f23	Reply to the original sender of the forwarded email	2022-07-06 21:16:55 +08:00
Hongbo Wu	c08afe6451	Update page updatedAt when updating labels	2022-06-28 12:03:37 +08:00
Hongbo Wu	00f38d1134	Update page updatedAt when updating highlights	2022-06-28 11:55:26 +08:00
Hongbo Wu	59efc5fea7	Add test	2022-06-28 10:14:15 +08:00
Hongbo Wu	bf1faaeea4	Include highlights in search page result in elastic	2022-06-28 09:56:39 +08:00
Hongbo Wu	6cae3dc9b5	Add highlights in search item	2022-06-28 09:53:48 +08:00
Hongbo Wu	c1855afc72	Update property unarchived=null instead of undefined in the page document in elastic	2022-06-26 18:36:54 +08:00
Jackson Harper	4c89323f0b	Make the content fetch GCF URL nullable	2022-06-24 11:30:52 -07:00
Jackson Harper	92c8047a66	Update environment variable names in docker	2022-06-24 10:52:13 -07:00
Hongbo Wu	75d3372426	Use gcf for low priority task	2022-06-24 10:13:14 -07:00
Hongbo Wu	e47415cf69	Add new environment variable for content-fetch url in gcf	2022-06-24 10:13:14 -07:00
Hongbo Wu	5f0cb493f2	Allow filter search result by updatedAt	2022-06-24 13:09:13 +08:00
Hongbo Wu	e96d5552fa	Fix search by is:read or is:unread getting empty result by using the correct property name	2022-06-23 12:35:48 +08:00
Hongbo Wu	9a2f4b300a	Fix a script error by deleting both labels in pages and highlights	2022-06-20 19:06:50 +08:00
Hongbo Wu	f88083e535	Merge pull request #813 from omnivore-app/advanced-search ... advanced search	2022-06-19 10:55:18 +08:00
Hongbo Wu	3f4b0f9b8c	Update resolver	2022-06-17 17:09:08 +08:00
Hongbo Wu	26e7ab178e	Fetch page savedAt, readingProgress from elastic	2022-06-17 17:08:26 +08:00
Hongbo Wu	2bcf97d2e8	Make updatedAt optional	2022-06-17 17:07:13 +08:00
Hongbo Wu	1be9c980e1	Add savedAt to searchItem and make readingProgress required in elastic	2022-06-17 16:03:49 +08:00
Hongbo Wu	e408143ea9	Add savedAt to searchItem and make readingProgress required	2022-06-17 16:02:46 +08:00
Hongbo Wu	e3c30dd076	Allow matching title, description, content keyword in elastic	2022-06-16 22:15:05 +08:00
Hongbo Wu	40bb49012a	Allow match keyword	2022-06-16 22:15:01 +08:00
Hongbo Wu	9ec11cfc22	Add title, description, content as keyword in search.ts	2022-06-16 22:14:08 +08:00
Hongbo Wu	215078bcb3	Add logseq to the allowed cors origin	2022-06-16 10:46:06 +08:00
Hongbo Wu	c5c8763767	Add readAt to searchItem	2022-06-15 09:32:53 +08:00
Hongbo Wu	6189e85b2c	Add readAt to article	2022-06-15 09:32:10 +08:00
Jackson Harper	ea4e4d6828	Use the Article type instead of Page in updatePage API ... Despite the name we are still using Article in many places.	2022-06-09 15:29:01 -07:00
Jackson Harper	3136705a75	Handle errors when creating a page save request on savePage	2022-06-09 13:52:49 -07:00
Jackson Harper	9baeb89742	Add more elastic debugging	2022-06-09 13:52:49 -07:00
Jackson Harper	85cbd4e9d0	Handle errors updating elastic when saving pages	2022-06-09 13:52:49 -07:00
Jackson Harper	05ea57f76f	Linting	2022-06-09 12:08:42 -07:00
Jackson Harper	e23eada168	Use the public GCS URL for local uploaded files	2022-06-09 11:59:01 -07:00
Jackson Harper	ef072cb2c9	When creating pages from uploaded files use their signed URL as the page URL	2022-06-09 11:51:01 -07:00
Jackson Harper	cffb5088b4	If creating a request from a file URL, dont try fuzzy matching on URL	2022-06-09 11:37:14 -07:00
Hongbo Wu	ff613287b4	Merge pull request #775 from omnivore-app/fix/delete-label ... Fix failure of lable deletion in highlight	2022-06-09 14:50:03 +08:00
Hongbo Wu	520ce0077b	Add refresh index method	2022-06-09 12:34:22 +08:00
Hongbo Wu	4f4cc51a37	Delete label by name	2022-06-09 12:33:58 +08:00
Hongbo Wu	4cf45af669	Add debug log for deleting label	2022-06-09 12:33:42 +08:00
Hongbo Wu	85317df671	Merge pull request #774 from omnivore-app/fix/missing-title ... Fallback to domain to url if no title	2022-06-09 11:48:49 +08:00
Hongbo Wu	d1d18315c8	Fallback to domain to url if no title	2022-06-09 11:30:13 +08:00
Jackson Harper	fa2b58eac0	When creating a page during an uploadfile request return its ID	2022-06-08 20:17:47 -07:00
Hongbo Wu	309c581cf5	Merge pull request #729 from omnivore-app/labels-in-highlights ... Support labels in highlights	2022-06-09 09:28:43 +08:00
Hongbo Wu	2dcab66af2	Update labels for both pages and highlights	2022-06-08 17:39:38 +08:00
Hongbo Wu	7c840f5b1e	Merge deleteLabel in page and highlight	2022-06-08 17:25:33 +08:00
Hongbo Wu	a981e13c86	Add test case for deleteLabel	2022-06-08 17:01:38 +08:00
Hongbo Wu	0d67d0fb90	Delete labels for highlights when label is deleted	2022-06-08 15:35:52 +08:00
Hongbo Wu	22c604c15b	Add deleteLabelForHighlights	2022-06-08 15:33:43 +08:00
Hongbo Wu	145867fe05	Add test for setLabelsForHighlight resolver	2022-06-08 15:25:11 +08:00
Hongbo Wu	4894b3cfd9	Add setLabelsForHighlight in function_resolvers.ts	2022-06-08 15:00:05 +08:00
Hongbo Wu	e9f4149c62	Add setLabelsForHighlight resolver	2022-06-08 14:59:46 +08:00
Hongbo Wu	eef3ee4114	Add getLabelsByIds service method	2022-06-08 14:59:27 +08:00
Hongbo Wu	93c0e53d1d	Generate setLabelsForHighlight graphql code	2022-06-08 14:59:11 +08:00
Hongbo Wu	427b58eedd	Add set labels for highlight gql schema	2022-06-06 19:15:51 +08:00
Hongbo Wu	bdda54d54a	Add set Labels For Highlight data access method	2022-06-06 19:15:03 +08:00
Hongbo Wu	aa15ef0282	Sort by expires at desc nulls last	2022-06-06 13:34:23 +08:00
Hongbo Wu	d00becc269	Add readAt to search API response	2022-06-04 22:12:59 +08:00
Hongbo Wu	fc68d51320	Support sort/filter by readAt	2022-06-04 22:12:03 +08:00
Hongbo Wu	b1d968ff23	Update readAt when saving reading progress	2022-06-04 22:11:44 +08:00
Hongbo Wu	577c6e84fb	Add readAt to page type	2022-06-04 22:11:14 +08:00
Jackson Harper	b61a8f6015	Merge pull request #747 from omnivore-app/fix/ios-sorting ... Disable sorting options that need API improvements	2022-06-03 14:04:19 -07:00
Jackson Harper	791e31d60f	Merge pull request #750 from omnivore-app/OMN-633 ... Reopening OMN-633 to rebase	2022-06-03 13:36:36 -07:00
Jackson Harper	b56c6dafa9	Disable sorting options that need API improvements	2022-06-03 12:47:43 -07:00
Jackson Harper	bf929db559	Add the updatedAt attribute to page items	2022-06-03 10:24:11 -07:00
Jackson Harper	975411d429	Merge pull request #719 from omnivore-app/fix/pdf-uploading-improvements	2022-06-02 16:47:03 -07:00
Hongbo Wu	89164a353f	Merge pull request #723 from omnivore-app/fix/tweet-not-saved ... Fix tweet not saved	2022-06-02 11:20:40 +08:00
Hongbo Wu	4f031a579c	Merge pull request #721 from omnivore-app/search-highlights-second-index ... When searching for highlights we should include document title, descr…	2022-06-02 11:05:30 +08:00
Hongbo Wu	04f7fdc31f	Add labels in highlight type	2022-06-01 19:01:26 +08:00
Hongbo Wu	195074fa49	Update search query	2022-06-01 13:49:51 +08:00
Hongbo Wu	304fe70113	Fix tests	2022-06-01 10:30:23 +08:00
Jackson Harper	11f1b78c45	Remove changes to upload.ts	2022-05-31 15:26:33 -07:00
Jackson Harper	0e10d5630d	Rebase and merge	2022-05-31 15:11:10 -07:00
Hongbo Wu	404805e0c0	Make async calls to parse()	2022-05-31 22:51:00 +08:00
Hongbo Wu	74054cbdbd	Merge pull request #716 from omnivore-app/api-token ... Api tokens	2022-05-31 14:22:23 +08:00
Jackson Harper	6f35740272	Linting fixes	2022-05-30 22:38:24 -07:00
Jackson Harper	53c423b714	WIP: upload PDFs using background task	2022-05-30 22:30:43 -07:00
gitstart-omnivore	d42b0c244c	fixed merge conflicts	2022-05-30 12:29:49 +00:00
Hongbo Wu	a04bfbb83e	When searching for highlights we should include document title, description, content as secondary indexes	2022-05-29 22:43:21 +08:00
Hongbo Wu	c0bb67a6b5	Verify api key for routers too	2022-05-28 22:00:01 +08:00
Jackson Harper	987336f438	Merge pull request #712 from omnivore-app/fix/ios-pdf-handling ... Better PDF upload handling on iOS	2022-05-27 08:43:07 -07:00
Hongbo Wu	6e16631f3b	Resolve conflicts	2022-05-27 18:37:10 +08:00
Hongbo Wu	fa9c5b3efe	Add test for list api keys	2022-05-27 18:36:18 +08:00
Hongbo Wu	b7206457b4	Add test for revoke api key	2022-05-27 18:36:17 +08:00
Hongbo Wu	0f2a55d274	Add list/delete api key resolver	2022-05-27 18:35:38 +08:00
Hongbo Wu	fd4fc8a4e7	Add list/delete api key in schema	2022-05-27 18:34:36 +08:00
Hongbo Wu	333b0259ba	Fix tests	2022-05-27 18:33:49 +08:00
Hongbo Wu	9051a3f43a	Update api key validation	2022-05-27 18:33:49 +08:00
Hongbo Wu	a09588d3a7	Update generateApiKey resolver	2022-05-27 18:33:49 +08:00
Hongbo Wu	849adf84d4	Add entity class	2022-05-27 18:33:49 +08:00
Jackson Harper	0cabd26344	Dont send the normalized URL to the content-fetch service ... URLs need to be sent to contentFetch exactly as they were received, this is because many apps will use signed URLs for PDFs being served from GCP or AWS. A signed URL needs to keep all its query params in the same order to validate.	2022-05-26 21:40:40 -07:00
Jackson Harper	93b095818d	Linting fixes	2022-05-26 21:40:40 -07:00
Jackson Harper	cb98a9cf86	Make clients opt into creating a page when uploading a file	2022-05-26 21:40:40 -07:00
Hongbo Wu	26681e5ecd	Add get/list/delete webhook resolver tests	2022-05-26 14:52:01 +08:00
Hongbo Wu	a75ce3c44d	Add get/list/delete webhook resolver	2022-05-26 14:52:00 +08:00
Hongbo Wu	cc07c00a83	Add get/list/delete webhook gql schema	2022-05-26 14:51:11 +08:00
Hongbo Wu	c9c6f83e65	Add tests for trigger endpoint	2022-05-26 14:51:11 +08:00
Hongbo Wu	7b0ebb9dbf	Add pubsub subscription to trigger webhook	2022-05-26 14:51:11 +08:00
Hongbo Wu	7157f38738	Add test	2022-05-26 14:51:11 +08:00
Hongbo Wu	6d0ce30a44	Add setWebhook resolver	2022-05-26 14:51:10 +08:00
Hongbo Wu	5528d1d349	Replace createWebhook with setWebhook	2022-05-26 14:50:38 +08:00
Hongbo Wu	f61f61312c	Add webhook entity	2022-05-26 14:50:10 +08:00
Hongbo Wu	a655275754	Add unique constraint on user_id and event_types	2022-05-26 14:50:10 +08:00
Hongbo Wu	aaab02c9d8	Add createWebhook gql schema	2022-05-26 14:50:08 +08:00
Hongbo Wu	7d4d1d7b67	Parse language in readability	2022-05-26 10:55:21 +08:00
Hongbo Wu	0ef9bb3944	Allow search by language in elastic	2022-05-26 10:55:20 +08:00
Hongbo Wu	87b0f683e1	Add language term in search body	2022-05-26 10:55:04 +08:00
Hongbo Wu	6e06429636	Add language keyword in search query	2022-05-26 10:55:02 +08:00
Hongbo Wu	1eeb5617a5	Search by language	2022-05-26 10:54:04 +08:00
Hongbo Wu	dd394144cd	Add language in schema	2022-05-26 10:52:19 +08:00
Hongbo Wu	a702993485	Add search by language query	2022-05-26 10:52:18 +08:00
Hongbo Wu	3e0e8b486f	Add language in searchitem	2022-05-26 10:52:18 +08:00
Hongbo Wu	be91562ca1	Add language in page	2022-05-26 10:52:18 +08:00
Jackson Harper	d704fad2ff	Merge pull request #701 from omnivore-app/feature/add-popular-reads ... API to add recommended (popular) reads to a users library	2022-05-24 21:09:24 -07:00
Hongbo Wu	f4c65d7fe1	Merge pull request #698 from omnivore-app/sort-by-publish-time ... Support sort or filter by publish time	2022-05-25 10:41:49 +08:00
Jackson Harper	a20ba60053	Add popular reads for onboarding	2022-05-24 15:31:55 -07:00
Jackson Harper	ea50f51644	Update to use image proxy URLs in parsed content	2022-05-24 13:43:57 -07:00
Jackson Harper	ed5dfde1a8	Linting fixes, remove debug	2022-05-24 13:06:40 -07:00
Jackson Harper	2b14d51ec6	API to add recommended (popular) reads to a users library ... This API is used in onboarding	2022-05-24 12:05:39 -07:00
Hongbo Wu	a35b096d73	Support sort and filter by published in resolver	2022-05-24 18:25:32 +08:00
Hongbo Wu	61065efd2b	Add published sort param in elastic query	2022-05-24 18:24:30 +08:00
Hongbo Wu	641f191fcd	Add published sort param in schema	2022-05-24 18:23:30 +08:00
gitstart-omnivore	75728cc144	fixing diff	2022-05-24 07:04:54 +00:00
Jackson Harper	2d2495a10e	Merge pull request #685 from omnivore-app/fix/pdf-api-simplification ... Simplify the API to create pages from PDFs	2022-05-21 21:19:30 -07:00
Jackson Harper	e5bc68ca9e	Simplify the API to create pages from PDFs ... This creates a stub page when an upload is initiated, so the SaveFile (or create article from file) APIs do not need to be called. The PDF uploaded trigger fired by GCS will mark the upload as completed and update the page status.	2022-05-21 12:05:45 -07:00
Jackson Harper	17e6f4091d	Merge pull request #638 from omnivore-app/feature/subscription-api ... Feature/subscription api	2022-05-20 15:34:48 -07:00
Hongbo Wu	2ee95a1c14	Fix cannot convert null to object error message	2022-05-20 20:38:30 +08:00
Hongbo Wu	444c21ad09	Fix labels test	2022-05-19 18:12:14 +08:00
Hongbo Wu	34349e925c	Add unit test	2022-05-19 16:36:35 +08:00
Hongbo Wu	b8225fc8c8	Fix labels not being updated in elastic	2022-05-19 12:08:50 +08:00
Jackson Harper	eadeccce81	Linting fixes	2022-05-18 11:31:42 -07:00
Jackson Harper	004c766588	If parsing fails, attempt adding <html> wrappers to a document ... LinkedDom seems less forgiving and expects the outerHTML of a document, however older extension versions still send innerHTML.	2022-05-18 10:55:31 -07:00
Hongbo Wu	d68549bcb7	Remove unused code	2022-05-18 15:52:31 +08:00
Hongbo Wu	d9d4deb14c	Update Money Stuff sign up url	2022-05-18 11:50:43 +08:00
Hongbo Wu	5b36f9270e	Update milk road query params	2022-05-17 22:43:00 +08:00
Hongbo Wu	13453e7b7f	Register subscribe resolver	2022-05-17 21:26:46 +08:00
Hongbo Wu	568472cc3c	Calling HTTP requests to subscribe to these newsletters	2022-05-17 21:15:25 +08:00
Hongbo Wu	4699458457	Implement subscribe api	2022-05-17 18:46:57 +08:00
Hongbo Wu	1da8b75c81	Add a subscribeHandler	2022-05-17 18:46:44 +08:00
Hongbo Wu	51dd9a5733	Return a list of subscriptions	2022-05-17 18:45:52 +08:00
Hongbo Wu	c30ef093b8	Update unsubscribe api	2022-05-17 16:47:34 +08:00
Hongbo Wu	906b2d28a0	Add subscribe API schema	2022-05-17 16:23:36 +08:00
Hongbo Wu	e76fb02f43	Fix window is not defined for parsing code blocks	2022-05-17 11:01:55 +08:00
Jackson Harper	7bfb8cfee4	Merge pull request #597 from omnivore-app/remove-chrome-aws-lambda ... Optimize puppeteer and remove chrome-aws-lambda dependencies	2022-05-13 16:12:24 -07:00
Jackson Harper	8b37a475c8	Merge pull request #612 from omnivore-app/fix/reminder-id-type ... Use pageId instead of articleSavingRequestId in reminders	2022-05-13 10:47:46 -07:00
Jackson Harper	d3b65c0ff8	Merge pull request #611 from omnivore-app/fix/elastic-exception ... Stop throwing document not found exception to sentry	2022-05-13 09:05:01 -07:00
Hongbo Wu	b306ca6e2e	Fix UpdateLabelResult undefined	2022-05-13 17:38:51 +08:00
Hongbo Wu	45213e7b54	Fix reminder trigger	2022-05-13 17:07:10 +08:00
Hongbo Wu	a51eba67ba	get reminder by page id	2022-05-13 16:17:25 +08:00
Hongbo Wu	d821f26440	create reminder with page id	2022-05-13 16:12:08 +08:00
Hongbo Wu	40cbfc9058	Use reminder by elasticPageId	2022-05-13 16:01:23 +08:00
Hongbo Wu	f51dbafdfc	Stop throwing document not found exception to sentry	2022-05-13 11:40:08 +08:00
Jackson Harper	eee116e71d	Add an analytics event when links are removed	2022-05-12 10:55:12 -07:00
Hongbo Wu	adfaf6b8ce	Use newsletter's title and author as page's ones	2022-05-12 18:08:38 +08:00
Hongbo Wu	1b8850ed33	Fix tests	2022-05-12 17:41:11 +08:00
Hongbo Wu	e1e0ddf7fc	Merge pull request #582 from omnivore-app/optimize-parsing ... Optimize parsing	2022-05-12 11:07:52 +08:00
Hongbo Wu	602d141dec	Rename doc to dom	2022-05-12 11:00:32 +08:00
Hongbo Wu	5dca02873d	Revert originalArticleUrl change	2022-05-12 10:45:13 +08:00
Hongbo Wu	8f75c43658	Add siteName in search result	2022-05-12 10:30:47 +08:00
Hongbo Wu	76aef35423	Remove www. from the siteName in the search api response	2022-05-11 23:07:34 +08:00
Hongbo Wu	a78a6c6ba4	Replace DomWindow with Document in handlers	2022-05-10 17:01:23 +08:00
Hongbo Wu	6a57281e74	Remove DomWindow usage	2022-05-10 17:00:56 +08:00
Hongbo Wu	acc7654a2f	Replace jsdom with linkedom	2022-05-10 16:59:09 +08:00
Hongbo Wu	5698790288	Pass url to readability	2022-05-10 16:53:45 +08:00
Hongbo Wu	5007b11a3c	Add url to readability option	2022-05-10 16:52:10 +08:00
Hongbo Wu	59a2639b7d	Reduce http call to get jsonld data if title or content or sitename or byline exists	2022-05-09 13:45:45 +08:00
Hongbo Wu	a457c9d128	Update article content only when code blocks exist	2022-05-09 13:45:45 +08:00
Hongbo Wu	eaad96acdd	Return parsed dom back to backend	2022-05-06 12:29:08 +08:00
Hongbo Wu	5f5076e864	Highlight code element without reinitialize jsdom	2022-05-06 12:20:54 +08:00
Hongbo Wu	7c6b810522	Remove redundant JSDOM	2022-05-06 10:53:36 +08:00
Hongbo Wu	93ab6bc87a	Add includePending=true in search API	2022-05-04 21:28:06 +08:00
Hongbo Wu	bc1ed3f054	Check state in articleSavingRequest	2022-05-04 19:33:13 +08:00
Hongbo Wu	9e3db0e053	Fix a bug to have multiple pages with the same url in lib	2022-05-04 19:12:22 +08:00
Hongbo Wu	afe8b6e948	make savedAt a required field in page	2022-05-04 17:54:59 +08:00
Hongbo Wu	a42bb7ab38	Replace createdAt with savedAt	2022-05-04 12:59:07 +08:00
Hongbo Wu	98ea1016d5	Remove <p> tag in description	2022-05-04 12:50:22 +08:00
Hongbo Wu	9048ccbab3	Show failed pages in library	2022-05-04 12:46:57 +08:00
Hongbo Wu	ca0f58ee22	Update page content and state if failed to parse	2022-05-04 12:21:41 +08:00
Hongbo Wu	80d0146f72	Show unable to parse if page is failed to save within 30 seconds	2022-05-04 11:55:16 +08:00
Jackson Harper	c8c4377346	Merge pull request #531 from omnivore-app/fix/saving-analytics ... Add more detailed analytics for saving. Remove intercom events	2022-05-02 21:14:18 -07:00
Jackson Harper	95b163a437	Prettier clean up	2022-05-02 20:58:05 -07:00
Jackson Harper	ab96bb09ab	Add more detailed analytics for saving. Remove intercom events	2022-05-02 20:41:10 -07:00
Jackson Harper	b649d583b7	Linting changes	2022-05-02 15:58:40 -07:00
Jackson Harper	453b301e74	Allow looking up articles by ID ... iOS needs to be able to lookup an article by its ID not its slug, because the slug can change once saved.	2022-05-02 15:48:21 -07:00
Jackson Harper	61b7ac3cb2	Enable graphql introspection when running locally	2022-05-02 09:44:13 -07:00
Hongbo Wu	77ca0ceb59	Show empty description if not exist	2022-04-29 15:50:39 +08:00
Hongbo Wu	2b70d480d2	Remove article saving request (#493) ... * Add state and taskName in elastic page mappings * Add state and taskName in elastic page interface * Create page with PROCESSING state before scrapping * Update createArticleRequest API * Fix tests * Add default state for pages * Update createArticle API * Update save page * Update save file * Update saving item description * Show unable to parse content for failed page * Fix date parsing * Search for not failed pages * Fix tests * Add test for saveUrl * Update get article saving request api * Update get article test * Add test for articleSavingRequest API * Add test for failure * Return new page id if clientRequestId empty * Update clientRequestId in savePage * Update clientRequestId in saveFile * Replace article with slug in articleSavingRequest * Add slug in articleSavingRequest response * Depreciate article * Use slug in web * Remove article and highlight fragments * Query article.slug on Prod * Show unable to parse description for failed page * Fix a bug having duplicate pages when saving the same url multiple times * Add state in response * Rename variables in removeArticle API * Rename state * Add state in response in web * Make state an enum * Open temporary page by link id * Use an empty reader view as the background for loading pages * Progressively load the article page as content is loaded * Add includePending flag in getArticles API * Set includePending = true in web * Add elastic update mappings in migration script * Add elastic mappings in docker image * Move index_settings.json to migrate package * Remove elastic index creation in api * Move elastic migrations to a separate directory * Remove index_settings from api docker image Co-authored-by: Jackson Harper <jacksonh@gmail.com>	2022-04-29 13:41:06 +08:00
Hongbo Wu	ccb79845bc	Fix subscriptions issues (#501) ... * Replace fetch with axios * Add url in subscription * Remove url in subscription * Unsubscribe all newsletters before deleting that email * Continue unsubscribing others if one got error	2022-04-29 11:16:46 +08:00
Hongbo Wu	d1c2721b95	Add expiration date to api key generation (#495) ... * Add exp to jwt token * Add expiredAt in generate api key input * Check exp in claim * Add tests	2022-04-28 09:09:40 +08:00
Hongbo Wu	bcbda50daa	Add basic UI for subscriptions	2022-04-22 12:17:09 -07:00
Hongbo Wu	3e35fb1aca	Email content display report	2022-04-22 19:14:59 +08:00
Hongbo Wu	71aa66ea9c	Allow search for subscription:name (#461)	2022-04-22 08:47:41 +08:00
Jackson Harper	0a4d55ef15	Increase the rate limit ... Turns out 10 is very low because we make CORs requests from the browser and SWR will make background requests for some objects like viewer. I was hitting the limit while testing search.	2022-04-21 14:59:57 -07:00
Hongbo Wu	dca6461b62	Add unsubscribe resolver (#453) ... * add unsubscribe resolver * add subscription to articles api response * revert yarn.lock changes	2022-04-21 13:18:15 +08:00
Hongbo Wu	83cdec42f7	Add rate limiter for api request (#454) ... * Add rate limiter for api request * apply rate limit on Demo/Prod	2022-04-21 09:41:33 +08:00
Hongbo Wu	5c7f4a22bf	Save subscriptions when saving newsletters (#447) ... * parse unsub mailTo or httpUrl and send to pubsub * save subscription from newsletter * fix tests	2022-04-20 09:24:40 +08:00
Jackson Harper	48e36c8635	Merge pull request #393 from omnivore-app/OMN-382 ... [Omn-382] - Updated landing page	2022-04-19 11:15:31 -07:00
Hongbo Wu	d7a2659fc4	fix searching for multiple labels (#444)	2022-04-19 12:13:47 +08:00
Hongbo Wu	1117a0c575	Feature/subscription list resolver (#432) ... * add subscriptions table * add listSubscriptions schema * add listSubscriptions resolver	2022-04-19 11:08:43 +08:00
Jackson Harper	c2fd58a809	Make landing a restricted username page	2022-04-18 15:22:59 -07:00
Hongbo Wu	0b2b93384f	fix return undefined as updateLabelResult	2022-04-18 20:45:06 +08:00
Jackson Harper	d33d383891	Merge pull request #422 from omnivore-app/fix/reminder-data-model-error ... Fix reminders resolver getting pages from postgres	2022-04-14 14:10:48 -07:00
Jackson Harper	c80ce4da10	Consistent naming convention for events	2022-04-14 10:31:10 -07:00
Jackson Harper	2c9c2a2adf	Merge pull request #394 from omnivore-app/header-and-reader-improvements ... Add margins on wider screens, keyboard commands to adjust	2022-04-13 21:26:24 -07:00
Hongbo Wu	c0966d908f	upload highlight and label to gcs if successfully updated in elastic (#420)	2022-04-14 12:14:15 +08:00
Hongbo Wu	88d8ca7b7d	fix reminders resolver getting page from postgres	2022-04-14 12:09:17 +08:00
Jackson Harper	e282fc8316	Merge pull request #417 from omnivore-app/feature/upload-highlight-gcs ... Upload highlights and labels data to GCS	2022-04-13 13:41:29 -07:00
Hongbo Wu	0b0b521fab	use generic method for uploading	2022-04-13 22:21:20 +08:00
Hongbo Wu	a806f55233	upload highlight to gcs	2022-04-13 21:18:58 +08:00
Hongbo Wu	8ba1be456c	fix sanitizer invalid name	2022-04-13 13:23:21 +08:00
Hongbo Wu	3b831fecb8	test both max length and pattern if declared	2022-04-13 12:20:41 +08:00
Hongbo Wu	928ddd69a9	validate color input in label as rgb hex value	2022-04-13 12:17:12 +08:00
Jackson Harper	73ef8fef09	Rename linkId to pageId in SetLabelsInput	2022-04-12 12:46:32 -07:00
Jackson Harper	d05f091876	Add debug logging for GQL errors	2022-04-12 12:23:55 -07:00
Jackson Harper	6c5e88cf12	Merge pull request #405 from omnivore-app/feature/label-max-length ... Set label name max length = 64 char	2022-04-12 11:14:20 -07:00
Hongbo Wu	ae0d1dd2ee	Feature/search highlights backend (#395) ... * add highlight mappings * return highlight in resolvers * temporarily skip highlight tests * add test for getting highlights * update merge highlight * separate elastic methods * roll back merge highlight test * add highlight to elastic script * update delete highlight in elastic * migrate highlight data from postgres to elastic * rescue not found exception when page is not found in the migration script * exclude highlights in searching pages results * search pages with highlights only with has:highlight query * add search endpoint to search pages or highlights * reduce code smell in search api * fix rebase error * fix tests * add test for search highlight * add test for new search endpoint * add labels to search results * update schema * update search query * fix update/share highlights * fix rebase error * fix tests * add highlight model in elastic * add savedAt and publishedAt date range in search query * add sort by updated and recently read * fix tests * close db connection when tests are done * test github action * revert github action test * fix rebase error * add docker-compose for api-test * remove unused env * remove highlights with no page attached to * allow get_articles resolver to search for query so we can merge it without web changes	2022-04-12 12:31:08 +08:00
Hongbo Wu	2ebdaba780	add generate api key api and test (#392) ... * add generate api key api and test * test if user can make api call with the api key	2022-04-12 12:11:45 +08:00
Hongbo Wu	bb665a704b	make label name <= 64 char length	2022-04-12 12:08:02 +08:00
Hongbo Wu	b4420839bd	fix: fail to search by excluding labels with capital letter (#400) ... * fix: fail to search by excluding labels with capital letter * make all the excluded labels lowcased * Revert "make all the excluded labels lowcased" This reverts commit 866bed4080.	2022-04-11 14:05:21 +08:00
Jackson Harper	48ca13818f	Prettier fixes	2022-04-06 10:45:01 -07:00
Jackson Harper	57f865c9a2	Update to typeorm 0.3	2022-04-06 10:37:13 -07:00
Jackson Harper	1cdb7753c3	Update typeorm usage	2022-04-06 09:56:32 -07:00
Jackson Harper	06360a5993	Merge branch 'main' into OMN-190	2022-04-06 09:24:49 -07:00
Hongbo Wu	d1269f815e	identify a newsletter by checking both list-url and unsubscribe-url in headers and email sender address	2022-04-06 20:59:14 +08:00
Hongbo Wu	7569e988bf	upgrade typeorm to 3.0 (#359) ... * upgrade typeorm to 3.0 * use new datasource object in typeorm 3 * fix tests * fix tests * migrate before creating connection * fail the test if migration failed	2022-04-06 10:32:41 +08:00
Jackson Harper	9845a49040	Remove semicolon	2022-04-04 08:17:46 -07:00
Jackson Harper	c8ec7d636e	Use require to import address parser function	2022-04-04 08:11:26 -07:00
Jackson Harper	92925b6808	Prettier fixes	2022-04-03 10:21:16 -07:00
Jackson Harper	da28998130	Pull ul list out of newsletter blurb ... next/react doesnt want child elements of the paragraphs Improve formatting Improve wording Use buttons in the subscribe directly blocks Simplify docs on setting up forwarding rules Add extra padding on bottom of help docs Remove unused style Add emails help page Improve formatting Prefetch page content on iOS Reduce the reader overly length now that items are precached Add invalidation when highlights are added to items fix missing index_settings.json file in api dockerfile for creating elastic index (#363) Handle full email address objects in the to param from sendgrid These come in a format like: "jacksonh-dfdf@inbox.omnivore.app" <jacksonh-dfdf@inbox-demo.omnivore.app> New IDs for short highlights because they dont cascade delete now Testing CI issues Simplify test CI test Use promises for async tests Temporarily remove test to debug CI Re-enable re-enable test, return error Specify a userId when looking up saved email pages create a unique url for newsletters without a URL Use 500ms on page test timeouts Increase timeout Dont use deep equal to match newsletter label Run just the labels API Run against just the newsletter emails Run without the page tests Fix Set the allow uncaught flag Remove highlight tests Remove newsletters tests more resolver tests Remove newsetter tests Comment out resolver tests Use nock for external requests in tests Specify puppeteer url for tests Comment out more tests uncomment tests re-enable re-enable email test Re-disable Re-enable one pdf attachment test Re-disable pdf attachment test Use promises on setTimeout tests rm label tests mv label tests into a context Comment out pdf tests Comment out pdf tests Async test Async wrappers Delay when creating test pages More debugging Unique short ids Remove potentially problematic test Fetch page before returning for test handler in before block more debugging More debugging Move errors checks into contexts Use a context when saving newsletters to force index refresh Prettier fix Fix newsletter label check, remove setTimeout Re-enable test timeout on pdf router handler Fix method call comment out PDF test Unique fake username Comment out PDF test Debugging signed urls Re-enable New email pdf test PDF tests Comment out pdf test Add nock stubs for email URLs Use full address for PDF test Remove debug Use full email addresses	2022-04-02 16:56:24 -07:00
gitstart-omnivore	1978a92a1b	Effected review corrections	2022-03-31 13:03:50 +00:00
Jackson Harper	a8faaf835c	Remove debug sort on labels	2022-03-30 14:13:26 -07:00
Jackson Harper	2c15516d96	Order labels by createdAt so they maintain a consistent order	2022-03-30 13:28:37 -07:00
gitstart-omnivore	11db7be473	[Omn-190] - [Settings View] - Labels	2022-03-30 12:24:29 +00:00
Hongbo Wu	6d405432af	add site_name and site_icon to page model and return in resolver (#341) ... * add site_name and site_icon to page model and return in resolver * fix tests	2022-03-30 10:43:10 +08:00
Jackson Harper	2bada0b9df	Merge pull request #293 from omnivore-app/feature/priority-by-rate-limit ... Use lower priority queue for large imports	2022-03-28 13:19:35 -07:00
Jackson Harper	d9af397b9c	Merge pull request #331 from omnivore-app/fix/update-page-after-deletion ... fix document_missing_exception being thrown to sentry when updating page in elastic	2022-03-28 10:30:31 -07:00
Hongbo Wu	fd18affaec	fix document_missing_exception being thrown to sentry when updating page in elastic	2022-03-28 21:16:33 +08:00
Hongbo Wu	b6c04f3508	fix saving report item by linking to elastic page	2022-03-28 20:38:54 +08:00
Jackson Harper	1cdbfaca26	Merge pull request #294 from omnivore-app/feature/add-created-or-updated_at-to-newsletter_email ... add created_at and updated_at to newsletter emails and sort by create…	2022-03-25 09:10:54 -07:00
Hongbo Wu	27735f7310	add page id in updated page json file in gcs (#319)	2022-03-25 13:50:43 +08:00
Hongbo Wu	e7203bebb5	add a function_resolver to return originalArticleUrl from article.url (#317) ... * add a function_resolver to return originalArticleUrl from article.url * add a test	2022-03-25 11:25:30 +08:00
Hongbo Wu	6597cc37b6	add created_at and updated_at to newsletter emails and sort by created_at desc	2022-03-22 18:43:47 +08:00
Hongbo Wu	acec3b1acb	Use lower priority queue for large imports ... * add a count query in elastic * if priority is not set, check the rate limit of user's libary - if 5 articles added in the last minute: use low queue - default: use normal queue	2022-03-22 18:08:08 +08:00
Hongbo Wu	1893e36375	Fix uploading endpoint returns 504 (#289) ... * add timeout = 60s for uploading to private bucket * add debug logs * send response to client to close connection * add tests for upload * add PUBSUB_VERIFICATION_TOKEN in .env.test	2022-03-22 14:20:40 +08:00
Hongbo Wu	3e9063c145	return empty content instead of null in search result (#291)	2022-03-22 13:34:44 +08:00
Hongbo Wu	bee3b3c6fa	fix article_id type mismatch (uuid => varchar) when saving highlights in postgres by setting article_id to be undefined (#285)	2022-03-22 11:48:01 +08:00
Hongbo Wu	4ca7c89622	format date string as yyyy-mm-dd (#288)	2022-03-22 11:46:39 +08:00
Hongbo Wu	c15efe6aad	Fix sentry log size error (#287) ... * reduce sentry log size error by partially logging page created in elastic * remove some debugging logs * remove more debugging logs	2022-03-22 11:45:58 +08:00
Jackson Harper	f9bce1932c	Print the logging level on startup	2022-03-21 09:31:05 -07:00
Hongbo Wu	77050905a5	retry max three times if update labels is conflicted	2022-03-18 10:39:11 +08:00
Hongbo Wu	9d8d2e40d7	reduce time to save reading progress in elastic update	2022-03-18 10:29:01 +08:00
Hongbo Wu	ac567f7e71	retry max three times if update is conflicted	2022-03-18 10:17:20 +08:00
Jackson Harper	7910565dbf	Merge pull request #251 from omnivore-app/fix/jsonld-encoding ... Use html decoding when getting values from fetched oembed	2022-03-17 16:53:21 -07:00
Hongbo Wu	563131ea23	Fix/newsletter label not add (#256) ... * default sort by savedAt to avoid recent read page to appear on top of the list * revert lint on generated code * fix elastic update script syntax error	2022-03-17 13:22:38 +08:00
Hongbo Wu	a2ce98229e	default sort by savedAt to avoid recent read page to appear on top of… (#254) ... * default sort by savedAt to avoid recent read page to appear on top of the list * revert lint on generated code	2022-03-17 10:54:00 +08:00
Hongbo Wu	af037a2837	make readingProgress required in the elastic page data (#253) ... * make readingProgress required in the elastic page data * delete readingProgress from function_resolvers because we have stored them in elastic	2022-03-17 10:06:21 +08:00
Jackson Harper	ff1200f3a1	Use html decoding when getting values from fetched oembed ... If we fetch oembed data from an external source, instead of handling it in readabilityjs we need to html decode it.	2022-03-16 15:29:42 -07:00
Jackson Harper	89f3719ba3	Fix linting	2022-03-16 14:23:39 -07:00
Jackson Harper	0314d8cf17	Add some extra logging	2022-03-16 13:53:55 -07:00
Jackson Harper	dc27c0c7a4	Return 0 if looking up page reading progress fails	2022-03-16 13:33:36 -07:00
Jackson Harper	27143cbedf	Remove function handlers for reading progress ... These shouldn't be needed anymore because the values are stored in elastic.	2022-03-16 13:11:36 -07:00
Hongbo Wu	07dcd5da26	add label to page only if not exists	2022-03-16 23:00:37 +08:00
Hongbo Wu	1c4dcd7b00	automatically update updatedAt when page is updated in elastic	2022-03-16 22:28:39 +08:00
Hongbo Wu	f412758040	add Newsletter label to the page created by newsletters email	2022-03-16 18:40:05 +08:00
Hongbo Wu	27157006c1	use private bucket to upload page events (#244) ... * use private bucket to upload page events * fix tests * add GCS_UPLOAD_PRIVATE_BUCKET in test env * allow GCS_UPLOAD_PRIVATE_BUCKET to be empty	2022-03-16 14:39:07 +08:00
Jackson Harper	e652a6ea8c	Rebased version of the elastic PR (#225) ... * Add elastic to our docker compose * add AND/OR/NOT search operations * add elastic and create article in elastic * change error code when elastic throws error * add search pages in elastic * add search by labels * Add elastic to GitHub Action * Update elastic version * Fix port for elastic * add url in search query * Set elastic features when running tests * add debug logs * Use localhost instead of service hostname * refresh elastic after create/update * update search labels query * add typescript support * search pages in elastic * fix search queries * use elastic for saving page * fix test failure * update getArticle api to use elastic * use generic get page function * add elastic migration python script * fix bulk helper param * save elastic page id in article_saving_request instead of postgres article_id * fix page archiving and deleting * add tests for deleteArticle * remove custom date type in elastic mappings which not exist in older version of elastic * fix timestamp format issue * add tests for save reading progress * add tests for save file * optimize search results * add alias to index * update migration script to receive env var as params * Add failing test to validate we don't decrease reading progress This test is failing with Elastic because we aren't fetching the reading progress from elastic here, and are fetching it from postgres. * Rename readingProgress to readingProgressPercent This is the name stored in elastic, so fixes issues pulling the value out. * Linting * Add failing test for creating highlights w/elastic This test fails because the highlight can't be looked up. Is there a different ID we should be passing in to query for highlights, or do we need to update the query to look for elastic_id? * add tests code coverage threshold * update nyc config * include more files in test coverage * change alias name * update updateContent to update pages in elastic * remove debug log * fix createhighlight test * search pages by alias in elastic * update set labels and delete labels in elastic * migration script enumeration * make BULK_SIZE an env var * fix pdf search indexing * debug github action exit issue * call pubsub when create/update/delete page in elastic * fix json parsing bug and reduce reading data from file * replace a depreciated pubsub api call * debug github action exit issue * debug github action exit issue * add handler to upload elastic page data to GCS * fix tests * Use http_auth instead of basic_auth * add index creation and existing postgres tables update in migration script * fix a typo to connect to elastic * rename readingProgress to readingProgressPercent * migrate elastic_page_id in highlights and article_saving_request tables * update migration script to include number of updated rows * update db migration query * read index mappings from file * fix upload pages to gcs * fix tests failure due to pageContext * fix upload file id not exist error * Handle savedAt & isArchived attributes w/out quering elastic * Fix prettier issues * fix content-type mismatching * revert pageId to linkId because frontend was not deployed yet * fix newsletters and attachment not saved in elastic * put linkId in article for setting labels * exclude orginalHtml in the result of searching to improve performace * exclude content in the result of searching to improve performace * remove score sorting * do not refresh immediately to reduce searching and indexing time * do not replace the backup data in gcs * fix no article id defined in articleSavingRequest * add logging of elastic api running time * reduce home feed pagination size to 15 * reduce home feed pagination size to 10 * stop revalidating first page * do not use a separate api to fetch reading progress * Remove unused comment * get reading progress if not exists * replace ngram tokenizer with standard tokenizer * fix tests * remove .env.local * add sort keyword in searching to sort by score Co-authored-by: Hongbo Wu <hongbo@omnivore.app>	2022-03-16 12:08:59 +08:00