prehistoric-systems/omnivore
1
0
Fork 0
mirror of https://github.com/omnivore-app/omnivore.git synced 2026-03-11 08:54:26 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
12057 commits 201 branches 31 tags 432 MiB
Commit graph

2536 commits

Author SHA1 Message Date
Jackson Harper
0bcb8a23c5 Use new API methods for saving 2022-07-16 12:17:58 -07:00
Jackson Harper
cb506fcf93 Remove extra quote from report string 2022-07-14 20:33:06 -07:00
Hongbo Wu
edd44fcafc fix get article test 2022-07-14 17:56:55 +08:00
Hongbo Wu
404798a01e use correct index name 2022-07-14 17:08:40 +08:00
Hongbo Wu
4eb55846fd create elastic index if not exists in testing 2022-07-14 16:53:23 +08:00
Hongbo Wu
5a8e2c40c7 fix delete users db permission error 2022-07-13 16:08:46 +08:00
Hongbo Wu
098771f566
Merge pull request #955 from omnivore-app/fix/duplicate-save 
check if page saved by adding links before creating by extension
 2022-07-13 11:57:20 +08:00
Hongbo Wu
01e95c3598 check if page saved by adding links before creating by extension 2022-07-13 11:34:20 +08:00
Hongbo Wu
cb1d64cd82 add siteName in api response 2022-07-13 10:57:54 +08:00
Hongbo Wu
970609c3b8 change size to first in query param 2022-07-13 10:46:03 +08:00
Hongbo Wu
931c519d44 typeahead searchs by siteName 2022-07-13 10:37:06 +08:00
Hongbo Wu
b68282fe97 rename typeahead to typeaheadSearch 2022-07-12 09:36:00 -07:00
Hongbo Wu
3a1f38a6f7 add typeahead api implementation 2022-07-12 09:36:00 -07:00
Hongbo Wu
dc9523e522 add searchAsYouType in elastic 2022-07-12 09:36:00 -07:00
Hongbo Wu
31ac511121 add typeahead gql schema 2022-07-12 09:36:00 -07:00
Jackson Harper
f7a023939d
Merge pull request #940 from omnivore-app/feature/allow-fetching-original 2022-07-11 22:08:27 -07:00
Hongbo Wu
c4a599d2ba support newsletters hosted on convertkit.com 2022-07-12 11:47:38 +08:00
Jackson Harper
27e39b396d Improve formatting for content display report emails 
With the previous formatting the `:` could be appended
to the end of URLs.
 2022-07-11 11:06:17 -07:00
Jackson Harper
e153966eee Linting 2022-07-09 15:38:09 -07:00
Jackson Harper
9feeb80cf3 Allow fetching original content 
This is useful when debugging
 2022-07-09 14:35:14 -07:00
Hongbo Wu
4fe81eaf92
Merge pull request #849 from omnivore-app/feature/delete-account 
Delete account API
 2022-07-09 08:54:57 +08:00
Jackson Harper
e63324c802
Merge pull request #934 from omnivore-app/fix/unsubscribe-email-not-saved 
fix/unsubscribe email not saved
 2022-07-08 13:47:16 -07:00
Jackson Harper
46d96b8f91
Merge pull request #932 from omnivore-app/newsletter-revue 
Add support for newsletters hosted on getrevue.co
 2022-07-08 13:36:10 -07:00
Hongbo Wu
de803ebbd5 fix auth error when deleting user in db 2022-07-08 11:59:55 +08:00
Hongbo Wu
ef0c2434fb Save unsubscribe email address or http url of newsletter emails 2022-07-07 17:14:12 +08:00
Hongbo Wu
322e135ead delete this user pages in elastic 2022-07-07 12:32:55 +08:00
Hongbo Wu
41f43b54ab Publish delete event 2022-07-07 12:31:46 +08:00
Hongbo Wu
efd47f3f83 add function to delete pages by param 2022-07-07 11:07:00 +08:00
Hongbo Wu
b898f55bb9 add delete account result in resolvers 2022-07-07 10:56:20 +08:00
Satindar Dhillon
e5d6386b96 return UserNotFound error if user does not exist in delete account trx 2022-07-06 09:49:01 -07:00
Satindar Dhillon
ddba50dee6 fix delete user resolver and map it to deleteAccount mutation 2022-07-06 09:38:24 -07:00
Satindar Dhillon
da76ae7aab attempt to write resolver functions 2022-07-06 09:38:24 -07:00
Satindar Dhillon
02411347a7 regenerate graphql files 2022-07-06 09:38:24 -07:00
Satindar Dhillon
d595bfb7f1 add delete account mutation to schema.ts 2022-07-06 09:38:24 -07:00
Hongbo Wu
9599edb9fb Add support for newsletters hosted on getrevue.co 2022-07-06 22:06:51 +08:00
Hongbo Wu
2e77241044 Check and get Revue newsletter url from email 2022-07-06 22:06:21 +08:00
Hongbo Wu
e5518e9f23 Reply to the original sender of the forwarded email 2022-07-06 21:16:55 +08:00
Hongbo Wu
c08afe6451 Update page updatedAt when updating labels 2022-06-28 12:03:37 +08:00
Hongbo Wu
00f38d1134 Update page updatedAt when updating highlights 2022-06-28 11:55:26 +08:00
Hongbo Wu
59efc5fea7 Add test 2022-06-28 10:14:15 +08:00
Hongbo Wu
bf1faaeea4 Include highlights in search page result in elastic 2022-06-28 09:56:39 +08:00
Hongbo Wu
6cae3dc9b5 Add highlights in search item 2022-06-28 09:53:48 +08:00
Hongbo Wu
c1855afc72 Update property unarchived=null instead of undefined in the page document in elastic 2022-06-26 18:36:54 +08:00
Jackson Harper
4c89323f0b Make the content fetch GCF URL nullable 2022-06-24 11:30:52 -07:00
Jackson Harper
92c8047a66 Update environment variable names in docker 2022-06-24 10:52:13 -07:00
Hongbo Wu
75d3372426 Use gcf for low priority task 2022-06-24 10:13:14 -07:00
Hongbo Wu
e47415cf69 Add new environment variable for content-fetch url in gcf 2022-06-24 10:13:14 -07:00
Hongbo Wu
5f0cb493f2 Allow filter search result by updatedAt 2022-06-24 13:09:13 +08:00
Hongbo Wu
e96d5552fa Fix search by is:read or is:unread getting empty result by using the correct property name 2022-06-23 12:35:48 +08:00
Hongbo Wu
9a2f4b300a Fix a script error by deleting both labels in pages and highlights 2022-06-20 19:06:50 +08:00
Hongbo Wu
f88083e535
Merge pull request #813 from omnivore-app/advanced-search 
advanced search
 2022-06-19 10:55:18 +08:00
Hongbo Wu
3f4b0f9b8c Update resolver 2022-06-17 17:09:08 +08:00
Hongbo Wu
26e7ab178e Fetch page savedAt, readingProgress from elastic 2022-06-17 17:08:26 +08:00
Hongbo Wu
2bcf97d2e8 Make updatedAt optional 2022-06-17 17:07:13 +08:00
Hongbo Wu
1be9c980e1 Add savedAt to searchItem and make readingProgress required in elastic 2022-06-17 16:03:49 +08:00
Hongbo Wu
e408143ea9 Add savedAt to searchItem and make readingProgress required 2022-06-17 16:02:46 +08:00
Hongbo Wu
e3c30dd076 Allow matching title, description, content keyword in elastic 2022-06-16 22:15:05 +08:00
Hongbo Wu
40bb49012a Allow match keyword 2022-06-16 22:15:01 +08:00
Hongbo Wu
9ec11cfc22 Add title, description, content as keyword in search.ts 2022-06-16 22:14:08 +08:00
Hongbo Wu
215078bcb3 Add logseq to the allowed cors origin 2022-06-16 10:46:06 +08:00
Hongbo Wu
c5c8763767 Add readAt to searchItem 2022-06-15 09:32:53 +08:00
Hongbo Wu
6189e85b2c Add readAt to article 2022-06-15 09:32:10 +08:00
Jackson Harper
ea4e4d6828 Use the Article type instead of Page in updatePage API 
Despite the name we are still using Article in many
places.
 2022-06-09 15:29:01 -07:00
Jackson Harper
3136705a75 Handle errors when creating a page save request on savePage 2022-06-09 13:52:49 -07:00
Jackson Harper
9baeb89742 Add more elastic debugging 2022-06-09 13:52:49 -07:00
Jackson Harper
85cbd4e9d0 Handle errors updating elastic when saving pages 2022-06-09 13:52:49 -07:00
Jackson Harper
05ea57f76f Linting 2022-06-09 12:08:42 -07:00
Jackson Harper
e23eada168 Use the public GCS URL for local uploaded files 2022-06-09 11:59:01 -07:00
Jackson Harper
ef072cb2c9 When creating pages from uploaded files use their signed URL as the page URL 2022-06-09 11:51:01 -07:00
Jackson Harper
cffb5088b4 If creating a request from a file URL, dont try fuzzy matching on URL 2022-06-09 11:37:14 -07:00
Hongbo Wu
ff613287b4
Merge pull request #775 from omnivore-app/fix/delete-label 
Fix failure of lable deletion in highlight
 2022-06-09 14:50:03 +08:00
Hongbo Wu
520ce0077b Add refresh index method 2022-06-09 12:34:22 +08:00
Hongbo Wu
4f4cc51a37 Delete label by name 2022-06-09 12:33:58 +08:00
Hongbo Wu
4cf45af669 Add debug log for deleting label 2022-06-09 12:33:42 +08:00
Hongbo Wu
85317df671
Merge pull request #774 from omnivore-app/fix/missing-title 
Fallback to domain to url if no title
 2022-06-09 11:48:49 +08:00
Hongbo Wu
d1d18315c8 Fallback to domain to url if no title 2022-06-09 11:30:13 +08:00
Jackson Harper
fa2b58eac0 When creating a page during an uploadfile request return its ID 2022-06-08 20:17:47 -07:00
Hongbo Wu
309c581cf5
Merge pull request #729 from omnivore-app/labels-in-highlights 
Support labels in highlights
 2022-06-09 09:28:43 +08:00
Hongbo Wu
2dcab66af2 Update labels for both pages and highlights 2022-06-08 17:39:38 +08:00
Hongbo Wu
7c840f5b1e Merge deleteLabel in page and highlight 2022-06-08 17:25:33 +08:00
Hongbo Wu
a981e13c86 Add test case for deleteLabel 2022-06-08 17:01:38 +08:00
Hongbo Wu
0d67d0fb90 Delete labels for highlights when label is deleted 2022-06-08 15:35:52 +08:00
Hongbo Wu
22c604c15b Add deleteLabelForHighlights 2022-06-08 15:33:43 +08:00
Hongbo Wu
145867fe05 Add test for setLabelsForHighlight resolver 2022-06-08 15:25:11 +08:00
Hongbo Wu
4894b3cfd9 Add setLabelsForHighlight in function_resolvers.ts 2022-06-08 15:00:05 +08:00
Hongbo Wu
e9f4149c62 Add setLabelsForHighlight resolver 2022-06-08 14:59:46 +08:00
Hongbo Wu
eef3ee4114 Add getLabelsByIds service method 2022-06-08 14:59:27 +08:00
Hongbo Wu
93c0e53d1d Generate setLabelsForHighlight graphql code 2022-06-08 14:59:11 +08:00
Hongbo Wu
427b58eedd Add set labels for highlight gql schema 2022-06-06 19:15:51 +08:00
Hongbo Wu
bdda54d54a Add set Labels For Highlight data access method 2022-06-06 19:15:03 +08:00
Hongbo Wu
aa15ef0282 Sort by expires at desc nulls last 2022-06-06 13:34:23 +08:00
Hongbo Wu
d00becc269 Add readAt to search API response 2022-06-04 22:12:59 +08:00
Hongbo Wu
fc68d51320 Support sort/filter by readAt 2022-06-04 22:12:03 +08:00
Hongbo Wu
b1d968ff23 Update readAt when saving reading progress 2022-06-04 22:11:44 +08:00
Hongbo Wu
577c6e84fb Add readAt to page type 2022-06-04 22:11:14 +08:00
Jackson Harper
b61a8f6015
Merge pull request #747 from omnivore-app/fix/ios-sorting 
Disable sorting options that need API improvements
 2022-06-03 14:04:19 -07:00
Jackson Harper
791e31d60f
Merge pull request #750 from omnivore-app/OMN-633 
Reopening OMN-633 to rebase
 2022-06-03 13:36:36 -07:00
Jackson Harper
b56c6dafa9 Disable sorting options that need API improvements 2022-06-03 12:47:43 -07:00
Jackson Harper
bf929db559 Add the updatedAt attribute to page items 2022-06-03 10:24:11 -07:00
Jackson Harper
975411d429
Merge pull request #719 from omnivore-app/fix/pdf-uploading-improvements 2022-06-02 16:47:03 -07:00
Hongbo Wu
89164a353f
Merge pull request #723 from omnivore-app/fix/tweet-not-saved 
Fix tweet not saved
 2022-06-02 11:20:40 +08:00
Hongbo Wu
4f031a579c
Merge pull request #721 from omnivore-app/search-highlights-second-index 
When searching for highlights we should include document title, descr…
 2022-06-02 11:05:30 +08:00
Hongbo Wu
04f7fdc31f Add labels in highlight type 2022-06-01 19:01:26 +08:00
Hongbo Wu
195074fa49 Update search query 2022-06-01 13:49:51 +08:00
Hongbo Wu
304fe70113 Fix tests 2022-06-01 10:30:23 +08:00
Jackson Harper
11f1b78c45 Remove changes to upload.ts 2022-05-31 15:26:33 -07:00
Jackson Harper
0e10d5630d Rebase and merge 2022-05-31 15:11:10 -07:00
Hongbo Wu
404805e0c0 Make async calls to parse() 2022-05-31 22:51:00 +08:00
Hongbo Wu
74054cbdbd
Merge pull request #716 from omnivore-app/api-token 
Api tokens
 2022-05-31 14:22:23 +08:00
Jackson Harper
6f35740272 Linting fixes 2022-05-30 22:38:24 -07:00
Jackson Harper
53c423b714 WIP: upload PDFs using background task 2022-05-30 22:30:43 -07:00
gitstart-omnivore
d42b0c244c fixed merge conflicts 2022-05-30 12:29:49 +00:00
Hongbo Wu
a04bfbb83e When searching for highlights we should include document title, description, content as secondary indexes 2022-05-29 22:43:21 +08:00
Hongbo Wu
c0bb67a6b5 Verify api key for routers too 2022-05-28 22:00:01 +08:00
Jackson Harper
987336f438
Merge pull request #712 from omnivore-app/fix/ios-pdf-handling 
Better PDF upload handling on iOS
 2022-05-27 08:43:07 -07:00
Hongbo Wu
6e16631f3b Resolve conflicts 2022-05-27 18:37:10 +08:00
Hongbo Wu
fa9c5b3efe Add test for list api keys 2022-05-27 18:36:18 +08:00
Hongbo Wu
b7206457b4 Add test for revoke api key 2022-05-27 18:36:17 +08:00
Hongbo Wu
0f2a55d274 Add list/delete api key resolver 2022-05-27 18:35:38 +08:00
Hongbo Wu
fd4fc8a4e7 Add list/delete api key in schema 2022-05-27 18:34:36 +08:00
Hongbo Wu
333b0259ba Fix tests 2022-05-27 18:33:49 +08:00
Hongbo Wu
9051a3f43a Update api key validation 2022-05-27 18:33:49 +08:00
Hongbo Wu
a09588d3a7 Update generateApiKey resolver 2022-05-27 18:33:49 +08:00
Hongbo Wu
849adf84d4 Add entity class 2022-05-27 18:33:49 +08:00
Jackson Harper
0cabd26344 Dont send the normalized URL to the content-fetch service 
URLs need to be sent to contentFetch exactly as they
were received, this is because many apps will use signed URLs
for PDFs being served from GCP or AWS. A signed URL needs to
keep all its query params in the same order to validate.
 2022-05-26 21:40:40 -07:00
Jackson Harper
93b095818d Linting fixes 2022-05-26 21:40:40 -07:00
Jackson Harper
cb98a9cf86 Make clients opt into creating a page when uploading a file 2022-05-26 21:40:40 -07:00
Hongbo Wu
26681e5ecd Add get/list/delete webhook resolver tests 2022-05-26 14:52:01 +08:00
Hongbo Wu
a75ce3c44d Add get/list/delete webhook resolver 2022-05-26 14:52:00 +08:00
Hongbo Wu
cc07c00a83 Add get/list/delete webhook gql schema 2022-05-26 14:51:11 +08:00
Hongbo Wu
c9c6f83e65 Add tests for trigger endpoint 2022-05-26 14:51:11 +08:00
Hongbo Wu
7b0ebb9dbf Add pubsub subscription to trigger webhook 2022-05-26 14:51:11 +08:00
Hongbo Wu
7157f38738 Add test 2022-05-26 14:51:11 +08:00
Hongbo Wu
6d0ce30a44 Add setWebhook resolver 2022-05-26 14:51:10 +08:00
Hongbo Wu
5528d1d349 Replace createWebhook with setWebhook 2022-05-26 14:50:38 +08:00
Hongbo Wu
f61f61312c Add webhook entity 2022-05-26 14:50:10 +08:00
Hongbo Wu
a655275754 Add unique constraint on user_id and event_types 2022-05-26 14:50:10 +08:00
Hongbo Wu
aaab02c9d8 Add createWebhook gql schema 2022-05-26 14:50:08 +08:00
Hongbo Wu
7d4d1d7b67 Parse language in readability 2022-05-26 10:55:21 +08:00
Hongbo Wu
0ef9bb3944 Allow search by language in elastic 2022-05-26 10:55:20 +08:00
Hongbo Wu
87b0f683e1 Add language term in search body 2022-05-26 10:55:04 +08:00
Hongbo Wu
6e06429636 Add language keyword in search query 2022-05-26 10:55:02 +08:00
Hongbo Wu
1eeb5617a5 Search by language 2022-05-26 10:54:04 +08:00
Hongbo Wu
dd394144cd Add language in schema 2022-05-26 10:52:19 +08:00
Hongbo Wu
a702993485 Add search by language query 2022-05-26 10:52:18 +08:00
Hongbo Wu
3e0e8b486f Add language in searchitem 2022-05-26 10:52:18 +08:00
Hongbo Wu
be91562ca1 Add language in page 2022-05-26 10:52:18 +08:00
Jackson Harper
d704fad2ff
Merge pull request #701 from omnivore-app/feature/add-popular-reads 
API to add recommended (popular) reads to a users library
 2022-05-24 21:09:24 -07:00
Hongbo Wu
f4c65d7fe1
Merge pull request #698 from omnivore-app/sort-by-publish-time 
Support sort or filter by publish time
 2022-05-25 10:41:49 +08:00
Jackson Harper
a20ba60053 Add popular reads for onboarding 2022-05-24 15:31:55 -07:00
Jackson Harper
ea50f51644 Update to use image proxy URLs in parsed content 2022-05-24 13:43:57 -07:00
Jackson Harper
ed5dfde1a8 Linting fixes, remove debug 2022-05-24 13:06:40 -07:00
Jackson Harper
2b14d51ec6 API to add recommended (popular) reads to a users library 
This API is used in onboarding
 2022-05-24 12:05:39 -07:00
Hongbo Wu
a35b096d73 Support sort and filter by published in resolver 2022-05-24 18:25:32 +08:00
Hongbo Wu
61065efd2b Add published sort param in elastic query 2022-05-24 18:24:30 +08:00
Hongbo Wu
641f191fcd Add published sort param in schema 2022-05-24 18:23:30 +08:00
gitstart-omnivore
75728cc144 fixing diff 2022-05-24 07:04:54 +00:00
Jackson Harper
2d2495a10e
Merge pull request #685 from omnivore-app/fix/pdf-api-simplification 
Simplify the API to create pages from PDFs
 2022-05-21 21:19:30 -07:00
Jackson Harper
e5bc68ca9e Simplify the API to create pages from PDFs 
This creates a stub page when an upload is initiated, so the
SaveFile (or create article from file) APIs do not need to be
called.

The PDF uploaded trigger fired by GCS will mark the upload
as completed and update the page status.
 2022-05-21 12:05:45 -07:00
Jackson Harper
17e6f4091d
Merge pull request #638 from omnivore-app/feature/subscription-api 
Feature/subscription api
 2022-05-20 15:34:48 -07:00
Hongbo Wu
2ee95a1c14 Fix cannot convert null to object error message 2022-05-20 20:38:30 +08:00
Hongbo Wu
444c21ad09 Fix labels test 2022-05-19 18:12:14 +08:00
Hongbo Wu
34349e925c Add unit test 2022-05-19 16:36:35 +08:00
Hongbo Wu
b8225fc8c8 Fix labels not being updated in elastic 2022-05-19 12:08:50 +08:00
Jackson Harper
eadeccce81 Linting fixes 2022-05-18 11:31:42 -07:00
Jackson Harper
004c766588 If parsing fails, attempt adding <html> wrappers to a document 
LinkedDom seems less forgiving and expects the outerHTML of a
document, however older extension versions still send innerHTML.
 2022-05-18 10:55:31 -07:00
Hongbo Wu
d68549bcb7 Remove unused code 2022-05-18 15:52:31 +08:00
Hongbo Wu
d9d4deb14c Update Money Stuff sign up url 2022-05-18 11:50:43 +08:00
Hongbo Wu
5b36f9270e Update milk road query params 2022-05-17 22:43:00 +08:00
Hongbo Wu
13453e7b7f Register subscribe resolver 2022-05-17 21:26:46 +08:00
Hongbo Wu
568472cc3c Calling HTTP requests to subscribe to these newsletters 2022-05-17 21:15:25 +08:00
Hongbo Wu
4699458457 Implement subscribe api 2022-05-17 18:46:57 +08:00
Hongbo Wu
1da8b75c81 Add a subscribeHandler 2022-05-17 18:46:44 +08:00
Hongbo Wu
51dd9a5733 Return a list of subscriptions 2022-05-17 18:45:52 +08:00
Hongbo Wu
c30ef093b8 Update unsubscribe api 2022-05-17 16:47:34 +08:00
Hongbo Wu
906b2d28a0 Add subscribe API schema 2022-05-17 16:23:36 +08:00
Hongbo Wu
e76fb02f43 Fix window is not defined for parsing code blocks 2022-05-17 11:01:55 +08:00
Jackson Harper
7bfb8cfee4
Merge pull request #597 from omnivore-app/remove-chrome-aws-lambda 
Optimize puppeteer and remove chrome-aws-lambda dependencies
 2022-05-13 16:12:24 -07:00
Jackson Harper
8b37a475c8
Merge pull request #612 from omnivore-app/fix/reminder-id-type 
Use pageId instead of articleSavingRequestId in reminders
 2022-05-13 10:47:46 -07:00
Jackson Harper
d3b65c0ff8
Merge pull request #611 from omnivore-app/fix/elastic-exception 
Stop throwing document not found exception to sentry
 2022-05-13 09:05:01 -07:00
Hongbo Wu
b306ca6e2e Fix UpdateLabelResult undefined 2022-05-13 17:38:51 +08:00
Hongbo Wu
45213e7b54 Fix reminder trigger 2022-05-13 17:07:10 +08:00
Hongbo Wu
a51eba67ba get reminder by page id 2022-05-13 16:17:25 +08:00
Hongbo Wu
d821f26440 create reminder with page id 2022-05-13 16:12:08 +08:00
Hongbo Wu
40cbfc9058 Use reminder by elasticPageId 2022-05-13 16:01:23 +08:00
Hongbo Wu
f51dbafdfc Stop throwing document not found exception to sentry 2022-05-13 11:40:08 +08:00
Jackson Harper
eee116e71d Add an analytics event when links are removed 2022-05-12 10:55:12 -07:00
Hongbo Wu
adfaf6b8ce Use newsletter's title and author as page's ones 2022-05-12 18:08:38 +08:00
Hongbo Wu
1b8850ed33 Fix tests 2022-05-12 17:41:11 +08:00
Hongbo Wu
e1e0ddf7fc
Merge pull request #582 from omnivore-app/optimize-parsing 
Optimize parsing
 2022-05-12 11:07:52 +08:00
Hongbo Wu
602d141dec Rename doc to dom 2022-05-12 11:00:32 +08:00
Hongbo Wu
5dca02873d Revert originalArticleUrl change 2022-05-12 10:45:13 +08:00
Hongbo Wu
8f75c43658 Add siteName in search result 2022-05-12 10:30:47 +08:00
Hongbo Wu
76aef35423 Remove www. from the siteName in the search api response 2022-05-11 23:07:34 +08:00
Hongbo Wu
a78a6c6ba4 Replace DomWindow with Document in handlers 2022-05-10 17:01:23 +08:00
Hongbo Wu
6a57281e74 Remove DomWindow usage 2022-05-10 17:00:56 +08:00
Hongbo Wu
acc7654a2f Replace jsdom with linkedom 2022-05-10 16:59:09 +08:00
Hongbo Wu
5698790288 Pass url to readability 2022-05-10 16:53:45 +08:00
Hongbo Wu
5007b11a3c Add url to readability option 2022-05-10 16:52:10 +08:00
Hongbo Wu
59a2639b7d Reduce http call to get jsonld data if title or content or sitename or byline exists 2022-05-09 13:45:45 +08:00
Hongbo Wu
a457c9d128 Update article content only when code blocks exist 2022-05-09 13:45:45 +08:00
Hongbo Wu
eaad96acdd Return parsed dom back to backend 2022-05-06 12:29:08 +08:00
Hongbo Wu
5f5076e864 Highlight code element without reinitialize jsdom 2022-05-06 12:20:54 +08:00
Hongbo Wu
7c6b810522 Remove redundant JSDOM 2022-05-06 10:53:36 +08:00
Hongbo Wu
93ab6bc87a Add includePending=true in search API 2022-05-04 21:28:06 +08:00
Hongbo Wu
bc1ed3f054 Check state in articleSavingRequest 2022-05-04 19:33:13 +08:00
Hongbo Wu
9e3db0e053 Fix a bug to have multiple pages with the same url in lib 2022-05-04 19:12:22 +08:00
Hongbo Wu
afe8b6e948 make savedAt a required field in page 2022-05-04 17:54:59 +08:00
Hongbo Wu
a42bb7ab38 Replace createdAt with savedAt 2022-05-04 12:59:07 +08:00
Hongbo Wu
98ea1016d5 Remove <p> tag in description 2022-05-04 12:50:22 +08:00
Hongbo Wu
9048ccbab3 Show failed pages in library 2022-05-04 12:46:57 +08:00
Hongbo Wu
ca0f58ee22 Update page content and state if failed to parse 2022-05-04 12:21:41 +08:00
Hongbo Wu
80d0146f72 Show unable to parse if page is failed to save within 30 seconds 2022-05-04 11:55:16 +08:00
Jackson Harper
c8c4377346
Merge pull request #531 from omnivore-app/fix/saving-analytics 
Add more detailed analytics for saving. Remove intercom events
 2022-05-02 21:14:18 -07:00
Jackson Harper
95b163a437 Prettier clean up 2022-05-02 20:58:05 -07:00
Jackson Harper
ab96bb09ab Add more detailed analytics for saving. Remove intercom events 2022-05-02 20:41:10 -07:00
Jackson Harper
b649d583b7 Linting changes 2022-05-02 15:58:40 -07:00
Jackson Harper
453b301e74 Allow looking up articles by ID 
iOS needs to be able to lookup an article by its ID not its
slug, because the slug can change once saved.
 2022-05-02 15:48:21 -07:00
Jackson Harper
61b7ac3cb2 Enable graphql introspection when running locally 2022-05-02 09:44:13 -07:00
Hongbo Wu
77ca0ceb59 Show empty description if not exist 2022-04-29 15:50:39 +08:00
Hongbo Wu
2b70d480d2
Remove article saving request (#493) 
* Add state and taskName in elastic page mappings

* Add state and taskName in elastic page interface

* Create page with PROCESSING state before scrapping

* Update createArticleRequest API

* Fix tests

* Add default state for pages

* Update createArticle API

* Update save page

* Update save file

* Update saving item description

* Show unable to parse content for failed page

* Fix date parsing

* Search for not failed pages

* Fix tests

* Add test for saveUrl

* Update get article saving request api

* Update get article test

* Add test for articleSavingRequest API

* Add test for failure

* Return new page id if clientRequestId empty

* Update clientRequestId in savePage

* Update clientRequestId in saveFile

* Replace article with slug in articleSavingRequest

* Add slug in articleSavingRequest response

* Depreciate article

* Use slug in web

* Remove article and highlight fragments

* Query article.slug on Prod

* Show unable to parse description for failed page

* Fix a bug having duplicate pages when saving the same url multiple times

* Add state in response

* Rename variables in removeArticle API

* Rename state

* Add state in response in web

* Make state an enum

* Open temporary page by link id

* Use an empty reader view as the background for loading pages

* Progressively load the article page as content is loaded

* Add includePending flag in getArticles API

* Set includePending = true in web

* Add elastic update mappings in migration script

* Add elastic mappings in docker image

* Move index_settings.json to migrate package

* Remove elastic index creation in api

* Move elastic migrations to a separate directory

* Remove index_settings from api docker image

Co-authored-by: Jackson Harper <jacksonh@gmail.com>
 2022-04-29 13:41:06 +08:00
Hongbo Wu
ccb79845bc
Fix subscriptions issues (#501) 
* Replace fetch with axios

* Add url in subscription

* Remove url in subscription

* Unsubscribe all newsletters before deleting that email

* Continue unsubscribing others if one got error
 2022-04-29 11:16:46 +08:00
Hongbo Wu
d1c2721b95
Add expiration date to api key generation (#495) 
* Add exp to jwt token

* Add expiredAt in generate api key input

* Check exp in claim

* Add tests
 2022-04-28 09:09:40 +08:00
Hongbo Wu
bcbda50daa Add basic UI for subscriptions 2022-04-22 12:17:09 -07:00
Hongbo Wu
3e35fb1aca Email content display report 2022-04-22 19:14:59 +08:00
Hongbo Wu
71aa66ea9c
Allow search for subscription:name (#461) 2022-04-22 08:47:41 +08:00
Jackson Harper
0a4d55ef15 Increase the rate limit 
Turns out 10 is very low because we make CORs requests from the
browser and SWR will make background requests for some objects
like viewer. I was hitting the limit while testing search.
 2022-04-21 14:59:57 -07:00
Hongbo Wu
dca6461b62
Add unsubscribe resolver (#453) 
* add unsubscribe resolver

* add subscription to articles api response

* revert yarn.lock changes
 2022-04-21 13:18:15 +08:00
Hongbo Wu
83cdec42f7
Add rate limiter for api request (#454) 
* Add rate limiter for api request

* apply rate limit on Demo/Prod
 2022-04-21 09:41:33 +08:00
Hongbo Wu
5c7f4a22bf
Save subscriptions when saving newsletters (#447) 
* parse unsub mailTo or httpUrl and send to pubsub

* save subscription from newsletter

* fix tests
 2022-04-20 09:24:40 +08:00
Jackson Harper
48e36c8635
Merge pull request #393 from omnivore-app/OMN-382 
[Omn-382] - Updated landing page
 2022-04-19 11:15:31 -07:00
Hongbo Wu
d7a2659fc4
fix searching for multiple labels (#444) 2022-04-19 12:13:47 +08:00
Hongbo Wu
1117a0c575
Feature/subscription list resolver (#432) 
* add subscriptions table

* add listSubscriptions schema

* add listSubscriptions resolver
 2022-04-19 11:08:43 +08:00
Jackson Harper
c2fd58a809 Make landing a restricted username page 2022-04-18 15:22:59 -07:00
Hongbo Wu
0b2b93384f fix return undefined as updateLabelResult 2022-04-18 20:45:06 +08:00
Jackson Harper
d33d383891
Merge pull request #422 from omnivore-app/fix/reminder-data-model-error 
Fix reminders resolver getting pages from postgres
 2022-04-14 14:10:48 -07:00
Jackson Harper
c80ce4da10 Consistent naming convention for events 2022-04-14 10:31:10 -07:00
Jackson Harper
2c9c2a2adf
Merge pull request #394 from omnivore-app/header-and-reader-improvements 
Add margins on wider screens, keyboard commands to adjust
 2022-04-13 21:26:24 -07:00
Hongbo Wu
c0966d908f
upload highlight and label to gcs if successfully updated in elastic (#420) 2022-04-14 12:14:15 +08:00
Hongbo Wu
88d8ca7b7d fix reminders resolver getting page from postgres 2022-04-14 12:09:17 +08:00
Jackson Harper
e282fc8316
Merge pull request #417 from omnivore-app/feature/upload-highlight-gcs 
Upload highlights and labels data to GCS
 2022-04-13 13:41:29 -07:00
Hongbo Wu
0b0b521fab use generic method for uploading 2022-04-13 22:21:20 +08:00
Hongbo Wu
a806f55233 upload highlight to gcs 2022-04-13 21:18:58 +08:00
Hongbo Wu
8ba1be456c fix sanitizer invalid name 2022-04-13 13:23:21 +08:00
Hongbo Wu
3b831fecb8 test both max length and pattern if declared 2022-04-13 12:20:41 +08:00
Hongbo Wu
928ddd69a9 validate color input in label as rgb hex value 2022-04-13 12:17:12 +08:00
Jackson Harper
73ef8fef09 Rename linkId to pageId in SetLabelsInput 2022-04-12 12:46:32 -07:00
Jackson Harper
d05f091876 Add debug logging for GQL errors 2022-04-12 12:23:55 -07:00
Jackson Harper
6c5e88cf12
Merge pull request #405 from omnivore-app/feature/label-max-length 
Set label name max length = 64 char
 2022-04-12 11:14:20 -07:00
Hongbo Wu
ae0d1dd2ee
Feature/search highlights backend (#395) 
* add highlight mappings

* return highlight in resolvers

* temporarily skip highlight tests

* add test for getting highlights

* update merge highlight

* separate elastic methods

* roll back merge highlight test

* add highlight to elastic script

* update delete highlight in elastic

* migrate highlight data from postgres to elastic

* rescue not found exception when page is not found in the migration script

* exclude highlights in searching pages results

* search pages with highlights only with has:highlight query

* add search endpoint to search pages or highlights

* reduce code smell in search api

* fix rebase error

* fix tests

* add test for search highlight

* add test for new search endpoint

* add labels to search results

* update schema

* update search query

* fix update/share highlights

* fix rebase error

* fix tests

* add highlight model in elastic

* add savedAt and publishedAt date range in search query

* add sort by updated and recently read

* fix tests

* close db connection when tests are done

* test github action

* revert github action test

* fix rebase error

* add docker-compose for api-test

* remove unused env

* remove highlights with no page attached to

* allow get_articles resolver to search for query so we can merge it without web changes
 2022-04-12 12:31:08 +08:00
Hongbo Wu
2ebdaba780
add generate api key api and test (#392) 
* add generate api key api and test

* test if user can make api call with the api key
 2022-04-12 12:11:45 +08:00
Hongbo Wu
bb665a704b make label name <= 64 char length 2022-04-12 12:08:02 +08:00
Hongbo Wu
b4420839bd
fix: fail to search by excluding labels with capital letter (#400) 
* fix: fail to search by excluding labels with capital letter

* make all the excluded labels lowcased

* Revert "make all the excluded labels lowcased"

This reverts commit 866bed4080.
 2022-04-11 14:05:21 +08:00
Jackson Harper
48ca13818f Prettier fixes 2022-04-06 10:45:01 -07:00
Jackson Harper
57f865c9a2 Update to typeorm 0.3 2022-04-06 10:37:13 -07:00
Jackson Harper
1cdb7753c3 Update typeorm usage 2022-04-06 09:56:32 -07:00
Jackson Harper
06360a5993
Merge branch 'main' into OMN-190 2022-04-06 09:24:49 -07:00
Hongbo Wu
d1269f815e identify a newsletter by checking both list-url and unsubscribe-url in headers and email sender address 2022-04-06 20:59:14 +08:00
Hongbo Wu
7569e988bf
upgrade typeorm to 3.0 (#359) 
* upgrade typeorm to 3.0

* use new datasource object in typeorm 3

* fix tests

* fix tests

* migrate before creating connection

* fail the test if migration failed
 2022-04-06 10:32:41 +08:00
Jackson Harper
9845a49040 Remove semicolon 2022-04-04 08:17:46 -07:00
Jackson Harper
c8ec7d636e Use require to import address parser function 2022-04-04 08:11:26 -07:00
Jackson Harper
92925b6808 Prettier fixes 2022-04-03 10:21:16 -07:00
Jackson Harper
da28998130 Pull ul list out of newsletter blurb 
next/react doesnt want child elements of the paragraphs

Improve formatting

Improve wording

Use buttons in the subscribe directly blocks

Simplify docs on setting up forwarding rules

Add extra padding on bottom of help docs

Remove unused style

Add emails help page

Improve formatting

Prefetch page content on iOS

Reduce the reader overly length now that items are precached

Add invalidation when highlights are added to items

fix missing index_settings.json file in api dockerfile for creating elastic index (#363)

Handle full email address objects in the to param from sendgrid

These come in a format like:

"jacksonh-dfdf@inbox.omnivore.app" <jacksonh-dfdf@inbox-demo.omnivore.app>

New IDs for short highlights because they dont cascade delete now

Testing CI issues

Simplify test

CI test

Use promises for async tests

Temporarily remove test to debug CI

Re-enable

re-enable test, return error

Specify a userId when looking up saved email pages

create a unique url for newsletters without a URL

Use 500ms on page test timeouts

Increase timeout

Dont use deep equal to match newsletter label

Run just the labels API

Run against just the newsletter emails

Run without the page tests

Fix

Set the allow uncaught flag

Remove highlight tests

Remove newsletters tests

more resolver tests

Remove newsetter tests

Comment out resolver tests

Use nock for external requests in tests

Specify puppeteer url for tests

Comment out more tests

uncomment tests

re-enable

re-enable email test

Re-disable

Re-enable one pdf attachment test

Re-disable pdf attachment test

Use promises on setTimeout tests

rm label tests

mv label tests into a context

Comment out pdf tests

Comment out pdf tests

Async test

Async wrappers

Delay when creating test pages

More debugging

Unique short ids

Remove potentially problematic test

Fetch page before returning for test

handler in before block

more debugging

More debugging

Move errors checks into contexts

Use a context when saving newsletters to force index refresh

Prettier fix

Fix newsletter label check, remove setTimeout

Re-enable test

timeout on pdf router handler

Fix method call

comment out PDF test

Unique fake username

Comment out PDF test

Debugging signed urls

Re-enable

New email

pdf test

PDF tests

Comment out pdf test

Add nock stubs for email URLs

Use full address for PDF test

Remove debug

Use full email addresses
 2022-04-02 16:56:24 -07:00
gitstart-omnivore
1978a92a1b Effected review corrections 2022-03-31 13:03:50 +00:00
Jackson Harper
a8faaf835c Remove debug sort on labels 2022-03-30 14:13:26 -07:00
Jackson Harper
2c15516d96 Order labels by createdAt so they maintain a consistent order 2022-03-30 13:28:37 -07:00
gitstart-omnivore
11db7be473 [Omn-190] - [Settings View] - Labels 2022-03-30 12:24:29 +00:00
Hongbo Wu
6d405432af
add site_name and site_icon to page model and return in resolver (#341) 
* add site_name and site_icon to page model and return in resolver

* fix tests
 2022-03-30 10:43:10 +08:00
Jackson Harper
2bada0b9df
Merge pull request #293 from omnivore-app/feature/priority-by-rate-limit 
Use lower priority queue for large imports
 2022-03-28 13:19:35 -07:00
Jackson Harper
d9af397b9c
Merge pull request #331 from omnivore-app/fix/update-page-after-deletion 
fix document_missing_exception being thrown to sentry when updating page in elastic
 2022-03-28 10:30:31 -07:00
Hongbo Wu
fd18affaec fix document_missing_exception being thrown to sentry when updating page in elastic 2022-03-28 21:16:33 +08:00
Hongbo Wu
b6c04f3508 fix saving report item by linking to elastic page 2022-03-28 20:38:54 +08:00
Jackson Harper
1cdbfaca26
Merge pull request #294 from omnivore-app/feature/add-created-or-updated_at-to-newsletter_email 
add created_at and updated_at to newsletter emails and sort by create…
 2022-03-25 09:10:54 -07:00
Hongbo Wu
27735f7310
add page id in updated page json file in gcs (#319) 2022-03-25 13:50:43 +08:00
Hongbo Wu
e7203bebb5
add a function_resolver to return originalArticleUrl from article.url (#317) 
* add a function_resolver to return originalArticleUrl from article.url

* add a test
 2022-03-25 11:25:30 +08:00
Hongbo Wu
6597cc37b6 add created_at and updated_at to newsletter emails and sort by created_at desc 2022-03-22 18:43:47 +08:00
Hongbo Wu
acec3b1acb Use lower priority queue for large imports 
* add a count query in elastic
  * if priority is not set, check the rate limit of user's libary
    - if 5 articles added in the last minute: use low queue
    - default: use normal queue
 2022-03-22 18:08:08 +08:00
Hongbo Wu
1893e36375
Fix uploading endpoint returns 504 (#289) 
* add timeout = 60s for uploading to private bucket

* add debug logs

* send response to client to close connection

* add tests for upload

* add PUBSUB_VERIFICATION_TOKEN in .env.test
 2022-03-22 14:20:40 +08:00
Hongbo Wu
3e9063c145
return empty content instead of null in search result (#291) 2022-03-22 13:34:44 +08:00
Hongbo Wu
bee3b3c6fa
fix article_id type mismatch (uuid => varchar) when saving highlights in postgres by setting article_id to be undefined (#285) 2022-03-22 11:48:01 +08:00
Hongbo Wu
4ca7c89622
format date string as yyyy-mm-dd (#288) 2022-03-22 11:46:39 +08:00
Hongbo Wu
c15efe6aad
Fix sentry log size error (#287) 
* reduce sentry log size error by partially logging page created in elastic

* remove some debugging logs

* remove more debugging logs
 2022-03-22 11:45:58 +08:00
Jackson Harper
f9bce1932c Print the logging level on startup 2022-03-21 09:31:05 -07:00
Hongbo Wu
77050905a5 retry max three times if update labels is conflicted 2022-03-18 10:39:11 +08:00
Hongbo Wu
9d8d2e40d7 reduce time to save reading progress in elastic update 2022-03-18 10:29:01 +08:00
Hongbo Wu
ac567f7e71 retry max three times if update is conflicted 2022-03-18 10:17:20 +08:00
Jackson Harper
7910565dbf
Merge pull request #251 from omnivore-app/fix/jsonld-encoding 
Use html decoding when getting values from fetched oembed
 2022-03-17 16:53:21 -07:00
Hongbo Wu
563131ea23
Fix/newsletter label not add (#256) 
* default sort by savedAt to avoid recent read page to appear on top of the list

* revert lint on generated code

* fix elastic update script syntax error
 2022-03-17 13:22:38 +08:00
Hongbo Wu
a2ce98229e
default sort by savedAt to avoid recent read page to appear on top of… (#254) 
* default sort by savedAt to avoid recent read page to appear on top of the list

* revert lint on generated code
 2022-03-17 10:54:00 +08:00
Hongbo Wu
af037a2837
make readingProgress required in the elastic page data (#253) 
* make readingProgress required in the elastic page data

* delete readingProgress from function_resolvers because we have stored them in elastic
 2022-03-17 10:06:21 +08:00
Jackson Harper
ff1200f3a1 Use html decoding when getting values from fetched oembed 
If we fetch oembed data from an external source, instead of
handling it in readabilityjs we need to html decode it.
 2022-03-16 15:29:42 -07:00
Jackson Harper
89f3719ba3 Fix linting 2022-03-16 14:23:39 -07:00
Jackson Harper
0314d8cf17 Add some extra logging 2022-03-16 13:53:55 -07:00
Jackson Harper
dc27c0c7a4 Return 0 if looking up page reading progress fails 2022-03-16 13:33:36 -07:00
Jackson Harper
27143cbedf Remove function handlers for reading progress 
These shouldn't be needed anymore because the values are stored
in elastic.
 2022-03-16 13:11:36 -07:00
Hongbo Wu
07dcd5da26 add label to page only if not exists 2022-03-16 23:00:37 +08:00
Hongbo Wu
1c4dcd7b00 automatically update updatedAt when page is updated in elastic 2022-03-16 22:28:39 +08:00
Hongbo Wu
f412758040 add Newsletter label to the page created by newsletters email 2022-03-16 18:40:05 +08:00
Hongbo Wu
27157006c1
use private bucket to upload page events (#244) 
* use private bucket to upload page events

* fix tests

* add GCS_UPLOAD_PRIVATE_BUCKET in test env

* allow GCS_UPLOAD_PRIVATE_BUCKET to be empty
 2022-03-16 14:39:07 +08:00
Jackson Harper
e652a6ea8c
Rebased version of the elastic PR (#225) 
* Add elastic to our docker compose

* add AND/OR/NOT search operations

* add elastic and create article in elastic

* change error code when elastic throws error

* add search pages in elastic

* add search by labels

* Add elastic to GitHub Action

* Update elastic version

* Fix port for elastic

* add url in search query

* Set elastic features when running tests

* add debug logs

* Use localhost instead of service hostname

* refresh elastic after create/update

* update search labels query

* add typescript support

* search pages in elastic

* fix search queries

* use elastic for saving page

* fix test failure

* update getArticle api to use elastic

* use generic get page function

* add elastic migration python script

* fix bulk helper param

* save elastic page id in article_saving_request instead of postgres article_id

* fix page archiving and deleting

* add tests for deleteArticle

* remove custom date type in elastic mappings which not exist in older version of elastic

* fix timestamp format issue

* add tests for save reading progress

* add tests for save file

* optimize search results

* add alias to index

* update migration script to receive env var as params

* Add failing test to validate we don't decrease reading progress

This test is failing with Elastic because we aren't fetching
the reading progress from elastic here, and are fetching it
from postgres.

* Rename readingProgress to readingProgressPercent

This is the name stored in elastic, so fixes issues pulling the
value out.

* Linting

* Add failing test for creating highlights w/elastic

This test fails because the highlight can't be looked up. Is there
a different ID we should be passing in to query for highlights,
or do we need to update the query to look for elastic_id?

* add tests code coverage threshold

* update nyc config

* include more files in test coverage

* change alias name

* update updateContent to update pages in elastic

* remove debug log

* fix createhighlight test

* search pages by alias in elastic

* update set labels and delete labels in elastic

* migration script enumeration

* make BULK_SIZE an env var

* fix pdf search indexing

* debug github action exit issue

* call pubsub when create/update/delete page in elastic

* fix json parsing bug and reduce reading data from file

* replace a depreciated pubsub api call

* debug github action exit issue

* debug github action exit issue

* add handler to upload elastic page data to GCS

* fix tests

* Use http_auth instead of basic_auth

* add index creation and existing postgres tables update in migration script

* fix a typo to connect to elastic

* rename readingProgress to readingProgressPercent

* migrate elastic_page_id in highlights and article_saving_request tables

* update migration script to include number of updated rows

* update db migration query

* read index mappings from file

* fix upload pages to gcs

* fix tests failure due to pageContext

* fix upload file id not exist error

* Handle savedAt & isArchived attributes w/out quering elastic

* Fix prettier issues

* fix content-type mismatching

* revert pageId to linkId because frontend was not deployed yet

* fix newsletters and attachment not saved in elastic

* put linkId in article for setting labels

* exclude orginalHtml in the result of searching to improve performace

* exclude content in the result of searching to improve performace

* remove score sorting

* do not refresh immediately to reduce searching and indexing time

* do not replace the backup data in gcs

* fix no article id defined in articleSavingRequest

* add logging of elastic api running time

* reduce home feed pagination size to 15

* reduce home feed pagination size to 10

* stop revalidating first page

* do not use a separate api to fetch reading progress

* Remove unused comment

* get reading progress if not exists

* replace ngram tokenizer with standard tokenizer

* fix tests

* remove .env.local

* add sort keyword in searching to sort by score

Co-authored-by: Hongbo Wu <hongbo@omnivore.app>
 2022-03-16 12:08:59 +08:00
Jackson Harper
a4533dc016
Merge pull request #201 from omnivore-app/feature/beehiiv-newsletter-support 
Support newsletters hosted on beehiiv
 2022-03-15 14:03:55 -07:00
Jackson Harper
8e1b4fb1a4 Formatting 2022-03-14 15:36:17 -07:00
Jackson Harper
a81181ee60 Dont make queries for readingProgressPercent unless we have to 2022-03-14 15:22:53 -07:00
Jackson Harper
78660c886d rm debug 2022-03-13 09:06:15 -07:00
Jackson Harper
a874482d11 Dont perform an extra query for isArchived 2022-03-13 09:00:25 -07:00
Jackson Harper
bea7d084c4 SetClaims when creating an email article 2022-03-09 19:45:52 -08:00
Jackson Harper
f7814a0c4a Remove unused function 2022-03-09 19:45:31 -08:00
Jackson Harper
b0fe9059a9 Dont try to generate highlight URL previews until share is re-enabled 2022-03-09 09:49:03 -08:00
Jackson Harper
384a0771a7
Merge pull request #200 from omnivore-app/fix/validate-urls 
Use the validateUrl method to validate URLs
 2022-03-08 20:28:33 -08:00
Jackson Harper
c45c408c14 Fix formatting 2022-03-08 15:27:05 -08:00
Jackson Harper
e8fca4a7a9 Remove debug line 2022-03-08 15:20:24 -08:00
Jackson Harper
26dadab4aa rm debug, we dont need to set claims on create 2022-03-08 15:10:01 -08:00
Jackson Harper
b982bf34d6 SetClaims on userArticle create/update in saveEmail 2022-03-08 14:43:10 -08:00
Jackson Harper
9ae81d7394 Add extra debugging on newsletter save errors 2022-03-08 14:00:35 -08:00
Jackson Harper
2184c2a8d3 Parse online URLs for beehiiv newsletters 2022-03-07 15:49:44 -08:00
Jackson Harper
2cb5cc065a Add support for identifying newsletters hosted on beehiiv.com 2022-03-07 15:23:56 -08:00
Jackson Harper
be7e36ffed Use the validateUrl method to validate URLs 2022-03-07 14:34:32 -08:00
Jackson Harper
b6fd3e786e Fix parsing authors from page metadata 2022-03-03 19:40:02 -08:00
Jackson Harper
49092b707d Remove async 2022-03-03 19:31:51 -08:00
Jackson Harper
05373ba3c7 add methid to parse content metadata 2022-03-03 17:10:06 -08:00
Jackson Harper
c2e08d0e8f Fetch title and author from page metadate if possible 2022-03-03 15:20:58 -08:00
Jackson Harper
b17f22949c Use the readability title if available 2022-03-03 14:21:30 -08:00
Jackson Harper
a4bb67deee Use the readability byline for email authors if available 2022-03-03 14:21:03 -08:00
Jackson Harper
b326a5f8e7 Add more matches on substack icons 2022-03-03 13:42:30 -08:00
Jackson Harper
484cd78ac5 prettier 2022-03-02 23:14:10 -08:00
Jackson Harper
65ce8353dc Attempt to pull URLs for probable newsletter emails out of content 2022-03-02 23:09:10 -08:00
Jackson Harper
21329949e5 Fix to email address in saveNewsletterEmail 2022-03-02 22:06:15 -08:00
Jackson Harper
02d505a7e2 Add some debugging 2022-03-02 21:39:10 -08:00
Jackson Harper
8fe9827ad2 If an email appears to be a newsletter at it to the library 2022-03-02 20:47:53 -08:00
Jackson Harper
65cc666579 Pass HTML instead of a JSDOM into isProbablyNewsletter to better encapsulate 2022-03-02 20:38:11 -08:00
Jackson Harper
c4e237927d Allow any on GCP func 2022-03-02 20:27:40 -08:00
Jackson Harper
e660e41ed5 Fix warnings 2022-03-02 19:49:10 -08:00
Jackson Harper
9206230659 Better name for the save newsletter service 2022-03-02 19:45:28 -08:00
Jackson Harper
d66c114a7d Remove file, this is moved into the parser 2022-03-02 16:36:36 -08:00
Jackson Harper
b5f9478350 Dont mutate function input 2022-03-02 16:34:56 -08:00
Jackson Harper
f7f83fe080 New function to determine if an HTML blob is probably a newsletter based on its content 2022-03-02 16:31:15 -08:00
Jackson Harper
505f888e37 Update to latest intercom client 2022-03-01 11:20:31 -08:00
Hongbo Wu
fc9aa9452c
Add a flag in readability to retain table elements in newsletter emails (#152) 
* add a flag in readability to retain table elements in newsletter emails

* remove header of axios newsletters
 2022-03-01 11:49:38 +08:00
Hongbo Wu
7bf454ae91
use dataloader to fetch all labels of a list of linkIds in a single q… (#133) 
* use dataloader to fetch all labels of a list of linkIds in a single query and cached

* add labels in GQL query in frontend
 2022-02-28 12:13:26 +08:00
Jackson Harper
328ebc48cb Apply code block highlighting before running DOM clean 2022-02-26 19:04:32 -08:00
Jackson Harper
fd39923907 Prettier improvements 2022-02-26 16:38:43 -08:00
Jackson Harper
84fbc9cd27 Add code highlighting using highlight.js 2022-02-26 14:57:59 -08:00
Hongbo Wu
42f2cffdf8 remove debug log 2022-02-24 14:47:33 +08:00
Hongbo Wu
276205d52a add linkId and labels in article type 2022-02-24 14:44:49 +08:00
Hongbo Wu
c7841b8e8a remove duplicate links because of join with link_labels table 2022-02-24 14:44:49 +08:00
Hongbo Wu
fc18004a5d add labels in search query 2022-02-24 14:44:49 +08:00
Hongbo Wu
edb9339897 add labels and link_labels table name 2022-02-24 14:44:49 +08:00
Hongbo Wu
5651c92efe ingore case when checking name uniqueness 2022-02-24 14:44:49 +08:00
Hongbo Wu
9599c98c77 add label filters in search query 2022-02-24 14:44:49 +08:00
Hongbo Wu
fc37d06fd2 update params variable 2022-02-24 14:44:49 +08:00
Hongbo Wu
d33b213092 add setLabels api and tests 2022-02-24 14:44:48 +08:00
Hongbo Wu
9150e7dbc2 add tests for create/delete labels 2022-02-24 14:44:48 +08:00
Hongbo Wu
e74f334db9 update entity 2022-02-24 14:44:48 +08:00
Jackson Harper
5f6f3ba3b2 Fix warnings in the API package 2022-02-21 16:46:10 -08:00
Jackson Harper
a276af5264 Remove extra white space 2022-02-21 13:55:09 -08:00
Jackson Harper
cf6733c70f When fetching the library item list dont add content/originalHtml from DB 2022-02-21 13:55:09 -08:00
Jackson Harper
5af4899415 Use null instead of undefined when unarchiving links 
This fixes unarchiving content by re-saving it via the Save*
API methods.

Setting to undefined is not the same as null in this case, and
will result in archived_at being non-null and the link then
being excluded from the GetArticles query.
 2022-02-21 11:26:22 -08:00
Hongbo Wu
4c84c40f97
add slug to the link (#84) 2022-02-18 10:43:34 +08:00
Jackson Harper
05aa5d737e
Merge pull request #82 from omnivore-app/feature/bloomberg-newsletters 
Add support for Bloomberg newsletter
 2022-02-18 01:08:46 +08:00
Hongbo Wu
9431e80be8 add parser 2022-02-17 20:17:02 +08:00
Hongbo Wu
e3839647e1 parse bloomberg newsletter 2022-02-17 20:14:06 +08:00
Hongbo Wu
ad8bca3d84 use subject as title 2022-02-17 18:59:07 +08:00
Hongbo Wu
7229c64da0
fix: Should not get fake uploading url in demo (#79) 
* fix not getting fake uploading url in demo

* use internal service endpoint env

* update internal svc endpoint
 2022-02-17 13:34:50 +08:00
Hongbo Wu
bd8178a464 add support for bloomberg newsletters 2022-02-17 13:19:58 +08:00
Jackson Harper
3a27681534
Merge pull request #74 from omnivore-app/chore/upgrade-gql-typegen-2.6.2 
Upgrade gql-codegen, regenerate GQL schema
 2022-02-17 04:03:31 +08:00
Jackson Harper
e80eb7ab06 Upgrade gql-codegen, regenerate GQL schema 2022-02-16 11:32:13 -08:00
Hongbo Wu
c1ffd9abdf fix sanitize bug by having different SanitizedString type name when creating new instances 2022-02-16 09:33:33 -08:00
Jackson Harper
c8abf87367 Cast the directive types 2022-02-16 09:33:33 -08:00
Jackson Harper
6d5554dd5f Implement the sanitize directive in applo 3.0 2022-02-16 09:33:33 -08:00
Jackson Harper
7e35281bd1 createApp is no longer async, linting 2022-02-16 09:33:33 -08:00
Jackson Harper
fb224d60c4 Remove subscriptions, comment out the sanitize directive 2022-02-16 09:33:30 -08:00
Jackson Harper
df2b36b33a Upgrade apollo-server-express, remove connections/subscriptions 2022-02-16 09:32:48 -08:00
Jackson Harper
5a457479e2
Merge pull request #64 from omnivore-app/feature/handle-pdf-attachment 
add pdf attachment to library
 2022-02-17 00:55:27 +08:00
Hongbo Wu
d53e94aeb2 add tests for create article 2022-02-16 16:14:14 +08:00
Hongbo Wu
77ba7531f3 add tests for upload 2022-02-16 15:26:11 +08:00
Hongbo Wu
9460197132 add more logs for debugging 2022-02-16 11:34:01 +08:00
Hongbo Wu
b826a101c5 add pdf attachment create-article 2022-02-16 09:24:16 +08:00
Jackson Harper
c3308085ee Upgrade gql-typegen and regen types 2022-02-15 10:42:05 -08:00
Hongbo Wu
5d95e6c734 add pdf attachment upload 2022-02-15 22:22:13 +08:00
Hongbo Wu
e85273d87a
add support for axios newsletters (#49) 
* add support for axios newsletters

* fix raw url

* fix getting newsletter url from axios

* add test for url

* add axios parser
 2022-02-15 11:56:03 +08:00
Jackson Harper
65cd8a95eb Better handling of situations where we fail to set the cookie during auth 2022-02-14 16:04:23 -08:00
Jackson Harper
165085a639 Handle upgraded axios API for headers 2022-02-14 15:48:36 -08:00
Hongbo Wu
a04e91472d
fix a bug of using article as table name in the reminders query (#34) 
* fix a bug of using article as table name in the reminders query

* fix test

* increase timeout value

* increase timeout value to 50000

* add import mocha

* add import mocha for label test

* add import mocha for every test

* fix test in deleting labels

* add timeout for each label test

* one more test

* fix reminders router by allowing sendrid template id to be null

* do not retry if reminders not found
 2022-02-14 13:59:04 +08:00
Jackson Harper
28a208ba2c Remove some unneeded logging on startup 2022-02-12 09:16:45 -08:00
Jackson Harper
ca7b79d1ba Simplify docker compose setup for new users 
This takes out some environment variables that are not needed
for new users testing out locally.

We also adjusted the port usage so docker-compose will use the
same ports that local dev typically uses.
 2022-02-11 11:45:37 -08:00
Jackson Harper
84f32935f5 Open source omnivore 2022-02-11 09:24:33 -08:00