prehistoric-systems/omnivore
1
0
Fork 0
mirror of https://github.com/omnivore-app/omnivore.git synced 2026-03-11 08:54:26 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
2889 commits 201 branches 31 tags 432 MiB
Commit graph

463 commits

Author SHA1 Message Date
Hongbo Wu
4cf45af669 Add debug log for deleting label 2022-06-09 12:33:42 +08:00
Hongbo Wu
85317df671
Merge pull request #774 from omnivore-app/fix/missing-title 
Fallback to domain to url if no title
 2022-06-09 11:48:49 +08:00
Hongbo Wu
d1d18315c8 Fallback to domain to url if no title 2022-06-09 11:30:13 +08:00
Jackson Harper
fa2b58eac0 When creating a page during an uploadfile request return its ID 2022-06-08 20:17:47 -07:00
Hongbo Wu
309c581cf5
Merge pull request #729 from omnivore-app/labels-in-highlights 
Support labels in highlights
 2022-06-09 09:28:43 +08:00
Hongbo Wu
2dcab66af2 Update labels for both pages and highlights 2022-06-08 17:39:38 +08:00
Hongbo Wu
7c840f5b1e Merge deleteLabel in page and highlight 2022-06-08 17:25:33 +08:00
Hongbo Wu
a981e13c86 Add test case for deleteLabel 2022-06-08 17:01:38 +08:00
Hongbo Wu
0d67d0fb90 Delete labels for highlights when label is deleted 2022-06-08 15:35:52 +08:00
Hongbo Wu
22c604c15b Add deleteLabelForHighlights 2022-06-08 15:33:43 +08:00
Hongbo Wu
145867fe05 Add test for setLabelsForHighlight resolver 2022-06-08 15:25:11 +08:00
Hongbo Wu
4894b3cfd9 Add setLabelsForHighlight in function_resolvers.ts 2022-06-08 15:00:05 +08:00
Hongbo Wu
e9f4149c62 Add setLabelsForHighlight resolver 2022-06-08 14:59:46 +08:00
Hongbo Wu
eef3ee4114 Add getLabelsByIds service method 2022-06-08 14:59:27 +08:00
Hongbo Wu
93c0e53d1d Generate setLabelsForHighlight graphql code 2022-06-08 14:59:11 +08:00
Hongbo Wu
427b58eedd Add set labels for highlight gql schema 2022-06-06 19:15:51 +08:00
Hongbo Wu
bdda54d54a Add set Labels For Highlight data access method 2022-06-06 19:15:03 +08:00
Hongbo Wu
aa15ef0282 Sort by expires at desc nulls last 2022-06-06 13:34:23 +08:00
Hongbo Wu
d00becc269 Add readAt to search API response 2022-06-04 22:12:59 +08:00
Hongbo Wu
fc68d51320 Support sort/filter by readAt 2022-06-04 22:12:03 +08:00
Hongbo Wu
b1d968ff23 Update readAt when saving reading progress 2022-06-04 22:11:44 +08:00
Hongbo Wu
577c6e84fb Add readAt to page type 2022-06-04 22:11:14 +08:00
Jackson Harper
b61a8f6015
Merge pull request #747 from omnivore-app/fix/ios-sorting 
Disable sorting options that need API improvements
 2022-06-03 14:04:19 -07:00
Jackson Harper
791e31d60f
Merge pull request #750 from omnivore-app/OMN-633 
Reopening OMN-633 to rebase
 2022-06-03 13:36:36 -07:00
Jackson Harper
b56c6dafa9 Disable sorting options that need API improvements 2022-06-03 12:47:43 -07:00
Jackson Harper
bf929db559 Add the updatedAt attribute to page items 2022-06-03 10:24:11 -07:00
Jackson Harper
975411d429
Merge pull request #719 from omnivore-app/fix/pdf-uploading-improvements 2022-06-02 16:47:03 -07:00
Hongbo Wu
89164a353f
Merge pull request #723 from omnivore-app/fix/tweet-not-saved 
Fix tweet not saved
 2022-06-02 11:20:40 +08:00
Hongbo Wu
4f031a579c
Merge pull request #721 from omnivore-app/search-highlights-second-index 
When searching for highlights we should include document title, descr…
 2022-06-02 11:05:30 +08:00
Hongbo Wu
04f7fdc31f Add labels in highlight type 2022-06-01 19:01:26 +08:00
Hongbo Wu
195074fa49 Update search query 2022-06-01 13:49:51 +08:00
Hongbo Wu
304fe70113 Fix tests 2022-06-01 10:30:23 +08:00
Jackson Harper
11f1b78c45 Remove changes to upload.ts 2022-05-31 15:26:33 -07:00
Jackson Harper
0e10d5630d Rebase and merge 2022-05-31 15:11:10 -07:00
Hongbo Wu
404805e0c0 Make async calls to parse() 2022-05-31 22:51:00 +08:00
Hongbo Wu
74054cbdbd
Merge pull request #716 from omnivore-app/api-token 
Api tokens
 2022-05-31 14:22:23 +08:00
Jackson Harper
6f35740272 Linting fixes 2022-05-30 22:38:24 -07:00
Jackson Harper
53c423b714 WIP: upload PDFs using background task 2022-05-30 22:30:43 -07:00
gitstart-omnivore
d42b0c244c fixed merge conflicts 2022-05-30 12:29:49 +00:00
Hongbo Wu
a04bfbb83e When searching for highlights we should include document title, description, content as secondary indexes 2022-05-29 22:43:21 +08:00
Hongbo Wu
c0bb67a6b5 Verify api key for routers too 2022-05-28 22:00:01 +08:00
Jackson Harper
987336f438
Merge pull request #712 from omnivore-app/fix/ios-pdf-handling 
Better PDF upload handling on iOS
 2022-05-27 08:43:07 -07:00
Hongbo Wu
6e16631f3b Resolve conflicts 2022-05-27 18:37:10 +08:00
Hongbo Wu
fa9c5b3efe Add test for list api keys 2022-05-27 18:36:18 +08:00
Hongbo Wu
b7206457b4 Add test for revoke api key 2022-05-27 18:36:17 +08:00
Hongbo Wu
0f2a55d274 Add list/delete api key resolver 2022-05-27 18:35:38 +08:00
Hongbo Wu
fd4fc8a4e7 Add list/delete api key in schema 2022-05-27 18:34:36 +08:00
Hongbo Wu
333b0259ba Fix tests 2022-05-27 18:33:49 +08:00
Hongbo Wu
9051a3f43a Update api key validation 2022-05-27 18:33:49 +08:00
Hongbo Wu
a09588d3a7 Update generateApiKey resolver 2022-05-27 18:33:49 +08:00
Hongbo Wu
849adf84d4 Add entity class 2022-05-27 18:33:49 +08:00
Jackson Harper
0cabd26344 Dont send the normalized URL to the content-fetch service 
URLs need to be sent to contentFetch exactly as they
were received, this is because many apps will use signed URLs
for PDFs being served from GCP or AWS. A signed URL needs to
keep all its query params in the same order to validate.
 2022-05-26 21:40:40 -07:00
Jackson Harper
93b095818d Linting fixes 2022-05-26 21:40:40 -07:00
Jackson Harper
cb98a9cf86 Make clients opt into creating a page when uploading a file 2022-05-26 21:40:40 -07:00
Hongbo Wu
26681e5ecd Add get/list/delete webhook resolver tests 2022-05-26 14:52:01 +08:00
Hongbo Wu
a75ce3c44d Add get/list/delete webhook resolver 2022-05-26 14:52:00 +08:00
Hongbo Wu
cc07c00a83 Add get/list/delete webhook gql schema 2022-05-26 14:51:11 +08:00
Hongbo Wu
c9c6f83e65 Add tests for trigger endpoint 2022-05-26 14:51:11 +08:00
Hongbo Wu
7b0ebb9dbf Add pubsub subscription to trigger webhook 2022-05-26 14:51:11 +08:00
Hongbo Wu
7157f38738 Add test 2022-05-26 14:51:11 +08:00
Hongbo Wu
6d0ce30a44 Add setWebhook resolver 2022-05-26 14:51:10 +08:00
Hongbo Wu
5528d1d349 Replace createWebhook with setWebhook 2022-05-26 14:50:38 +08:00
Hongbo Wu
f61f61312c Add webhook entity 2022-05-26 14:50:10 +08:00
Hongbo Wu
a655275754 Add unique constraint on user_id and event_types 2022-05-26 14:50:10 +08:00
Hongbo Wu
aaab02c9d8 Add createWebhook gql schema 2022-05-26 14:50:08 +08:00
Hongbo Wu
7d4d1d7b67 Parse language in readability 2022-05-26 10:55:21 +08:00
Hongbo Wu
0ef9bb3944 Allow search by language in elastic 2022-05-26 10:55:20 +08:00
Hongbo Wu
87b0f683e1 Add language term in search body 2022-05-26 10:55:04 +08:00
Hongbo Wu
6e06429636 Add language keyword in search query 2022-05-26 10:55:02 +08:00
Hongbo Wu
1eeb5617a5 Search by language 2022-05-26 10:54:04 +08:00
Hongbo Wu
dd394144cd Add language in schema 2022-05-26 10:52:19 +08:00
Hongbo Wu
a702993485 Add search by language query 2022-05-26 10:52:18 +08:00
Hongbo Wu
3e0e8b486f Add language in searchitem 2022-05-26 10:52:18 +08:00
Hongbo Wu
be91562ca1 Add language in page 2022-05-26 10:52:18 +08:00
Jackson Harper
d704fad2ff
Merge pull request #701 from omnivore-app/feature/add-popular-reads 
API to add recommended (popular) reads to a users library
 2022-05-24 21:09:24 -07:00
Hongbo Wu
f4c65d7fe1
Merge pull request #698 from omnivore-app/sort-by-publish-time 
Support sort or filter by publish time
 2022-05-25 10:41:49 +08:00
Jackson Harper
a20ba60053 Add popular reads for onboarding 2022-05-24 15:31:55 -07:00
Jackson Harper
ea50f51644 Update to use image proxy URLs in parsed content 2022-05-24 13:43:57 -07:00
Jackson Harper
ed5dfde1a8 Linting fixes, remove debug 2022-05-24 13:06:40 -07:00
Jackson Harper
2b14d51ec6 API to add recommended (popular) reads to a users library 
This API is used in onboarding
 2022-05-24 12:05:39 -07:00
Hongbo Wu
a35b096d73 Support sort and filter by published in resolver 2022-05-24 18:25:32 +08:00
Hongbo Wu
61065efd2b Add published sort param in elastic query 2022-05-24 18:24:30 +08:00
Hongbo Wu
641f191fcd Add published sort param in schema 2022-05-24 18:23:30 +08:00
gitstart-omnivore
75728cc144 fixing diff 2022-05-24 07:04:54 +00:00
Jackson Harper
2d2495a10e
Merge pull request #685 from omnivore-app/fix/pdf-api-simplification 
Simplify the API to create pages from PDFs
 2022-05-21 21:19:30 -07:00
Jackson Harper
e5bc68ca9e Simplify the API to create pages from PDFs 
This creates a stub page when an upload is initiated, so the
SaveFile (or create article from file) APIs do not need to be
called.

The PDF uploaded trigger fired by GCS will mark the upload
as completed and update the page status.
 2022-05-21 12:05:45 -07:00
Jackson Harper
17e6f4091d
Merge pull request #638 from omnivore-app/feature/subscription-api 
Feature/subscription api
 2022-05-20 15:34:48 -07:00
Hongbo Wu
2ee95a1c14 Fix cannot convert null to object error message 2022-05-20 20:38:30 +08:00
Hongbo Wu
444c21ad09 Fix labels test 2022-05-19 18:12:14 +08:00
Hongbo Wu
34349e925c Add unit test 2022-05-19 16:36:35 +08:00
Hongbo Wu
b8225fc8c8 Fix labels not being updated in elastic 2022-05-19 12:08:50 +08:00
Jackson Harper
eadeccce81 Linting fixes 2022-05-18 11:31:42 -07:00
Jackson Harper
004c766588 If parsing fails, attempt adding <html> wrappers to a document 
LinkedDom seems less forgiving and expects the outerHTML of a
document, however older extension versions still send innerHTML.
 2022-05-18 10:55:31 -07:00
Hongbo Wu
d68549bcb7 Remove unused code 2022-05-18 15:52:31 +08:00
Hongbo Wu
d9d4deb14c Update Money Stuff sign up url 2022-05-18 11:50:43 +08:00
Hongbo Wu
5b36f9270e Update milk road query params 2022-05-17 22:43:00 +08:00
Hongbo Wu
13453e7b7f Register subscribe resolver 2022-05-17 21:26:46 +08:00
Hongbo Wu
568472cc3c Calling HTTP requests to subscribe to these newsletters 2022-05-17 21:15:25 +08:00
Hongbo Wu
4699458457 Implement subscribe api 2022-05-17 18:46:57 +08:00
Hongbo Wu
1da8b75c81 Add a subscribeHandler 2022-05-17 18:46:44 +08:00
Hongbo Wu
51dd9a5733 Return a list of subscriptions 2022-05-17 18:45:52 +08:00
Hongbo Wu
c30ef093b8 Update unsubscribe api 2022-05-17 16:47:34 +08:00
Hongbo Wu
906b2d28a0 Add subscribe API schema 2022-05-17 16:23:36 +08:00
Hongbo Wu
e76fb02f43 Fix window is not defined for parsing code blocks 2022-05-17 11:01:55 +08:00
Jackson Harper
7bfb8cfee4
Merge pull request #597 from omnivore-app/remove-chrome-aws-lambda 
Optimize puppeteer and remove chrome-aws-lambda dependencies
 2022-05-13 16:12:24 -07:00
Jackson Harper
8b37a475c8
Merge pull request #612 from omnivore-app/fix/reminder-id-type 
Use pageId instead of articleSavingRequestId in reminders
 2022-05-13 10:47:46 -07:00
Jackson Harper
d3b65c0ff8
Merge pull request #611 from omnivore-app/fix/elastic-exception 
Stop throwing document not found exception to sentry
 2022-05-13 09:05:01 -07:00
Hongbo Wu
b306ca6e2e Fix UpdateLabelResult undefined 2022-05-13 17:38:51 +08:00
Hongbo Wu
45213e7b54 Fix reminder trigger 2022-05-13 17:07:10 +08:00
Hongbo Wu
a51eba67ba get reminder by page id 2022-05-13 16:17:25 +08:00
Hongbo Wu
d821f26440 create reminder with page id 2022-05-13 16:12:08 +08:00
Hongbo Wu
40cbfc9058 Use reminder by elasticPageId 2022-05-13 16:01:23 +08:00
Hongbo Wu
f51dbafdfc Stop throwing document not found exception to sentry 2022-05-13 11:40:08 +08:00
Jackson Harper
eee116e71d Add an analytics event when links are removed 2022-05-12 10:55:12 -07:00
Hongbo Wu
adfaf6b8ce Use newsletter's title and author as page's ones 2022-05-12 18:08:38 +08:00
Hongbo Wu
1b8850ed33 Fix tests 2022-05-12 17:41:11 +08:00
Hongbo Wu
e1e0ddf7fc
Merge pull request #582 from omnivore-app/optimize-parsing 
Optimize parsing
 2022-05-12 11:07:52 +08:00
Hongbo Wu
602d141dec Rename doc to dom 2022-05-12 11:00:32 +08:00
Hongbo Wu
5dca02873d Revert originalArticleUrl change 2022-05-12 10:45:13 +08:00
Hongbo Wu
8f75c43658 Add siteName in search result 2022-05-12 10:30:47 +08:00
Hongbo Wu
76aef35423 Remove www. from the siteName in the search api response 2022-05-11 23:07:34 +08:00
Hongbo Wu
a78a6c6ba4 Replace DomWindow with Document in handlers 2022-05-10 17:01:23 +08:00
Hongbo Wu
6a57281e74 Remove DomWindow usage 2022-05-10 17:00:56 +08:00
Hongbo Wu
acc7654a2f Replace jsdom with linkedom 2022-05-10 16:59:09 +08:00
Hongbo Wu
5698790288 Pass url to readability 2022-05-10 16:53:45 +08:00
Hongbo Wu
5007b11a3c Add url to readability option 2022-05-10 16:52:10 +08:00
Hongbo Wu
59a2639b7d Reduce http call to get jsonld data if title or content or sitename or byline exists 2022-05-09 13:45:45 +08:00
Hongbo Wu
a457c9d128 Update article content only when code blocks exist 2022-05-09 13:45:45 +08:00
Hongbo Wu
eaad96acdd Return parsed dom back to backend 2022-05-06 12:29:08 +08:00
Hongbo Wu
5f5076e864 Highlight code element without reinitialize jsdom 2022-05-06 12:20:54 +08:00
Hongbo Wu
7c6b810522 Remove redundant JSDOM 2022-05-06 10:53:36 +08:00
Hongbo Wu
93ab6bc87a Add includePending=true in search API 2022-05-04 21:28:06 +08:00
Hongbo Wu
bc1ed3f054 Check state in articleSavingRequest 2022-05-04 19:33:13 +08:00
Hongbo Wu
9e3db0e053 Fix a bug to have multiple pages with the same url in lib 2022-05-04 19:12:22 +08:00
Hongbo Wu
afe8b6e948 make savedAt a required field in page 2022-05-04 17:54:59 +08:00
Hongbo Wu
a42bb7ab38 Replace createdAt with savedAt 2022-05-04 12:59:07 +08:00
Hongbo Wu
98ea1016d5 Remove <p> tag in description 2022-05-04 12:50:22 +08:00
Hongbo Wu
9048ccbab3 Show failed pages in library 2022-05-04 12:46:57 +08:00
Hongbo Wu
ca0f58ee22 Update page content and state if failed to parse 2022-05-04 12:21:41 +08:00
Hongbo Wu
80d0146f72 Show unable to parse if page is failed to save within 30 seconds 2022-05-04 11:55:16 +08:00
Jackson Harper
c8c4377346
Merge pull request #531 from omnivore-app/fix/saving-analytics 
Add more detailed analytics for saving. Remove intercom events
 2022-05-02 21:14:18 -07:00
Jackson Harper
95b163a437 Prettier clean up 2022-05-02 20:58:05 -07:00
Jackson Harper
ab96bb09ab Add more detailed analytics for saving. Remove intercom events 2022-05-02 20:41:10 -07:00
Jackson Harper
b649d583b7 Linting changes 2022-05-02 15:58:40 -07:00
Jackson Harper
453b301e74 Allow looking up articles by ID 
iOS needs to be able to lookup an article by its ID not its
slug, because the slug can change once saved.
 2022-05-02 15:48:21 -07:00
Jackson Harper
61b7ac3cb2 Enable graphql introspection when running locally 2022-05-02 09:44:13 -07:00
Hongbo Wu
77ca0ceb59 Show empty description if not exist 2022-04-29 15:50:39 +08:00
Hongbo Wu
2b70d480d2
Remove article saving request (#493) 
* Add state and taskName in elastic page mappings

* Add state and taskName in elastic page interface

* Create page with PROCESSING state before scrapping

* Update createArticleRequest API

* Fix tests

* Add default state for pages

* Update createArticle API

* Update save page

* Update save file

* Update saving item description

* Show unable to parse content for failed page

* Fix date parsing

* Search for not failed pages

* Fix tests

* Add test for saveUrl

* Update get article saving request api

* Update get article test

* Add test for articleSavingRequest API

* Add test for failure

* Return new page id if clientRequestId empty

* Update clientRequestId in savePage

* Update clientRequestId in saveFile

* Replace article with slug in articleSavingRequest

* Add slug in articleSavingRequest response

* Depreciate article

* Use slug in web

* Remove article and highlight fragments

* Query article.slug on Prod

* Show unable to parse description for failed page

* Fix a bug having duplicate pages when saving the same url multiple times

* Add state in response

* Rename variables in removeArticle API

* Rename state

* Add state in response in web

* Make state an enum

* Open temporary page by link id

* Use an empty reader view as the background for loading pages

* Progressively load the article page as content is loaded

* Add includePending flag in getArticles API

* Set includePending = true in web

* Add elastic update mappings in migration script

* Add elastic mappings in docker image

* Move index_settings.json to migrate package

* Remove elastic index creation in api

* Move elastic migrations to a separate directory

* Remove index_settings from api docker image

Co-authored-by: Jackson Harper <jacksonh@gmail.com>
 2022-04-29 13:41:06 +08:00
Hongbo Wu
ccb79845bc
Fix subscriptions issues (#501) 
* Replace fetch with axios

* Add url in subscription

* Remove url in subscription

* Unsubscribe all newsletters before deleting that email

* Continue unsubscribing others if one got error
 2022-04-29 11:16:46 +08:00
Hongbo Wu
d1c2721b95
Add expiration date to api key generation (#495) 
* Add exp to jwt token

* Add expiredAt in generate api key input

* Check exp in claim

* Add tests
 2022-04-28 09:09:40 +08:00
Hongbo Wu
bcbda50daa Add basic UI for subscriptions 2022-04-22 12:17:09 -07:00
Hongbo Wu
3e35fb1aca Email content display report 2022-04-22 19:14:59 +08:00
Hongbo Wu
71aa66ea9c
Allow search for subscription:name (#461) 2022-04-22 08:47:41 +08:00
Jackson Harper
0a4d55ef15 Increase the rate limit 
Turns out 10 is very low because we make CORs requests from the
browser and SWR will make background requests for some objects
like viewer. I was hitting the limit while testing search.
 2022-04-21 14:59:57 -07:00
Hongbo Wu
dca6461b62
Add unsubscribe resolver (#453) 
* add unsubscribe resolver

* add subscription to articles api response

* revert yarn.lock changes
 2022-04-21 13:18:15 +08:00
Hongbo Wu
83cdec42f7
Add rate limiter for api request (#454) 
* Add rate limiter for api request

* apply rate limit on Demo/Prod
 2022-04-21 09:41:33 +08:00
Hongbo Wu
5c7f4a22bf
Save subscriptions when saving newsletters (#447) 
* parse unsub mailTo or httpUrl and send to pubsub

* save subscription from newsletter

* fix tests
 2022-04-20 09:24:40 +08:00
Jackson Harper
48e36c8635
Merge pull request #393 from omnivore-app/OMN-382 
[Omn-382] - Updated landing page
 2022-04-19 11:15:31 -07:00
Hongbo Wu
d7a2659fc4
fix searching for multiple labels (#444) 2022-04-19 12:13:47 +08:00
Hongbo Wu
1117a0c575
Feature/subscription list resolver (#432) 
* add subscriptions table

* add listSubscriptions schema

* add listSubscriptions resolver
 2022-04-19 11:08:43 +08:00
Jackson Harper
c2fd58a809 Make landing a restricted username page 2022-04-18 15:22:59 -07:00
Hongbo Wu
0b2b93384f fix return undefined as updateLabelResult 2022-04-18 20:45:06 +08:00
Jackson Harper
d33d383891
Merge pull request #422 from omnivore-app/fix/reminder-data-model-error 
Fix reminders resolver getting pages from postgres
 2022-04-14 14:10:48 -07:00
Jackson Harper
c80ce4da10 Consistent naming convention for events 2022-04-14 10:31:10 -07:00
Jackson Harper
2c9c2a2adf
Merge pull request #394 from omnivore-app/header-and-reader-improvements 
Add margins on wider screens, keyboard commands to adjust
 2022-04-13 21:26:24 -07:00
Hongbo Wu
c0966d908f
upload highlight and label to gcs if successfully updated in elastic (#420) 2022-04-14 12:14:15 +08:00
Hongbo Wu
88d8ca7b7d fix reminders resolver getting page from postgres 2022-04-14 12:09:17 +08:00
Jackson Harper
e282fc8316
Merge pull request #417 from omnivore-app/feature/upload-highlight-gcs 
Upload highlights and labels data to GCS
 2022-04-13 13:41:29 -07:00
Hongbo Wu
0b0b521fab use generic method for uploading 2022-04-13 22:21:20 +08:00
Hongbo Wu
a806f55233 upload highlight to gcs 2022-04-13 21:18:58 +08:00
Hongbo Wu
8ba1be456c fix sanitizer invalid name 2022-04-13 13:23:21 +08:00
Hongbo Wu
3b831fecb8 test both max length and pattern if declared 2022-04-13 12:20:41 +08:00
Hongbo Wu
928ddd69a9 validate color input in label as rgb hex value 2022-04-13 12:17:12 +08:00
Jackson Harper
73ef8fef09 Rename linkId to pageId in SetLabelsInput 2022-04-12 12:46:32 -07:00
Jackson Harper
d05f091876 Add debug logging for GQL errors 2022-04-12 12:23:55 -07:00
Jackson Harper
6c5e88cf12
Merge pull request #405 from omnivore-app/feature/label-max-length 
Set label name max length = 64 char
 2022-04-12 11:14:20 -07:00
Hongbo Wu
ae0d1dd2ee
Feature/search highlights backend (#395) 
* add highlight mappings

* return highlight in resolvers

* temporarily skip highlight tests

* add test for getting highlights

* update merge highlight

* separate elastic methods

* roll back merge highlight test

* add highlight to elastic script

* update delete highlight in elastic

* migrate highlight data from postgres to elastic

* rescue not found exception when page is not found in the migration script

* exclude highlights in searching pages results

* search pages with highlights only with has:highlight query

* add search endpoint to search pages or highlights

* reduce code smell in search api

* fix rebase error

* fix tests

* add test for search highlight

* add test for new search endpoint

* add labels to search results

* update schema

* update search query

* fix update/share highlights

* fix rebase error

* fix tests

* add highlight model in elastic

* add savedAt and publishedAt date range in search query

* add sort by updated and recently read

* fix tests

* close db connection when tests are done

* test github action

* revert github action test

* fix rebase error

* add docker-compose for api-test

* remove unused env

* remove highlights with no page attached to

* allow get_articles resolver to search for query so we can merge it without web changes
 2022-04-12 12:31:08 +08:00
Hongbo Wu
2ebdaba780
add generate api key api and test (#392) 
* add generate api key api and test

* test if user can make api call with the api key
 2022-04-12 12:11:45 +08:00
Hongbo Wu
bb665a704b make label name <= 64 char length 2022-04-12 12:08:02 +08:00
Hongbo Wu
b4420839bd
fix: fail to search by excluding labels with capital letter (#400) 
* fix: fail to search by excluding labels with capital letter

* make all the excluded labels lowcased

* Revert "make all the excluded labels lowcased"

This reverts commit 866bed4080.
 2022-04-11 14:05:21 +08:00
Jackson Harper
48ca13818f Prettier fixes 2022-04-06 10:45:01 -07:00
Jackson Harper
57f865c9a2 Update to typeorm 0.3 2022-04-06 10:37:13 -07:00
Jackson Harper
1cdb7753c3 Update typeorm usage 2022-04-06 09:56:32 -07:00
Jackson Harper
06360a5993
Merge branch 'main' into OMN-190 2022-04-06 09:24:49 -07:00
Hongbo Wu
d1269f815e identify a newsletter by checking both list-url and unsubscribe-url in headers and email sender address 2022-04-06 20:59:14 +08:00
Hongbo Wu
7569e988bf
upgrade typeorm to 3.0 (#359) 
* upgrade typeorm to 3.0

* use new datasource object in typeorm 3

* fix tests

* fix tests

* migrate before creating connection

* fail the test if migration failed
 2022-04-06 10:32:41 +08:00
Jackson Harper
9845a49040 Remove semicolon 2022-04-04 08:17:46 -07:00
Jackson Harper
c8ec7d636e Use require to import address parser function 2022-04-04 08:11:26 -07:00
Jackson Harper
92925b6808 Prettier fixes 2022-04-03 10:21:16 -07:00
Jackson Harper
da28998130 Pull ul list out of newsletter blurb 
next/react doesnt want child elements of the paragraphs

Improve formatting

Improve wording

Use buttons in the subscribe directly blocks

Simplify docs on setting up forwarding rules

Add extra padding on bottom of help docs

Remove unused style

Add emails help page

Improve formatting

Prefetch page content on iOS

Reduce the reader overly length now that items are precached

Add invalidation when highlights are added to items

fix missing index_settings.json file in api dockerfile for creating elastic index (#363)

Handle full email address objects in the to param from sendgrid

These come in a format like:

"jacksonh-dfdf@inbox.omnivore.app" <jacksonh-dfdf@inbox-demo.omnivore.app>

New IDs for short highlights because they dont cascade delete now

Testing CI issues

Simplify test

CI test

Use promises for async tests

Temporarily remove test to debug CI

Re-enable

re-enable test, return error

Specify a userId when looking up saved email pages

create a unique url for newsletters without a URL

Use 500ms on page test timeouts

Increase timeout

Dont use deep equal to match newsletter label

Run just the labels API

Run against just the newsletter emails

Run without the page tests

Fix

Set the allow uncaught flag

Remove highlight tests

Remove newsletters tests

more resolver tests

Remove newsetter tests

Comment out resolver tests

Use nock for external requests in tests

Specify puppeteer url for tests

Comment out more tests

uncomment tests

re-enable

re-enable email test

Re-disable

Re-enable one pdf attachment test

Re-disable pdf attachment test

Use promises on setTimeout tests

rm label tests

mv label tests into a context

Comment out pdf tests

Comment out pdf tests

Async test

Async wrappers

Delay when creating test pages

More debugging

Unique short ids

Remove potentially problematic test

Fetch page before returning for test

handler in before block

more debugging

More debugging

Move errors checks into contexts

Use a context when saving newsletters to force index refresh

Prettier fix

Fix newsletter label check, remove setTimeout

Re-enable test

timeout on pdf router handler

Fix method call

comment out PDF test

Unique fake username

Comment out PDF test

Debugging signed urls

Re-enable

New email

pdf test

PDF tests

Comment out pdf test

Add nock stubs for email URLs

Use full address for PDF test

Remove debug

Use full email addresses
 2022-04-02 16:56:24 -07:00
gitstart-omnivore
1978a92a1b Effected review corrections 2022-03-31 13:03:50 +00:00
Jackson Harper
a8faaf835c Remove debug sort on labels 2022-03-30 14:13:26 -07:00
Jackson Harper
2c15516d96 Order labels by createdAt so they maintain a consistent order 2022-03-30 13:28:37 -07:00
gitstart-omnivore
11db7be473 [Omn-190] - [Settings View] - Labels 2022-03-30 12:24:29 +00:00
Hongbo Wu
6d405432af
add site_name and site_icon to page model and return in resolver (#341) 
* add site_name and site_icon to page model and return in resolver

* fix tests
 2022-03-30 10:43:10 +08:00
Jackson Harper
2bada0b9df
Merge pull request #293 from omnivore-app/feature/priority-by-rate-limit 
Use lower priority queue for large imports
 2022-03-28 13:19:35 -07:00
Jackson Harper
d9af397b9c
Merge pull request #331 from omnivore-app/fix/update-page-after-deletion 
fix document_missing_exception being thrown to sentry when updating page in elastic
 2022-03-28 10:30:31 -07:00
Hongbo Wu
fd18affaec fix document_missing_exception being thrown to sentry when updating page in elastic 2022-03-28 21:16:33 +08:00
Hongbo Wu
b6c04f3508 fix saving report item by linking to elastic page 2022-03-28 20:38:54 +08:00
Jackson Harper
1cdbfaca26
Merge pull request #294 from omnivore-app/feature/add-created-or-updated_at-to-newsletter_email 
add created_at and updated_at to newsletter emails and sort by create…
 2022-03-25 09:10:54 -07:00