prehistoric-systems/omnivore
1
0
Fork 0
mirror of https://github.com/omnivore-app/omnivore.git synced 2026-03-11 08:54:26 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
12057 commits 201 branches 31 tags 432 MiB
Commit graph

2536 commits

Author SHA1 Message Date
Hongbo Wu
e23d3d5933 update readable content instead 2024-04-12 13:26:11 +08:00
Hongbo Wu
f95642be2b remove redundant field 2024-04-11 18:47:19 +08:00
Hongbo Wu
448c78a87e add dotenv 2024-04-11 18:25:24 +08:00
Hongbo Wu
54e7c52b5e always update the state of the pdf item to succeeded/failed, even if we fail to parse the content 2024-04-11 18:07:53 +08:00
Hongbo Wu
f66d3fa36c migrate inbound-email tasks from pubsub to bullmq 2024-04-11 11:08:22 +08:00
Hongbo Wu
cd6594a2fd add logs 2024-04-11 11:08:22 +08:00
Hongbo Wu
67657773cf track the usage of the api 2024-04-11 11:08:22 +08:00
Hongbo Wu
cd6f3e6bbe enforce usage limits for the API 2024-04-11 11:08:22 +08:00
Hongbo Wu
5f239d2dcb update graceful shutdown 2024-04-11 11:08:22 +08:00
Hongbo Wu
f2f18eb12b reply only allowed text 2024-04-11 11:08:22 +08:00
Hongbo Wu
e7fc1316dd enqueue reply email job 2024-04-11 11:08:22 +08:00
Hongbo Wu
74f5b922dd update received email reply 2024-04-11 11:08:22 +08:00
Hongbo Wu
dce0b15f4f send to the reply-to address if it exists or the from address 2024-04-11 11:08:22 +08:00
Hongbo Wu
764b05fff4 save reply-to in recent emails 2024-04-11 11:08:22 +08:00
Hongbo Wu
5cc5c559ac add reply to email API which replies Okay to the sender 2024-04-11 11:08:22 +08:00
Hongbo Wu
a59c850c5f fix telegram channel 2024-04-09 19:44:16 +08:00
Hongbo Wu
e7fc2c4567 reset failed_at to null if a feed is refreshed successfully after failure 2024-04-09 18:39:17 +08:00
Hongbo Wu
77c11fb80f
Merge pull request #3767 from omnivore-app/fix/verification 
Enqueue emails
 2024-04-08 12:30:37 +08:00
Hongbo Wu
ec309f5b0d verify captcha in email-login and forgot-password api 2024-04-08 12:27:29 +08:00
Hongbo Wu
7abb396032 remove stub 2024-04-05 22:53:03 +08:00
Hongbo Wu
af140cc8b4 remove user name from email template 2024-04-05 16:59:23 +08:00
Hongbo Wu
8af2bb076c rename send mail functions 2024-04-05 16:59:20 +08:00
Hongbo Wu
c37b31dcb6 fix tests 2024-04-05 16:57:02 +08:00
Hongbo Wu
10dcb922f2 Enqueue confirmation email 2024-04-05 16:57:02 +08:00
Hongbo Wu
207241934d show label_attached instead of label_created in rules 2024-04-05 16:51:11 +08:00
Hongbo Wu
25fb9fd698 deduplicate labels when there are case-insensitive identical names 2024-04-05 16:27:17 +08:00
Hongbo Wu
6f03b214b5 allow delete internal labels 2024-04-05 15:58:39 +08:00
Hongbo Wu
9f4f1d07ca do not create label if label name case-insensitively matches existing label name 2024-04-05 15:50:01 +08:00
Hongbo Wu
906b6c1ea5 fix web 2024-04-04 18:53:10 +08:00
Hongbo Wu
58708a507e use old event format for the compatibility with the old webhooks 2024-04-04 18:22:18 +08:00
Hongbo Wu
45a5382d7c do not send bad_request error if db failure 2024-04-04 12:19:18 +08:00
Hongbo Wu
5e7a82562b resolve conflicts after rebasing 2024-04-04 12:19:17 +08:00
Hongbo Wu
b2cd43136f hide webhooks page 2024-04-04 12:19:16 +08:00
Hongbo Wu
0fab3c7904 Add highlight_updated event 2024-04-04 12:17:28 +08:00
Hongbo Wu
88a7e8d85b fix tests 2024-04-04 12:17:15 +08:00
Hongbo Wu
81f21141fe optimize ts-node-dev 2024-04-04 12:17:15 +08:00
Hongbo Wu
841a0026ff do not send page_update event for thumbnailer 2024-04-04 12:17:15 +08:00
Hongbo Wu
1b24a24f7c remove slug from highlight event 2024-04-04 12:17:15 +08:00
Hongbo Wu
d67aea6d8a deduplicate notion page by omnivore id 2024-04-04 12:17:13 +08:00
Hongbo Wu
0428da3b71 fix notion syncing error 2024-04-04 12:16:45 +08:00
Hongbo Wu
6954fffd24 fix readwise syncing error 2024-04-04 12:16:26 +08:00
Hongbo Wu
ff78a4b266 fix readwise syncing error 2024-04-04 12:16:12 +08:00
Hongbo Wu
f35ac036eb add UI for export in rules 2024-04-04 12:15:28 +08:00
Hongbo Wu
ad2e8ee002 trigger export in rules 2024-04-04 12:14:38 +08:00
Hongbo Wu
58511049a2 rename LABEL_CREATED to LABEL_ATTACHED in webhook data 2024-04-04 12:02:29 +08:00
Hongbo Wu
e2b3235046 delete unneccessary properties from webhook data 2024-04-04 12:02:29 +08:00
Hongbo Wu
88d6455222 default filter is in:all 2024-04-04 12:02:29 +08:00
Hongbo Wu
c551d9fe6b allow send data to webhook with rules 2024-04-04 12:02:29 +08:00
Hongbo Wu
07cee98ff9 allow label and highlight events to trigger rules 2024-04-04 12:02:29 +08:00
Hongbo Wu
e28dd73f9c
Merge pull request #3780 from omnivore-app/fix/notion 
sync items and highlights to user created database in notion
 2024-04-04 11:58:59 +08:00
Hongbo Wu
10cca7f05d find the title property and update its name to Title 2024-04-04 10:43:39 +08:00
Jackson Harper
c45a9d365b Also use mailjet for password changes 2024-04-03 22:25:31 +08:00
Hongbo Wu
491611b05f sync items and highlights to user created database in notion 2024-04-03 22:25:19 +08:00
Jackson Harper
3c1abb964b Also use mailjet for reset password emails 2024-04-03 22:07:28 +08:00
Hongbo Wu
f794cf707a
Merge pull request #3769 from omnivore-app/fix/notion 
export last 100 items in one job
 2024-04-03 10:22:56 +08:00
Hongbo Wu
08fb92d599 fix: site filter in rules matching both site name and url 2024-04-02 18:32:36 +08:00
Hongbo Wu
cc07d2bfbf remove comment 2024-04-02 17:56:46 +08:00
Hongbo Wu
d83dc17651 fix tests 2024-04-02 17:49:29 +08:00
Hongbo Wu
e19d6c86d8 export last 100 items in one job 2024-04-02 15:51:02 +08:00
Jackson Harper
aea354c3f2 No longer make captcha optional on web signup 2024-04-01 19:07:21 +08:00
Jackson Harper
3a59317217 Force captcha on sign up 2024-04-01 17:06:45 +08:00
Hongbo Wu
6ca092fbce use different prefix for different rate limiter 2024-04-01 13:44:10 +08:00
Hongbo Wu
a97542d4ba use redis for hourly rate limiter 2024-04-01 13:01:02 +08:00
Jackson Harper
788460c144
Merge pull request #3761 from omnivore-app/feat/signup-captcha 
Add recaptcha on the web sign up flow
 2024-04-01 12:24:39 +08:00
Jackson Harper
23f5377b62 Clean up isRecaptchaResponse function 2024-04-01 12:03:36 +08:00
Jackson Harper
0fe471da82 Dont require score on challenge recaptcha 2024-04-01 11:32:10 +08:00
Jackson Harper
822fcb1e74 Add some logging 2024-04-01 11:17:15 +08:00
Jackson Harper
9a159084c8 Update the isValidSignUp to check for URLs 2024-04-01 11:17:15 +08:00
Jackson Harper
442475d684 Make this optional for self hosters 2024-04-01 11:17:15 +08:00
Jackson Harper
7518e164a5 Utils for recaptcha 2024-04-01 11:15:19 +08:00
Hongbo Wu
5dd308d78e store the rate limit data in redis if redis is connected 2024-03-31 20:13:27 +08:00
Jackson Harper
2f6ef5e447 Return false if can not send with Mailjet 2024-03-31 19:22:56 +08:00
Jackson Harper
cfe4315fe2 Add an email fallback to Mailjet 2024-03-31 19:03:26 +08:00
Jackson Harper
200354d181 Add url present check on email sign up 2024-03-31 09:55:53 +08:00
Jackson Harper
ede99dcd33 Better matching for youtube video URLs 2024-03-30 10:42:01 +08:00
Jackson Harper
b69ba150c5 Remove debug 2024-03-29 12:28:26 +08:00
Jackson Harper
3a352a9705 Dont need to chunk this anymore 2024-03-29 12:05:32 +08:00
Jackson Harper
08ed061756 Change model for youtube transcripts 2024-03-29 11:31:27 +08:00
Hongbo Wu
f677f1167f get feature list on web 2024-03-28 15:27:37 +08:00
Hongbo Wu
cc2fe04568 add featureList to the User type which contains a list of feature type 2024-03-28 14:42:08 +08:00
Hongbo Wu
1afec1ddf3 check if feature is granted when process youtube video 2024-03-28 14:41:30 +08:00
Hongbo Wu
45417efd0c bump youtube users to 500 2024-03-28 12:15:36 +08:00
Hongbo Wu
8d8c7e8a26 fix: allow max 1000 users to use notion feature 2024-03-28 12:13:28 +08:00
Jackson Harper
b5bea17ffe Export the age of the oldest job in the queue 2024-03-28 10:25:44 +08:00
Jackson Harper
82168fe439
Merge pull request #3739 from omnivore-app/fix/youtube-queue 
Only check for summaries if the updated entity is a page item
 2024-03-28 09:40:44 +08:00
Hongbo Wu
d4ed994914
Merge pull request #3740 from omnivore-app/fix/logger 
replace logger.warn with warning
 2024-03-27 16:00:08 +08:00
Jackson Harper
38f959685f Change warn to warning 2024-03-27 15:41:15 +08:00
Hongbo Wu
92abfed2df replace logger.warn with warning 2024-03-27 15:41:13 +08:00
Jackson Harper
d23a7d56c0 Only check for summaries if the updated entity is a page item 2024-03-27 13:54:54 +08:00
Jackson Harper
5b736c1c2e
Merge pull request #3738 from omnivore-app/fix/api-youtube-logging 
Add more logging to youtube jobs
 2024-03-27 11:44:37 +08:00
Hongbo Wu
7c281359e8 fix recommendations not filtered correctly by rules 2024-03-27 11:29:47 +08:00
Jackson Harper
28222dcd56 Add more logging to youtube jobs 2024-03-27 11:25:46 +08:00
Hongbo Wu
c022cf0a22 label match wildcard 2024-03-27 10:54:24 +08:00
Hongbo Wu
3544dd1709 remove feed content from job data 2024-03-27 10:38:51 +08:00
Hongbo Wu
af78b6f4f0 append newsletter labels to avoid labels attached by rules being replaced 2024-03-27 10:37:04 +08:00
Hongbo Wu
41b90cbfd1 fix some undefined error 2024-03-26 17:53:48 +08:00
Hongbo Wu
d02d9444a9 remove content from update event too 2024-03-26 17:46:38 +08:00
Hongbo Wu
c3dba7e9da fix typo 2024-03-26 17:44:05 +08:00
Hongbo Wu
4c458c7118 fallback to search query if implicit fields are in the filter 2024-03-26 17:34:57 +08:00
Hongbo Wu
37459f64c7 remove content from job data 2024-03-26 16:49:26 +08:00
Hongbo Wu
209b08e2ff add notion to the features list 2024-03-26 12:32:01 +08:00
Hongbo Wu
f7c25fe05a do not create a new connection for the discover api 2024-03-23 10:03:06 +08:00
Hongbo Wu
282ed75a5f
Merge pull request #3698 from omnivore-app/fix/notion 
Fix: notion
 2024-03-21 14:08:16 +08:00
Hongbo Wu
445d2846b5 export last 100 items 2024-03-21 14:06:56 +08:00
Hongbo Wu
390d12eea1 add export to integration and get task api 2024-03-21 11:14:23 +08:00
Hongbo Wu
b1e3811e4f
Merge pull request #3707 from omnivore-app/fix/rule 
send labels and highlight annotations in page_create event
 2024-03-21 10:52:15 +08:00
Hongbo Wu
6385b40e38 revert batch size change 2024-03-20 23:07:46 +08:00
Hongbo Wu
3d986a9036 send labels and highlight annotations in page_create event 2024-03-20 22:33:24 +08:00
Jackson Harper
59cc954618 Add 12hrs option, fix linting issue 2024-03-20 15:36:53 +08:00
Jackson Harper
82ef57a36a Add last24hrs query for time based search queries to get recently saved/changed items 2024-03-20 15:15:18 +08:00
Jackson Harper
014ccb1eae Set published date from youtube data 
its not clear how consistent the formatting is on these dates,
the one i tested seems like it was a UTC formatted dd M, yyyy string
 2024-03-20 13:10:41 +08:00
Hongbo Wu
689eeaeacd disable sending pubsub events for rss feed temporarily to resolve conflicts 2024-03-20 10:46:13 +08:00
Hongbo Wu
2e374557bf send labels in the page_created events 2024-03-20 10:36:36 +08:00
Hongbo Wu
abf9706539 resolve conflicts after rebasing 2024-03-20 10:36:21 +08:00
Hongbo Wu
7f224cc1e1 running text search on all text fields 2024-03-20 10:36:21 +08:00
Hongbo Wu
edeafc9a71 fix bug 2024-03-20 10:36:21 +08:00
Hongbo Wu
de28a9ad5f fix bug 2024-03-20 10:36:21 +08:00
Hongbo Wu
f1eff130af filter data locally with liqe 2024-03-20 10:36:21 +08:00
Hongbo Wu
e87e6dd102 filter data locally 2024-03-20 10:36:21 +08:00
Hongbo Wu
a49366b4cc remove unnecessary fields from item events 2024-03-20 10:36:19 +08:00
Hongbo Wu
208a5895ef remove unnecessary fields from item events 2024-03-20 10:35:58 +08:00
Hongbo Wu
9af4235233 skip failed to parse rules and show the timestamp in the UI 2024-03-20 10:35:58 +08:00
Hongbo Wu
c05721c87e validate the filter before creating rules 2024-03-20 10:35:57 +08:00
Hongbo Wu
76f3321680 revalidate integration once updated 2024-03-20 10:32:32 +08:00
Hongbo Wu
bf59fbf26e add integration api 2024-03-20 10:32:31 +08:00
Hongbo Wu
7e04b25497
Merge pull request #3696 from omnivore-app/fix/thumbnail 
fix/thumbnail
 2024-03-20 10:26:34 +08:00
Hongbo Wu
315c625482 fix title not fetched correctly when using preview content in rss feeds 2024-03-19 18:07:27 +08:00
Jackson Harper
e47e169402
Merge branch 'main' into discover-no-ts-changes 2024-03-19 15:50:51 +08:00
Hongbo Wu
ad32aa82ce Export highlight notes as children of highlights block in notion 2024-03-19 11:17:42 +08:00
Jackson Harper
21d4efc2b5 Allow opting into the YouTube transcript feature 2024-03-18 13:56:37 +08:00
Jackson Harper
4e50c51598
Merge pull request #3661 from omnivore-app/feat/job-youtube-transcript-generator 
Pull metadata from YouTube for library items
 2024-03-17 09:40:28 +08:00
Jackson Harper
596ab5a7aa Feature flag the transcripts 2024-03-15 20:28:02 +08:00
Jackson Harper
b87073b1fb Add youtube-transcripts beta feature 2024-03-15 20:25:44 +08:00
Jackson Harper
d3aa671200 Prettier 2024-03-15 10:44:40 +08:00
Jackson Harper
fdc8055b79 Prettier 2024-03-15 10:43:16 +08:00
Hongbo Wu
17e66aa010 append new highlights only 2024-03-14 22:13:27 +08:00
Hongbo Wu
1085dcc824 read settings 2024-03-14 21:36:47 +08:00
Hongbo Wu
69e74d432d append highlights to the existing page 2024-03-14 19:50:26 +08:00
Jackson Harper
f2d23626a5 Improve error logging / fix linting 2024-03-14 17:39:13 +08:00
Jackson Harper
dc6c047aec Add GCS cache 2024-03-14 16:12:06 +08:00
Jackson Harper
629c044273 Queue the transcript processing as a separate job 
Handle YouTube in two steps, first get metadata then
get the transcript.
 2024-03-14 15:09:53 +08:00
Hongbo Wu
1ea09e20bd fix: subscription not updated correctly after rss feed refreshed 
* sort the shallow copy of the userIds array so the original array not mutated
 2024-03-14 12:56:42 +08:00
Jackson Harper
fef28d1c6a Linting fix 2024-03-14 12:48:56 +08:00
Jackson Harper
3ee6787e39 Improve transcript generation 2024-03-14 12:48:56 +08:00
Jackson Harper
39dfa920b5 More fixes to youtube processor 2024-03-14 12:48:56 +08:00
Jackson Harper
6bcdbfa8f0 Linting 2024-03-14 12:48:56 +08:00
Jackson Harper
7c3d15e31a Add some scrolling on youtube videos 2024-03-14 12:48:56 +08:00
Jackson Harper
2dbd16a61e Pull duration and description from YouTube metadata 2024-03-14 12:48:56 +08:00
Jackson Harper
f7225b298a Rebase 2024-03-14 12:48:56 +08:00
Hongbo Wu
81e489021e add popular reads to the continue reading section of new user library 2024-03-14 11:14:37 +08:00
Hongbo Wu
6310e5f83b update form 2024-03-13 15:55:34 +08:00
Hongbo Wu
ff157bd453
Merge pull request #3620 from omnivore-app/feature/notion-integration 
Feature: Notion Integration
 2024-03-12 22:37:51 +08:00
Jackson Harper
3266d6a9a3 Linting fixes 2024-03-12 21:34:20 +08:00
Hongbo Wu
8eaf754538 do not get content from db when exporting item to integrations 2024-03-12 15:37:19 +08:00
Jackson Harper
d2c22eaea2
Merge pull request #3649 from Podginator/discover-clean 
Discover clean
 2024-03-12 10:11:00 +08:00
Jackson Harper
e15ae29961 Use folders when handling push notifications 2024-03-08 21:11:16 +08:00
Jackson Harper
e093738dcc Send libraryItemId with push notifications 2024-03-08 17:24:02 +08:00
Jackson Harper
7d77291f11 Push configuration on iOS 2024-03-08 17:15:59 +08:00
Jackson Harper
6bc100942f Send libraryItemId and thumbnail with push notifications 2024-03-08 16:40:49 +08:00
Thomas Rogers
7d291f084b Merge all changes from main, update theming of Discover 2024-03-07 17:39:57 +01:00
Hongbo Wu
44178c1c1e Add author 2024-03-07 13:24:02 +08:00
Hongbo Wu
39778feb88 export highlights and labels 2024-03-06 20:23:42 +08:00
Hongbo Wu
d41459a5cd log unhandled error 2024-03-06 12:04:16 +08:00
Hongbo Wu
289f1cf5e2 better error message 2024-03-06 11:54:41 +08:00
Hongbo Wu
c346ba8703 do not throw error if failed to find tokens 2024-03-06 11:47:17 +08:00
Hongbo Wu
ec32945ba2 update block 2024-03-05 21:00:39 +08:00
Hongbo Wu
dfa5512512 get parentPageId from settings 2024-03-05 16:18:15 +08:00
Hongbo Wu
1edf801b02 add settings jsonb column to the integrations 2024-03-05 16:13:50 +08:00
Hongbo Wu
f1b1f2c4c1 send created event only 2024-03-05 15:39:09 +08:00
Hongbo Wu
741f3213d9 integrate with notion api client 2024-03-05 12:43:18 +08:00
Hongbo Wu
e20908a0c2 add export to notion method 2024-03-04 18:44:25 +08:00
Hongbo Wu
d36c937765 fix redirect_uri error 2024-03-04 17:49:59 +08:00
Hongbo Wu
94db3e0de7 Add authorization with notion 2024-03-04 17:30:05 +08:00
Hongbo Wu
4a70291898 add directionality to the SearchItem and Article Grahql type in the schema 2024-03-04 13:23:52 +08:00
Hongbo Wu
0e46dc2302 save dir in the database 2024-03-04 12:28:51 +08:00
Hongbo Wu
08e5caddc2 retry export-item job if fails 2024-03-01 15:03:01 +08:00
Hongbo Wu
e33cf465fe throw error if export-item job fails 2024-03-01 15:00:28 +08:00
Hongbo Wu
a337e0e39d reduce export-item job priority to 50 2024-03-01 14:52:49 +08:00
Jackson Harper
a74377556b
Merge pull request #3596 from omnivore-app/fix/web-layout-fixes 
Small layout fixes for mobile
 2024-02-29 18:20:33 +08:00
Hongbo Wu
2636524642 fix typo 2024-02-29 16:35:52 +08:00
Hongbo Wu
c3e8173e5e fix search results for archived items which state is not archived 2024-02-29 16:34:55 +08:00
Jackson Harper
34ee75c555 Dont re-run summarize job 2024-02-29 15:50:25 +08:00
Jackson Harper
531f23fc4c
Merge pull request #3591 from omnivore-app/feat/summaries 
Summaries
 2024-02-29 14:35:12 +08:00
Hongbo Wu
3af62d5d89 fix typo 2024-02-28 17:13:11 +08:00
Hongbo Wu
7567e3d0f6 * return only count if size is 0 because limit 0 is not allowed in typeorm 2024-02-28 16:57:15 +08:00
Jackson Harper
6861d271bd Clean up assertions in summary job 2024-02-28 15:49:24 +08:00
Jackson Harper
0382fb4913 Fix comparison of library item state 2024-02-28 15:34:36 +08:00
Jackson Harper
692a992e77 Remove some debug, clean up summaries 2024-02-28 15:18:23 +08:00
Jackson Harper
c20d8e6b4d Add summaries, display accont features 2024-02-28 15:14:32 +08:00
Jackson Harper
bbe6204b91 summaries 2024-02-28 15:12:09 +08:00
Hongbo Wu
0369a39f08 replace $set with $set_once 2024-02-26 18:14:10 +08:00
Hongbo Wu
41aa976625 flush remaining events when server exits 2024-02-26 17:28:28 +08:00
Hongbo Wu
d26a2f403d replace segment with posthog 2024-02-26 16:47:04 +08:00
Hongbo Wu
c13896aeb2
Merge pull request #3567 from omnivore-app/feature/feed-summary 
feature/feed summary
 2024-02-26 12:19:57 +08:00
Jackson Harper
33fa36710f
Merge pull request #3568 from omnivore-app/feat/rules-delete-item-rule 
Add a rule to delete items
 2024-02-26 12:13:40 +08:00
Hongbo Wu
7db7099822 fix tests 2024-02-23 19:43:43 +08:00
Hongbo Wu
6b0cefe190 add representation column to the highlight table and gql schema which is either CONTENT or FEED_CONTENT 2024-02-23 17:14:52 +08:00
Hongbo Wu
67f54a1372 add fetchContent resolver to the function list 2024-02-23 15:44:53 +08:00
Hongbo Wu
49d1ae5037 fix bug 2024-02-23 15:32:41 +08:00
Hongbo Wu
510cd18b17 use json_agg to aggregate enum in db 2024-02-23 14:54:37 +08:00
Jackson Harper
90c049506d Add a rule to delete items 2024-02-23 14:42:10 +08:00
Hongbo Wu
df0bda9cb2 fix a typo 2024-02-23 14:21:34 +08:00
Hongbo Wu
f4b6e470f2 replace fetch_content with fetch_content_type as a enum: ALWAYS, NEVER, WHEN_EMPTY 2024-02-23 14:19:22 +08:00
Hongbo Wu
08e4bed146 rename preview_content to feed_content in library_item table and gql schema 2024-02-23 12:53:39 +08:00
Hongbo Wu
61e9ab3a17
Merge pull request #3566 from omnivore-app/perf/optimize-slow-query 
perf/optimize slow query
 2024-02-23 12:53:05 +08:00
Hongbo Wu
8ce7550582 create an index for upload_file_id in library_item table 2024-02-23 11:33:48 +08:00
Hongbo Wu
7c7d594b91 use low priority queue if user saved 5 items in the last minute. previously it was created 2024-02-23 10:17:18 +08:00
Hongbo Wu
b947100278 increase batch size to 100 2024-02-22 20:23:37 +08:00
Hongbo Wu
f541b8f8ca export at most 1000 most recent items 2024-02-22 20:22:37 +08:00
Hongbo Wu
6af6f8ac6c delete log 2024-02-22 15:40:09 +08:00
Hongbo Wu
f47f4c1c47 fix tests 2024-02-22 15:23:53 +08:00
Hongbo Wu
fb576a58a0 initial export 2024-02-22 15:20:24 +08:00
Hongbo Wu
f94267ee1a enqueue export job 2024-02-22 13:38:51 +08:00
Hongbo Wu
34db7bb2d8 create a job for exporting item to integrations 2024-02-22 11:55:51 +08:00
Hongbo Wu
649ccc692b log unknown job when getting priority 2024-02-22 09:23:32 +08:00
Hongbo Wu
ce19218e15 get queue priority from a table 2024-02-21 22:01:04 +08:00
Hongbo Wu
927394e07c fix: save url operation is delayed 2024-02-21 17:54:08 +08:00
Hongbo Wu
2fff9a5720
Merge pull request #3380 from omnivore-app/fix/recreate-item 
fix/recreate item
 2024-02-21 15:49:28 +08:00
Hongbo Wu
8f39985f0f fix deleted labels and highlights are still searchable 2024-02-21 15:06:04 +08:00
Hongbo Wu
51c1d33a09 fix tests 2024-02-21 14:55:49 +08:00
Hongbo Wu
af24856603 resolve conflicts after rebasing 2024-02-21 14:29:21 +08:00
Hongbo Wu
b1c45599f6 resolve conflicts after rebasing 2024-02-21 14:29:15 +08:00
Hongbo Wu
eb895b49bf delete labels and highlights attached to the recreated item if the item was deleted before 2024-02-21 14:18:06 +08:00
Hongbo Wu
abfa15e56f revert soft deletes all the labels and highlights attached to the deleted item 2024-02-21 14:16:21 +08:00
Hongbo Wu
38ee6c1331 merge labels and highlights when saving a duplicate item 2024-02-21 14:16:20 +08:00
Hongbo Wu
242e6e0fa3 delete all the labels and highlights attached to the item when item was deleted 2024-02-21 13:52:39 +08:00
Hongbo Wu
cd311d6454 do not update db options in queue-processor 2024-02-20 19:09:33 +08:00
Hongbo Wu
59d4d78f78 connection idle timeout in 10 seconds 2024-02-20 18:33:21 +08:00
Hongbo Wu
942b2374b8 get postgres connection pool size from env 2024-02-20 18:08:55 +08:00
Hongbo Wu
d0500f8524
Merge pull request #3542 from omnivore-app/fix/duplicate-key-violation 
fix/duplicate key violation
 2024-02-19 12:46:57 +08:00
Hongbo Wu
26d05a2e6e delete the new item if it is different from the existing one 2024-02-19 11:47:09 +08:00
Hongbo Wu
afd4d2a397 fix: duplicate key value violates unique constraint "library_item_pkey" by getting the item by user_id and md5(url) for update before upserting the item 2024-02-19 11:35:43 +08:00
Jackson Harper
9c493d29db More graceful shutdown for the api server 2024-02-19 11:25:22 +08:00
Jackson Harper
e0ce598be5 Move apollo prom init into apollo.ts file 2024-02-12 11:29:08 +08:00
Jackson Harper
a52e8f7295 Add service label 2024-02-10 17:21:28 +08:00
Jackson Harper
5c29a6c185 Add http metrics 2024-02-10 16:05:35 +08:00
Jackson Harper
545d618c11
Merge pull request #3524 from omnivore-app/feat/library-navigation-updates 
Updated library layout
 2024-02-09 14:17:01 +08:00
Jackson Harper
11463ba6f7 Colour clean ups 2024-02-08 15:01:40 +08:00
Hongbo Wu
5d201521e1
Merge pull request #3518 from omnivore-app/fix/library-item 
fix: duplicate key value violates unique constraint "library_item_pkey"
 2024-02-08 13:18:22 +08:00
Hongbo Wu
03f89d890e fix tests 2024-02-08 10:42:37 +08:00
Hongbo Wu
c1ba9b8017 fix tests 2024-02-08 10:15:05 +08:00
Hongbo Wu
86f4553dd1 fix: duplicate key value violates unique constraint "library_item_pkey" 2024-02-07 20:41:34 +08:00
Hongbo Wu
485caec803 append job version to the job id 2024-02-07 18:42:10 +08:00
Hongbo Wu
ef6c6349de add remove_on_complete and remove_on_fail to the unique job 2024-02-07 18:03:16 +08:00
Hongbo Wu
8c23cff17b remove limit on the sync reading progress job 2024-02-07 11:09:30 +08:00
Hongbo Wu
d7a81483cf resolve conflicts 2024-02-06 17:02:00 +08:00
Hongbo Wu
1b60694579 lint 2024-02-06 16:54:44 +08:00
Hongbo Wu
efd8119e02 reduce sql error 2024-02-06 16:53:32 +08:00
Hongbo Wu
7e022e3294 reduce find user query 2024-02-06 16:53:32 +08:00
Hongbo Wu
a37bfab348 update create article api 2024-02-06 16:53:32 +08:00
Hongbo Wu
4eba7df84c always upsert library items 2024-02-06 16:53:32 +08:00
Hongbo Wu
468f835f9e
Merge pull request #3502 from omnivore-app/fix/bullmq-job-retry-backoff 
increase the seed valud of backoff delay to 2 seconds
 2024-02-06 15:09:40 +08:00
Hongbo Wu
95fabe1dbf keep completed jobs for up to 24 hours and failed jobs for 7 days 2024-02-06 14:21:00 +08:00
Hongbo Wu
a939180dc9 fix: dedupe update-db job by using library_item_id as job_id 2024-02-06 14:10:53 +08:00
Hongbo Wu
3477da59bb
Merge pull request #3503 from omnivore-app/fix/force-update-reading-progress 
do not update db immediately for force update of reading progress
 2024-02-06 13:12:23 +08:00
Hongbo Wu
d77fad804a fix tests 2024-02-05 18:29:58 +08:00
Hongbo Wu
37c75865d1 do not update db immediately for force update of reading progress 2024-02-05 16:56:48 +08:00
Hongbo Wu
53f63b90d7 increase the seed valud of backoff delay to 2 seconds 2024-02-05 16:26:05 +08:00
Jackson Harper
2d6bd0779f Dont fetch from rsshub 2024-02-03 19:08:25 +08:00
Jackson Harper
0a38cd6422 More attempts now that lock timeouts are lower 2024-02-03 19:08:13 +08:00
Jackson Harper
dde80091b4
Merge pull request #3494 from omnivore-app/fix/rss-more-content-fetch-blocks 
Add rss.arxiv and daringfireball to content fetch block lists
 2024-02-03 10:57:04 +08:00
Jackson Harper
7fffa8ce75 Add some logging 2024-02-03 10:04:40 +08:00
Jackson Harper
46affc0508 Add xkcd 2024-02-03 10:02:25 +08:00
Jackson Harper
36915ec1ec Add rss.arxiv and daringfireball to content fetch block lists 2024-02-03 10:01:18 +08:00
Jackson Harper
7937c73de1 Remove old warning thats no longer needed 2024-02-03 09:54:54 +08:00
Jackson Harper
9102fc9ae4
Merge pull request #3491 from omnivore-app/fix/dont-sync-readingtime-too-soon 
Only sync reading times if they are more than a minute old
 2024-02-03 09:09:52 +08:00
Jackson Harper
a745e72ce4 Allow setting postgres extra options via envvar 2024-02-03 08:57:41 +08:00
Jackson Harper
9cc046e3a2 Only sync reading times if they are more than a minute old 
this should hopefully reduce contention with other rules running
 2024-02-02 20:41:14 +08:00
Hongbo Wu
b68adc5983 fix tests 2024-02-02 17:05:09 +08:00
Hongbo Wu
df2dbb2916 fix: find library item by url not using correct index 2024-02-02 16:46:13 +08:00
Hongbo Wu
74783313da fix query 2024-02-02 16:18:41 +08:00
Hongbo Wu
1742ffda24 change priority to 5 2024-02-02 16:18:41 +08:00
Hongbo Wu
25e374f6ff fix: replace webhook endpoint with a bullmq job 2024-02-02 16:18:40 +08:00
Hongbo Wu
5e49131fc2
Merge pull request #3470 from omnivore-app/fix/async-bulk-action 
async bulk action on item count > 100 and create batch jobs
 2024-02-02 16:00:12 +08:00
Hongbo Wu
54a4022409 reduce size 2024-02-02 15:45:11 +08:00
Hongbo Wu
e980be7771 skip test 2024-02-02 14:14:41 +08:00
Hongbo Wu
26b7103f0b improve batch update 2024-02-02 13:13:18 +08:00
Jackson Harper
d339c5e8f1 Use the md5 index when looking up library items by original url 2024-02-02 13:08:00 +08:00
Hongbo Wu
2d1eebfad8 remove size 2024-02-02 12:21:08 +08:00
Hongbo Wu
9c28726a9b add parent config 2024-02-02 11:49:02 +08:00
Hongbo Wu
1bccf33320 add job_id and parent for tracking 2024-02-02 10:50:07 +08:00
Jackson Harper
ed7ac6c1b4 Fix typo in lifecycle endpoint path 2024-02-01 20:25:52 +08:00
Hongbo Wu
2a0e6f8fa4 fix tests 2024-02-01 17:44:32 +08:00
Hongbo Wu
702bd6c3c6 async bulk action on item count > 100 and create batch jobs 2024-02-01 17:44:30 +08:00
Jackson Harper
7f4f7c6a80
Merge pull request #3479 from omnivore-app/fix/queue-processor-lifecycle-hook 
Implement a lifecycle hook for queue processor so it can shut down gracefully
 2024-02-01 17:01:11 +08:00
Hongbo Wu
8ac696e87d
Merge pull request #3480 from omnivore-app/fix/mark-read-action 
fix: update reading progress in cache when mark as read action was triggered by a rule
 2024-02-01 16:56:21 +08:00
Hongbo Wu
252443f1e4 fix: update reading progress in cache when mark as read action was triggered in a rule 2024-02-01 16:45:08 +08:00
Jackson Harper
0e9e331e18 Implement a lifecycle hook for queue processor so it can shut down gracefully 2024-02-01 16:43:57 +08:00
Jackson Harper
3feef7ca66 Use the DB stored reading progress when calculating maxes 2024-02-01 16:23:21 +08:00
Jackson Harper
f03b7182b3
Merge pull request #3460 from omnivore-app/feat/reading-progress-write-behind 
Write behind cache for reading progress
 2024-02-01 14:43:45 +08:00
Hongbo Wu
abaf726044 revert the change of priority 2024-02-01 13:22:06 +08:00
Hongbo Wu
19fe60d27a change the update-db job priority to 5 and use exponential backoff strategy for retrying 2024-02-01 13:14:01 +08:00
Hongbo Wu
e13c418389 remove the inner join with highlights table in the sql of update-label job 2024-02-01 13:09:50 +08:00
Jackson Harper
02c28be403 reduce unction is no longer async 2024-02-01 11:24:13 +08:00
Jackson Harper
792f13edfc Remove async from non-async function 2024-02-01 10:49:48 +08:00
Jackson Harper
1b44cb860b Function for cron jobs dont include queue in the read position metrics since its not part of that queue 2024-02-01 10:33:51 +08:00
Jackson Harper
7d6645b87a Rebase 2024-02-01 10:02:38 +08:00
Jackson Harper
570b7a4b78 Use a set in redis and push items to the set 
This lets us grab all available items, apply them, and then
remove them from the set. Once completed the items that were
applied can be removed from the set, if any new items were
added during that time they wont be effected.
 2024-02-01 09:59:42 +08:00
Jackson Harper
0e37bfa03d We need to update readAt because that isnt resolved from the cache 2024-02-01 09:59:42 +08:00
Jackson Harper
c47a720fe4 Do not cache on write, so the most updated item is always available on read 
This is mostly an issue on tests where all the queries can be run
in a single process, but if we cache on write, it means the next
read will just have the cached value, instead of the calculated
maximum value.
 2024-02-01 09:59:42 +08:00
Jackson Harper
bdf01ddaab Clear cache on force, return updated read position values 2024-02-01 09:59:42 +08:00
Jackson Harper
a060aaf496 Make reading position parsing safer 2024-02-01 09:59:42 +08:00
Jackson Harper
9dc070243a Revert temp 2024-02-01 09:59:41 +08:00
Jackson Harper
d8ea4ff1c5 Add reading position messages to exported metrics for queue processor 2024-02-01 09:59:41 +08:00
Jackson Harper
7bb3718a8f Add job to sync read position data 2024-02-01 09:59:41 +08:00
Jackson Harper
a40f5bed55 Create a service for interacting with cached read positions 2024-02-01 09:59:41 +08:00
Jackson Harper
fbfa934479 Write behind reading progress as a list that can be reduced at commit or read time 2024-02-01 09:59:39 +08:00
Jackson Harper
c2327781e1 Remove empty ctor 2024-02-01 09:57:11 +08:00
Jackson Harper
86c80d991f Write behing cache for reading progress 2024-02-01 09:57:11 +08:00
Hongbo Wu
c3276fdaf1
Merge pull request #3466 from omnivore-app/fix/rewrite-trigger-functions-in-db 
create a job for db trigger
 2024-01-31 19:47:45 +08:00
Hongbo Wu
6d5ce451ac fix bulk action 2024-01-31 19:24:18 +08:00
Hongbo Wu
1b8111b907 wait until all jobs are done in test 2024-01-31 19:10:57 +08:00
Hongbo Wu
5e883cb2ba running worker in the test 2024-01-31 17:45:49 +08:00
Hongbo Wu
cf898299e4 some lint 2024-01-31 14:18:34 +08:00
Hongbo Wu
5646413b59 add a job for updating highlight annotations in library_item 2024-01-31 14:15:33 +08:00
Hongbo Wu
43855247b8 update bulk action to add labels 2024-01-31 13:56:56 +08:00
Hongbo Wu
5ed35bbeff create a job for update labels in library item 2024-01-31 13:32:37 +08:00
Hongbo Wu
ddddf8234e create a job for db trigger 2024-01-30 23:53:05 +08:00
Hongbo Wu
36d2c21466 remove thumbnail job on complete or failure 2024-01-30 18:24:48 +08:00
Hongbo Wu
2eedba1522 fix: assign the rule job with priority 1 and do not retry 2024-01-30 18:14:34 +08:00
Hongbo Wu
e724bbd465 fallback to worker redis client if fetched content not found in cache 2024-01-30 12:27:45 +08:00
Hongbo Wu
8d8309d3bb fix: empty client request id in save-page job data when saving pdf in rss item 2024-01-29 21:22:26 +08:00
Hongbo Wu
5faa24556e fix: string interpolation in logs 2024-01-29 19:11:26 +08:00
Hongbo Wu
809c572ca7 reduce the size of the log entry by truncating any string values to 500 characters 2024-01-29 14:32:45 +08:00
Hongbo Wu
d5d581fc54 remove redundant logs 2024-01-29 14:32:43 +08:00
Hongbo Wu
0181a2bfcd
Merge pull request #3446 from omnivore-app/fix/refresh-feed 
fix: add rss label and feed url to the pdf in rss feed item
 2024-01-29 14:21:08 +08:00
Hongbo Wu
58d384f4f6 fix pdf saving 2024-01-29 14:11:07 +08:00
Hongbo Wu
4f061546bf remove api call to upload file request 2024-01-29 12:55:19 +08:00
Hongbo Wu
a2bdddf8c9 fix tests 2024-01-29 12:30:31 +08:00
Hongbo Wu
738f85923c
Merge pull request #3442 from omnivore-app/fix/rule-handler 
move trigger rule into a job
 2024-01-29 12:20:21 +08:00
Hongbo Wu
ac5acf7f21 fix: add rss label and feed url to the pdf in rss feed item 2024-01-26 22:34:55 +08:00
Hongbo Wu
fd80724d79 fallback to guid if link of rss item is not available 2024-01-26 22:09:53 +08:00
Hongbo Wu
30fef30384 resolve conflicts 2024-01-26 20:32:56 +08:00
Hongbo Wu
e03895c8b7 enqueue trigger rule job once page is created or updated 2024-01-26 17:46:17 +08:00
Hongbo Wu
7556c6da5f move trigger rule into a job 2024-01-26 17:45:40 +08:00
Hongbo Wu
52419c39da resolve conflicts 2024-01-26 13:01:53 +08:00
Hongbo Wu
7e89235806 update content-fetch 2024-01-26 13:01:53 +08:00
Hongbo Wu
be0a2e8c52 use different redis server for mq and cache 2024-01-26 13:01:53 +08:00
Hongbo Wu
4c765378e3 do not retry if find_thumbnail job fails 2024-01-26 13:01:53 +08:00
Hongbo Wu
e54c691c09 create a job for finding thumbnail 2024-01-26 13:01:51 +08:00
Jackson Harper
f3c1b9d9a0
Merge pull request #3430 from omnivore-app/fix/queue-processor-custom-metrics 
Expose custom metrics for the queue processor
 2024-01-25 21:17:49 +08:00
Hongbo Wu
fd7c2ffb49 fix code blocks not formatted correctly in articles from wechat official accounts 2024-01-25 16:30:59 +08:00
Hongbo Wu
dae90aaffa fix code blocks not formatted correctly 2024-01-25 16:30:59 +08:00
Hongbo Wu
8834efabd4 throw error if save-page job failed 2024-01-25 16:30:59 +08:00
Hongbo Wu
5e239d2568 run readability in save-page instead of puppeteer 2024-01-25 16:30:59 +08:00
Jackson Harper
a639a2fa10 Add types to exported metrics 2024-01-25 12:37:02 +08:00
Jackson Harper
5f9be385b7 Remove unneeded escapes 2024-01-24 21:34:51 +08:00
Jackson Harper
c581372a93 Improve exported metrics naming convention 2024-01-24 21:13:13 +08:00
Hongbo Wu
7bb63b4ecd for campability with old clients, use refreshedAt to replace lastFetchedAt 2024-01-24 15:32:43 +08:00
Hongbo Wu
3f3a96297f update entity 2024-01-24 14:55:50 +08:00
Hongbo Wu
1563b44197 save the timestamp when feed fails 2024-01-24 14:54:39 +08:00
Hongbo Wu
9e55fe0d7b do not retry if user is deleted 2024-01-24 14:12:34 +08:00
Hongbo Wu
d357d2b0af fix tests 2024-01-24 13:41:49 +08:00
Hongbo Wu
86c4bbabee update logs 2024-01-24 13:30:57 +08:00
Hongbo Wu
9d378d3f46 get active user subscriptions only 2024-01-24 13:21:37 +08:00
Hongbo Wu
fd74ebffa2 validate feed item url and convert relative url to absolute url 2024-01-24 13:15:46 +08:00
Hongbo Wu
6f3fccfcd0 mark subscription as error if item was failed 2024-01-24 13:09:43 +08:00
Jackson Harper
d24db92e03 Dont force init this queue 2024-01-24 12:40:00 +08:00
Jackson Harper
c29f31d618 Add _count to prom metrics names 2024-01-24 12:39:04 +08:00
Jackson Harper
e4109a63f0 Expose custom metrics for the queue processor 2024-01-24 12:15:38 +08:00
Hongbo Wu
3cb7c38e7f update refreshed_at when feed is processed 2024-01-24 12:12:26 +08:00
Hongbo Wu
68a4fb298d add refreshedAt to the gql schema 2024-01-24 12:03:26 +08:00
Jackson Harper
58d41951c2 Dont not fetch content for medium.com rss feeds 
All these are blocked
 2024-01-24 02:11:39 +08:00
Jackson Harper
f8264bfc10 Hard code tweet url to avoid import 2024-01-23 23:45:28 +08:00
Hongbo Wu
c1d5ec722f do not encode url when getting file data 2024-01-23 22:06:27 +08:00
Hongbo Wu
d4e487e181 set the state to failed if content is not fetched 2024-01-23 21:50:10 +08:00
Hongbo Wu
47da6fb3dc
Merge pull request #3421 from omnivore-app/fix/title 
fix: do not throw error if title is undefined in cache
 2024-01-23 19:33:22 +08:00
Hongbo Wu
5495ad782e fix: do not throw error if title is undefined in cache 2024-01-23 19:19:46 +08:00
Hongbo Wu
8f5d4ae8a0 fix: catch rejected promise if it is in a try-catch block 2024-01-23 18:45:46 +08:00
Hongbo Wu
567ce172e6 make title optional for cache in save-page job 2024-01-23 16:55:17 +08:00
Hongbo Wu
9f7669b010 fix: do not use encoded url as cache key 2024-01-22 16:49:05 +08:00
Jackson Harper
21405e6328 Fix await 2024-01-22 15:48:15 +08:00
Jackson Harper
e407db914a Add new job for handling upload content 2024-01-22 15:36:46 +08:00
Jackson Harper
3e0f8c884e
Merge pull request #3404 from omnivore-app/debug/extra-logging 
Add logging to save_page service
 2024-01-22 13:59:19 +08:00
Jackson Harper
6dd4f49c2f Add logging to save_page service 2024-01-22 13:32:17 +08:00
Jackson Harper
4da93cbd2d
Merge pull request #3402 from omnivore-app/fix/unique-constraint 
fix: ignore error when saving existing library item in db
 2024-01-22 12:38:09 +08:00
Hongbo Wu
141036dc6c fix tests 2024-01-22 12:18:08 +08:00
Jackson Harper
db8284dc0b Wait until backend queue is ready before putting jobs into it 2024-01-22 11:31:46 +08:00
Jackson Harper
f7b17cb93f Only create queue object once in queue processor 2024-01-22 11:27:13 +08:00
Jackson Harper
0f78419e1a Remove some debug 2024-01-22 11:26:59 +08:00
Hongbo Wu
5964d827c8 fix tests 2024-01-22 11:04:11 +08:00
Jackson Harper
33c994ad5c Remove RSS feed task handler url as this handled with jobs now 2024-01-22 11:00:55 +08:00
Hongbo Wu
a807a4a18e fix: do not throw error when creating duplicate item in db 2024-01-22 10:45:47 +08:00
Jackson Harper
9d3a183c45 Dont map label input 2024-01-21 15:44:41 +08:00
Jackson Harper
6cedffc249 Only require original content in savePageJob 2024-01-21 15:30:06 +08:00
Jackson Harper
202d19c9d3 Fix api status mapping 2024-01-21 14:11:53 +08:00
Jackson Harper
5393743c9d Fix imports of authorized function 2024-01-21 13:33:05 +08:00
Jackson Harper
56a8ef23b6 Remove authorized function from imported helpers 
This creates some issues with init order when using the
queue processor.
 2024-01-21 13:22:17 +08:00
Jackson Harper
eefceeb1ef Move checks to have the PDF check 2024-01-21 13:19:50 +08:00
Jackson Harper
9abd0a59da Clean up imports 2024-01-21 13:16:35 +08:00
Jackson Harper
db3d7dca36 Add missing file 2024-01-21 07:43:50 +08:00
Jackson Harper
a4e707075f Refactor savePage a bit to use it as a service from the jobs 2024-01-20 21:07:02 +08:00
Jackson Harper
2f405ebed0 Use savePage service instead of API call 2024-01-20 14:07:22 +08:00
Jackson Harper
3a18327230 Remove unused 2024-01-20 13:43:17 +08:00
Jackson Harper
7786221fbc Fix update_subscription awaits 2024-01-20 13:38:25 +08:00
Jackson Harper
89537c13de Simplify function input 2024-01-20 13:32:11 +08:00
Jackson Harper
1898e32607 Use service instead of API to update subscription 2024-01-20 13:27:54 +08:00
Jackson Harper
e5dec228f3 Remove some debug 2024-01-20 12:46:22 +08:00
Jackson Harper
4eeb012b78 Pass lits of fetchContentTasks into rss handlers 
This prevents the list from growing on each run
 2024-01-20 11:36:14 +08:00
Jackson Harper
2b536c6086 Add more debug 2024-01-20 10:48:57 +08:00
Jackson Harper
52050728ba Add more debugging logs 2024-01-20 10:29:07 +08:00
Jackson Harper
f4af4593c2 Fix debug line 2024-01-20 10:04:54 +08:00
Jackson Harper
4c1a182c7f Add a refresh context to make debugging refresh runs easier 2024-01-20 09:50:21 +08:00
Jackson Harper
2961f69f02 Add a refresh context to make debugging rss jobs easier 2024-01-20 09:47:46 +08:00
Jackson Harper
a9eeda9369 Remove non-useful logging line 2024-01-20 09:23:45 +08:00
Jackson Harper
e9f22bba43 Some debug 2024-01-20 09:09:25 +08:00
Hongbo Wu
780cda6e4b use worker redis client when connect to the queue in queue-processor 2024-01-19 20:44:54 +08:00
Hongbo Wu
338733829f
Merge pull request #3394 from omnivore-app/fix/cache-content-fetched 
fix/cache content fetched
 2024-01-19 18:16:05 +08:00
Hongbo Wu
356ad38678 reduce logs 2024-01-19 18:01:09 +08:00
Hongbo Wu
4340286b89 add retry on failure in jobs 2024-01-19 17:38:25 +08:00
Jackson Harper
defb79d8c8 Use let instead of var 2024-01-19 17:12:34 +08:00
Hongbo Wu
51d5166321 get cached content from redis when saving pages 2024-01-19 17:06:37 +08:00
Jackson Harper
8ba1242ffb Back off refresh-all-feeds if queue is very deep 2024-01-19 16:59:41 +08:00
Jackson Harper
d2491d3eb7 Reduce refresh-all-feeds priority 2024-01-19 16:59:15 +08:00
Jackson Harper
e06ff24cea Fix logging 2024-01-19 16:44:20 +08:00
Jackson Harper
02818d94a2 Adjust priorities 2024-01-19 16:37:57 +08:00
Hongbo Wu
35805e6e8f use user provided url for feed added 2024-01-19 16:28:37 +08:00
Jackson Harper
84375b7c36 Add validation before refreshing feed URLs 2024-01-19 16:25:05 +08:00
Jackson Harper
2dbff6a6ab Add some more debug output 2024-01-19 16:12:44 +08:00
Jackson Harper
0e590e53a6 Improve RSS priorities for refreshing feeds 2024-01-19 16:12:22 +08:00
Hongbo Wu
d135a7fe6a
Merge pull request #3391 from omnivore-app/fix/content-fetch-dockerfile 
fix: build content-fetch in dockerfile
 2024-01-19 12:45:19 +08:00
Hongbo Wu
3d42973c5f fix api endpint 2024-01-19 12:36:15 +08:00
Jackson Harper
f03766d05b Fix construction of service endpoint 
NEed to fix how we construct all these
 2024-01-19 12:29:50 +08:00
Hongbo Wu
c2a084803a validate feed url before adding 2024-01-18 21:56:19 +08:00
Hongbo Wu
d2c17f7539 graceful shutdown redis 2024-01-18 21:52:33 +08:00
Hongbo Wu
3e291d1019 fix tests 2024-01-18 21:13:32 +08:00
Hongbo Wu
d4f49dceb6 remove job id 2024-01-18 18:48:46 +08:00
Hongbo Wu
a6f0e2f2d9 add save page job processor 2024-01-18 18:48:46 +08:00
Hongbo Wu
8a3b50f07f do not create thumbnail task if the item already has a thumbnail 2024-01-18 18:48:46 +08:00
Hongbo Wu
d9feb740cb convert content-fetch to typescript 2024-01-18 18:48:46 +08:00
Jackson Harper
3f707c06f2 Fix checks for env vars 2024-01-18 16:59:53 +08:00
Jackson Harper
3b0338967d Fix endpoint for internal refresh feed API call 
This should eventually be replaced.
 2024-01-18 16:48:45 +08:00
Jackson Harper
6193c78288 Fix typo 2024-01-18 15:37:38 +08:00
Jackson Harper
7754139207 Remove debug 2024-01-18 15:33:51 +08:00
Jackson Harper
fb28d7b3d1 Enqueue new feeds at front 2024-01-18 14:34:07 +08:00
Jackson Harper
ee142d3d7c Handle SIGINT in server better by moving DB call to after redis shutdown 2024-01-18 12:00:13 +08:00
Jackson Harper
bb55137bb4 Disconnect one client while quitting the other 2024-01-18 11:39:41 +08:00
Jackson Harper
fa685d4a29 Tear down redis with disconnect instead of quit 2024-01-18 11:28:09 +08:00
Hongbo Wu
d558fe2b17 fix tests 2024-01-18 11:24:26 +08:00
Jackson Harper
5a7056b7ec Dont fail whole job if we cant update a subscription group 2024-01-18 10:18:45 +08:00
Jackson Harper
65347a1996 Use quit when shutting down redis 2024-01-17 17:30:16 +08:00
Jackson Harper
fd81f65e53 Remove server.close call 2024-01-17 16:37:51 +08:00
Jackson Harper
9e6b749604 Update error message 2024-01-17 16:21:42 +08:00
Jackson Harper
e1e2f86219 Remove duplicated error 2024-01-17 16:20:42 +08:00
Jackson Harper
dc4126d52c REvert some unneeded env changes 2024-01-17 16:15:00 +08:00
Jackson Harper
361d57ac1d Remove secrets loading as this is handled by k8s now. Fix redis init 2024-01-17 13:33:20 +08:00
Jackson Harper
d1438fe800 Remove old code, clean up redis logging 2024-01-17 11:47:31 +08:00
Jackson Harper
4f23271f29 Remove some debug 2024-01-17 11:45:05 +08:00
Jackson Harper
7a925f3fdc Clean up redis code, remove connection close in job 2024-01-17 11:14:59 +08:00
Jackson Harper
d92ee4bcfe Clean up redis instantiation 2024-01-17 11:03:52 +08:00
Jackson Harper
68b7b93586 More logging 2024-01-17 10:42:38 +08:00
Jackson Harper
ec765c0f5c Some more logging 2024-01-17 10:21:19 +08:00
Jackson Harper
0936942378 Remove connect call 2024-01-16 23:32:53 +08:00
Jackson Harper
50b48a4225 Separate worker and client code for redis connections 2024-01-16 23:21:48 +08:00
Jackson Harper
c5e4e2216b Simplify connection options 2024-01-16 23:02:46 +08:00
Jackson Harper
49734efec7 Add logging 2024-01-16 22:43:06 +08:00
Jackson Harper
7c8304ac15 Fix client imports 2024-01-16 22:08:30 +08:00
Jackson Harper
7ee3866f34 Pull in the newer connection and retry strategy code 2024-01-16 22:04:19 +08:00
Jackson Harper
90a0455c36 Update job to use ioredis 2024-01-16 22:00:25 +08:00
Jackson Harper
fcf2d07c99 Update redis data source 2024-01-16 21:58:34 +08:00
Jackson Harper
8580398d6f Clean up post rebase 2024-01-16 21:35:46 +08:00
Jackson Harper
ba3e5b053d Cleanup after rebase 2024-01-16 21:31:44 +08:00
Jackson Harper
f184c8baff Fix path 2024-01-16 21:30:17 +08:00
Jackson Harper
84195f10c2 Fix endpoint 2024-01-16 21:30:17 +08:00
Jackson Harper
4df9b70c95 Fix for detecting environment 2024-01-16 21:30:17 +08:00
Jackson Harper
e050864a91 Add more debugging 2024-01-16 21:30:17 +08:00
Jackson Harper
9633e7df79 Update envar name 2024-01-16 21:30:17 +08:00
Jackson Harper
62fdb866a8 Add some extra logging 2024-01-16 21:30:17 +08:00
Jackson Harper
26d832bb92 Trim URLs from feeds 2024-01-16 21:30:17 +08:00
Jackson Harper
652f4b6180 Use createHttpTask for save operation 2024-01-16 21:30:17 +08:00
Jackson Harper
26657c5611 Use cloud task function from backend code instead of from rss-handler 2024-01-16 21:30:17 +08:00
Jackson Harper
a8e8fe23da Remove azure config from backend 2024-01-16 21:30:17 +08:00
Jackson Harper
69eb22ae3c Remove some env vars that are not used 2024-01-16 21:30:17 +08:00
Jackson Harper
c7f605d00a If redis fails assume we need to refetch the item 2024-01-16 21:30:17 +08:00
Jackson Harper
00b72f1c3a Add lwn.net to block list 2024-01-16 21:30:17 +08:00
Jackson Harper
d31d309bc0 MOre on env parsing for k8s 2024-01-16 21:30:17 +08:00
Jackson Harper
5986bd0a68 Change around redis initialization so we can pull in secrets from secret manager async on startup 2024-01-16 21:30:15 +08:00
Jackson Harper
0a82e5721e Graceful k8s shutdown 2024-01-16 21:27:22 +08:00
Jackson Harper
7bb6602454 Fix linting 2024-01-16 21:27:22 +08:00
Jackson Harper
7aa76b70fe Add missing jobs 2024-01-16 21:27:22 +08:00
Jackson Harper
819052f51f Add queue-processor endpoint to the api 2024-01-16 21:27:22 +08:00
Jackson Harper
b4914f5346 Fix creation of redis clients 2024-01-16 21:27:20 +08:00
Jackson Harper
447bba23ba Fix init 2024-01-16 21:26:30 +08:00
Jackson Harper
20df5c6212 Parse redis cert on load 2024-01-16 21:26:28 +08:00
Jackson Harper
45fcde8ebc Update redis config 2024-01-16 21:25:34 +08:00
Jackson Harper
b4ce07efd3 Better handling of cases when redis isnt configured 2024-01-16 21:25:34 +08:00
Jackson Harper
d29c26d93b Some redis setup 2024-01-16 21:25:34 +08:00
Jackson Harper
fc25d6d615 Enqueue rss-refresh-feed jobs in bull 2024-01-16 21:25:31 +08:00
Hongbo Wu
150a456c35 replace redis client library with ioredis 2024-01-16 15:42:50 +08:00
Hongbo Wu
1e05f7dade log error 2024-01-16 12:17:07 +08:00
Hongbo Wu
5141c2a042 do not retry saving following items if item is saved 2024-01-16 12:15:21 +08:00
Hongbo Wu
816019e6a3 extend ttl of the saved item cache to 26 hours 2024-01-16 11:17:21 +08:00
Hongbo Wu
06067c1d66 fix tests 2024-01-15 11:26:40 +08:00
Hongbo Wu
bd612bd310 replace username too 2024-01-15 11:23:01 +08:00
Hongbo Wu
deddb51c42 fix: change user email address after user deleted account 2024-01-12 11:45:28 +08:00
Hongbo Wu
d9d800cfc4 add index for sorting by saved_at 2024-01-10 17:09:15 +08:00
Tom Rogers
451939832f Probably a good idea to add back the schema changes... 2024-01-09 11:11:28 +00:00
Thomas Rogers
c0373646cb Fix Linting... 2024-01-09 11:22:32 +01:00
Hongbo Wu
1ec5b7affe fix labels not saved when merging highlights 2024-01-05 23:22:09 +08:00
Hongbo Wu
ccd4f2d3f0 catch redis client error and continue the process 2024-01-05 15:20:42 +08:00
Hongbo Wu
32a58e49f9 make redis optional for backend 2024-01-04 21:55:41 +08:00
Hongbo Wu
bcd05cc7fc set the url in redis when an item is inserted into db 2024-01-04 18:10:14 +08:00
Hongbo Wu
08c4624fce save the url in redis for 8 hours so rss-feeder will not try to re-save it 2024-01-04 18:03:34 +08:00
Hongbo Wu
d1db49f9bb
Merge pull request #3300 from omnivore-app/fix/batch-prune 
fix/batch prune
 2024-01-04 15:19:50 +08:00
Hongbo Wu
366c97ead7
Merge pull request #3314 from omnivore-app/fix/update-event 
skip sending item update event for reading progress update
 2024-01-04 15:17:08 +08:00
Hongbo Wu
d31eb7a88b skip sending item update event for reading progress update 2024-01-04 15:06:31 +08:00
Hongbo Wu
554c4cee12 add test cases 2024-01-03 18:26:07 +08:00
Hongbo Wu
d31c520a90 add empty trash api to prune soft deleted items in trash 2024-01-03 18:26:06 +08:00
Hongbo Wu
5eb19cda71 prune users in batch 2024-01-03 18:25:16 +08:00
Jackson Harper
a1ea20b964 Update highlights query for new users 2024-01-03 18:21:13 +08:00
Jackson Harper
0b098e640f
Merge pull request #3309 from omnivore-app/fix/includes-query 
if the query contains includes:, we want to search in all folders
 2024-01-03 17:48:41 +08:00
Hongbo Wu
060690609d
Merge pull request #3264 from omnivore-app/feature/prune-following-items 
add endpoint to prune items from folder older than some days
 2024-01-03 17:46:21 +08:00
Hongbo Wu
6e73b10771 if the query contains includes:, we want to search in all folders 2024-01-03 17:38:12 +08:00
Hongbo Wu
b348fae677 fix multi-select bulk action api 2023-12-29 18:42:07 +08:00
Hongbo Wu
42502f9ff1
Merge pull request #3280 from omnivore-app/fix/slow-db-query 
execute update query in batch to avoid slow bulk action query
 2023-12-29 16:48:18 +08:00
Hongbo Wu
7153014a87 fix duplicate labels 2023-12-29 16:36:55 +08:00
Hongbo Wu
6e32c1247d insert labels on name conflict do nothing 2023-12-29 16:11:11 +08:00
Hongbo Wu
943db3332e
Merge pull request #3276 from omnivore-app/fix/in-library 
by default we want to search in the inbox if no in: is specified
 2023-12-29 14:51:00 +08:00
Hongbo Wu
cbb07318e5 get the unchanged result set of sub query by checking updated_at 2023-12-29 12:53:49 +08:00
Hongbo Wu
b5d84098d0 increase batch size to 100 2023-12-29 09:53:23 +08:00
Hongbo Wu
de027ee93f use cursor and fetch instead of limit and offset because the subquery result set could change during update 2023-12-29 09:53:23 +08:00
Hongbo Wu
55518139fb generate raw sql because postgres does not support prepared statements in DO blocks 2023-12-29 09:53:23 +08:00
Hongbo Wu
1079a19cd9 execute update query in batch to avoid slow bulk action query 2023-12-29 09:53:23 +08:00
Hongbo Wu
881626d35a fix type of highlight is null from update since api 2023-12-28 16:55:23 +08:00
Hongbo Wu
2b886d7e68 return highlights in update since api response 2023-12-28 14:03:07 +08:00
Hongbo Wu
47aa2eaae5 by default we want to search in the inbox if no in: is specified 2023-12-22 18:01:38 +08:00
Hongbo Wu
c21ba9e11f fix tests 2023-12-22 17:12:24 +08:00
Hongbo Wu
ccec94dae0 insert labels only if name unique 2023-12-22 15:14:07 +08:00
Hongbo Wu
c939735678 remove invalid-url error from subscribing feed 2023-12-22 09:51:27 +08:00
Hongbo Wu
f6e04daf60 return invalid url if feed url is different from user provided url 2023-12-22 09:05:43 +08:00
Hongbo Wu
696de8ea50 check both user provided url and parsed feed url when looking for existing subscription 2023-12-22 08:08:15 +08:00
Hongbo Wu
db28bbd0f9 fix invalid unknown item type 2023-12-20 11:51:19 +08:00
Hongbo Wu
0a4758105a feat: update newsletter email api 2023-12-20 11:29:10 +08:00
Hongbo Wu
e4b6cba2ff
Merge pull request #3261 from omnivore-app/feature/add-state-for-non-fetch-content 
feature/add state for non fetch content
 2023-12-19 18:30:13 +08:00
Hongbo Wu
92aa11b98e add fetchContent api 2023-12-19 18:05:02 +08:00
Hongbo Wu
07b96c46d5 fix tests 2023-12-19 17:45:12 +08:00
Hongbo Wu
b5f82d448e return folder when newsletter email api is called 2023-12-19 14:52:30 +08:00
Hongbo Wu
8e77c047bd add endpoint to prune items from folder older than some days 2023-12-19 14:25:55 +08:00
Hongbo Wu
22778af285 run readability on the preview content 2023-12-19 10:41:20 +08:00
Hongbo Wu
b1aca117f4 if fetch_content is false, set the state to content_not_fetched and use preview content 2023-12-19 10:41:20 +08:00
Hongbo Wu
6f1ef075c6 add content_not_fetched state for items saved without content fetched 2023-12-19 10:41:20 +08:00
Hongbo Wu
15dc05ca1a fix sending rss feed to inbox folder 2023-12-18 19:52:44 +08:00
Hongbo Wu
55ed5a0a80 fix typo 2023-12-18 16:20:40 +08:00
Hongbo Wu
97735d0633 make existing newsletters go to inbox folder 2023-12-18 16:08:36 +08:00
Hongbo Wu
7a2cf6c6fe fix sort by word count not working 2023-12-18 10:40:10 +08:00
Hongbo Wu
58f16ec62c
Merge pull request #3247 from omnivore-app/feature/feeds 
feat: fetch content and folder settings for newsletters and subscriptions
 2023-12-15 16:46:02 +08:00
Hongbo Wu
d579a8f907 not throw error if sort keyword not found 2023-12-15 13:31:22 +08:00
Hongbo Wu
f915c18dfa allow plural and singular form for has/no filter 2023-12-15 13:18:25 +08:00
Hongbo Wu
42749c5a32 throw error if sort column not exists 2023-12-15 11:47:00 +08:00
Hongbo Wu
b86ee139e0 move to folder api will create a task to fetch content if content not exists 2023-12-14 19:20:03 +08:00