prehistoric-systems/omnivore
1
0
Fork 0
mirror of https://github.com/omnivore-app/omnivore.git synced 2026-03-11 08:54:26 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
12057 commits 201 branches 31 tags 432 MiB
Commit graph

2536 commits

Author SHA1 Message Date
Hongbo Wu
32be8126df reduce number of db calls for search library items 2024-04-30 12:48:19 +08:00
Hongbo Wu
14e91d338d do not create user profile and topics 2024-04-30 12:48:19 +08:00
Hongbo Wu
d0d088aae3 fix: remove redundant user query 2024-04-30 12:48:19 +08:00
Hongbo Wu
cfc1d12835
Merge pull request #3866 from omnivore-app/feature/data-loader 
feat: use dataloader for labels and highlights
 2024-04-29 15:20:53 +08:00
Jackson Harper
4632d08aa0
Merge pull request #3863 from omnivore-app/fix/fetch_feed 
increase rss parser timeout to 20 seconds
 2024-04-28 22:45:55 -07:00
Jackson Harper
c70c36ec7d
Merge pull request #3826 from kimlimjustin/feat/increase-daily-upload-limit 
feat: increase daily upload limit
 2024-04-28 22:45:10 -07:00
Hongbo Wu
8549c16ceb fix import 2024-04-28 21:55:09 +08:00
Hongbo Wu
ef2ff36f0e fetch api key once 2024-04-28 21:51:30 +08:00
Hongbo Wu
a78d438e21 fix: findOne with relations will execute query twice 2024-04-28 15:58:55 +08:00
Hongbo Wu
7ef6973a47 load labels when getting item 2024-04-28 14:25:11 +08:00
Hongbo Wu
89d6564d74 increase log metadata size to 10000 characters 2024-04-26 15:18:34 +08:00
Hongbo Wu
342db3928c batching upload_files 2024-04-26 14:44:17 +08:00
Hongbo Wu
01246207cb fix logger 2024-04-26 12:06:20 +08:00
Hongbo Wu
66d7649620 fix tests 2024-04-26 11:54:55 +08:00
Hongbo Wu
2231f03948 reduce joins 2024-04-26 11:36:16 +08:00
Hongbo Wu
364f8f102f use enum 2024-04-26 11:24:07 +08:00
Hongbo Wu
6493b8e2f6 batching recommendations 2024-04-26 11:17:52 +08:00
Hongbo Wu
fbe816d2d2 convert content if requested 2024-04-26 11:07:41 +08:00
Hongbo Wu
c7ecbd53f8 put dataloaders in apollo context 2024-04-25 23:26:55 +08:00
Hongbo Wu
2904c321e9 batching highlights 2024-04-25 23:03:08 +08:00
Hongbo Wu
1445dee933 batching labels 2024-04-25 22:20:56 +08:00
Hongbo Wu
83526e1b56 increase rss parser timeout to 20 seconds 2024-04-25 10:55:27 +08:00
Jackson Harper
636fa17712 Default to 5 items 2024-04-24 16:39:46 -07:00
Jackson Harper
ae8790b545
Merge pull request #3848 from omnivore-app/feat/ios-digest-view 
iOS Explain and Digest views
 2024-04-24 16:13:50 -07:00
Jackson Harper
3dcc514be7
Merge pull request #3835 from omnivore-app/fix/guess-user-email 
return UNKNOWN as error code if email exists when user signs up
 2024-04-24 16:06:46 -07:00
Jackson Harper
5a6d57e3da Remove uneeded 2024-04-24 15:58:39 -07:00
Hongbo Wu
0c0a95a79c fix newsletter dir not saved correctly 2024-04-24 21:10:13 +08:00
Hongbo Wu
e788c71eff rename variables 2024-04-24 16:49:46 +08:00
Hongbo Wu
e680202b7e better error message 2024-04-24 16:17:05 +08:00
Hongbo Wu
824b256d20 fix memory leak from axios error 2024-04-24 15:55:54 +08:00
Jackson Harper
e349f6c675
Merge pull request #3853 from omnivore-app/fix/android-obsidian-cors 
Add localhost to allow android Obsidian to query the API
 2024-04-23 08:01:37 -07:00
Jackson Harper
dfdb04af92 Add localhost to allow android Obsidian to query the API 2024-04-23 07:57:41 -07:00
Hongbo Wu
3fb7193fb5 reduce getTextNodesBetween timeout to 60 seconds 2024-04-23 21:28:02 +08:00
Hongbo Wu
002e455bbd fix tests 2024-04-23 17:54:52 +08:00
Hongbo Wu
0faafa0786 fix rebase conflicts 2024-04-23 16:04:59 +08:00
Hongbo Wu
2f7c18b363 add more logs 2024-04-23 15:50:15 +08:00
Hongbo Wu
1b6f4976d3 remove timeout 2024-04-23 15:50:15 +08:00
Hongbo Wu
de258ae0bf timeout if job takes more than 10 minutes 2024-04-23 15:50:15 +08:00
Hongbo Wu
a77ade0890 fix: export all items job stuck 2024-04-23 15:50:14 +08:00
Hongbo Wu
201a5817ef
Merge pull request #3843 from omnivore-app/fix/newsletter-language 
fix language and dir not saved for newsletters
 2024-04-23 14:45:36 +08:00
Jackson Harper
2449c872bf Add Obsidian's capacitor origin to our CORs config 2024-04-22 19:15:20 -07:00
Jackson Harper
cd5940b840 Feature flag explain 2024-04-22 19:04:10 -07:00
Jackson Harper
f0be6a41a9 Clean up, remove old stubbed stuff 2024-04-22 17:40:50 -07:00
Jackson Harper
7f0a95b454 Add iOS digest views 2024-04-22 15:52:28 -07:00
Hongbo Wu
dcf7c570c4 fix tests 2024-04-22 10:57:48 +08:00
Hongbo Wu
9f65588ba0 fix language and dir not saved for newsletters 2024-04-21 10:41:29 +08:00
Hongbo Wu
8501509564 include content in updateSince api response by default for offline use 2024-04-19 14:26:17 +08:00
Hongbo Wu
fa4b3bf462 fix cron schedule bug with bull 2024-04-19 14:25:18 +08:00
Hongbo Wu
ac775ae8f2 Drain httpServer when apollo server stops 2024-04-19 14:25:18 +08:00
Hongbo Wu
273ff9d4be increase max ai-digest beta users to 10 2024-04-19 14:25:18 +08:00
Hongbo Wu
07475e69a4 change cron to schedule at 10:30 2024-04-19 14:25:18 +08:00
Hongbo Wu
a69d1240c8 add urlsToAudio back 2024-04-19 14:25:18 +08:00
Hongbo Wu
efa930a31a send push notifications when digest is ready 2024-04-19 14:25:18 +08:00
Hongbo Wu
7871154160 mark failed task 2024-04-19 14:25:18 +08:00
Hongbo Wu
b93573460c filter out summary if the length is longer than the original content 2024-04-19 14:25:18 +08:00
Hongbo Wu
099310ed95 reuse existing candidates if there is no one 2024-04-19 14:25:18 +08:00
Hongbo Wu
0d8b17f6aa remove urlsToAudio from digest 2024-04-19 14:25:18 +08:00
Hongbo Wu
1da9b51dec do not reuse candidates after refreshing 2024-04-19 14:25:18 +08:00
Hongbo Wu
09ce46e7ea include readable content in search api response by default 2024-04-19 11:30:09 +08:00
Hongbo Wu
22850bb6ab return UNKNOWN if email exists when user signs up 2024-04-18 14:04:38 +08:00
Jackson Harper
9ed94b4571
Merge pull request #3796 from omnivore-app/feature/captcha 
feature: add captcha to email-login and reset-password pages
 2024-04-18 09:48:08 +08:00
Hongbo Wu
26579d9cbb use ### for title in the digest content 2024-04-17 18:26:48 +08:00
Hongbo Wu
3e51aa7c03 generate description 2024-04-17 16:49:19 +08:00
Hongbo Wu
0eb6460bec fix tests 2024-04-17 16:35:29 +08:00
Hongbo Wu
aeda4f5d5d add libraryItemIds to the payload 2024-04-17 16:20:00 +08:00
Hongbo Wu
57cb897839 generate content 2024-04-17 15:45:05 +08:00
Hongbo Wu
af184ded22 dedupe repeated jobs 2024-04-17 15:36:16 +08:00
Hongbo Wu
6222d567a4 add byline to the digest 2024-04-17 14:32:07 +08:00
Hongbo Wu
448e432807 runs at 1pm pst 2024-04-17 13:03:48 +08:00
Hongbo Wu
b2a55a78d3 check if feature is granted 2024-04-17 12:53:29 +08:00
Hongbo Wu
259b778aba add schedule and other parameters 2024-04-17 12:16:15 +08:00
Hongbo Wu
6a4faf6f3c add description to the digest 2024-04-17 10:04:52 +08:00
Hongbo Wu
47fd4d9663 save id, url, thumbnail and createdAt in the digest 2024-04-17 09:58:11 +08:00
Hongbo Wu
6bfb4842e2 fix idx 2024-04-16 21:27:20 +08:00
Hongbo Wu
4bc1246a47 fixing html 2024-04-16 21:00:53 +08:00
Hongbo Wu
71477b1712 implement create digest job 2024-04-16 21:00:51 +08:00
Jackson Harper
daaf493d4b Update stubbed data 2024-04-16 21:00:27 +08:00
Jackson Harper
bcddf556d5 Write speech files to redis 2024-04-16 21:00:27 +08:00
Jackson Harper
fbaa9ce663 Digest job 2024-04-16 21:00:27 +08:00
Jackson Harper
d2ea93a7dd Stub out some changes to the digest response 2024-04-16 21:00:27 +08:00
Hongbo Wu
e30fa13a2e add env to the posthog event 2024-04-16 21:00:27 +08:00
Hongbo Wu
03e497b489 use a separate queue for digest job 2024-04-16 21:00:26 +08:00
Hongbo Wu
cdb0b7525e cleanup 2024-04-16 21:00:10 +08:00
Hongbo Wu
bc955fbeeb remove stub job 2024-04-16 21:00:10 +08:00
Hongbo Wu
111dcd1562 send ranking models and summary models to posthog and log comment 2024-04-16 21:00:10 +08:00
Hongbo Wu
4342cdeb3e expire digest in one day 2024-04-16 21:00:10 +08:00
Hongbo Wu
521698646d update digest mock data 2024-04-16 21:00:10 +08:00
Hongbo Wu
a47a5a7d9c mock saving digest in redis 2024-04-16 21:00:10 +08:00
Hongbo Wu
e37bcd7408 send feedback api 2024-04-16 21:00:10 +08:00
Hongbo Wu
1edb3419f8 get digest api 2024-04-16 21:00:10 +08:00
Hongbo Wu
12aa7d4447 create digest job api 2024-04-16 21:00:10 +08:00
Hongbo Wu
b4911639c7 stub create digest job 2024-04-16 21:00:09 +08:00
Jackson Harper
be57f81bf1 Temp re-add features attribute to User 2024-04-16 13:59:04 +08:00
Hongbo Wu
8e31ac4fea fix tests 2024-04-16 12:24:20 +08:00
Hongbo Wu
d79cc1c39f wrap select in a auth transaction 2024-04-16 11:26:29 +08:00
Hongbo Wu
a9779a51e8 fix: LABEL_ALREADY_EXIST bug occured during label creation 
* comparing both user_id and label name when looking for an existing matching label
 2024-04-16 09:56:57 +08:00
Justin Maximillian Kimlim
56b87672fd
feat: increase daily upload limit 2024-04-16 00:49:26 +07:00
Hongbo Wu
9ccd1e90fe save received email as non-article by default 2024-04-15 18:54:15 +08:00
Hongbo Wu
4346746e0a save empty url for email attachment in upload_file table as it is required 2024-04-15 17:36:42 +08:00
Hongbo Wu
cbddc83829
Merge pull request #3784 from omnivore-app/feature/reply-to-email 
add reply to email API which replies Okay to the sender
 2024-04-15 15:24:32 +08:00
Hongbo Wu
e23d3d5933 update readable content instead 2024-04-12 13:26:11 +08:00
Hongbo Wu
f95642be2b remove redundant field 2024-04-11 18:47:19 +08:00
Hongbo Wu
448c78a87e add dotenv 2024-04-11 18:25:24 +08:00
Hongbo Wu
54e7c52b5e always update the state of the pdf item to succeeded/failed, even if we fail to parse the content 2024-04-11 18:07:53 +08:00
Hongbo Wu
f66d3fa36c migrate inbound-email tasks from pubsub to bullmq 2024-04-11 11:08:22 +08:00
Hongbo Wu
cd6594a2fd add logs 2024-04-11 11:08:22 +08:00
Hongbo Wu
67657773cf track the usage of the api 2024-04-11 11:08:22 +08:00
Hongbo Wu
cd6f3e6bbe enforce usage limits for the API 2024-04-11 11:08:22 +08:00
Hongbo Wu
5f239d2dcb update graceful shutdown 2024-04-11 11:08:22 +08:00
Hongbo Wu
f2f18eb12b reply only allowed text 2024-04-11 11:08:22 +08:00
Hongbo Wu
e7fc1316dd enqueue reply email job 2024-04-11 11:08:22 +08:00
Hongbo Wu
74f5b922dd update received email reply 2024-04-11 11:08:22 +08:00
Hongbo Wu
dce0b15f4f send to the reply-to address if it exists or the from address 2024-04-11 11:08:22 +08:00
Hongbo Wu
764b05fff4 save reply-to in recent emails 2024-04-11 11:08:22 +08:00
Hongbo Wu
5cc5c559ac add reply to email API which replies Okay to the sender 2024-04-11 11:08:22 +08:00
Hongbo Wu
a59c850c5f fix telegram channel 2024-04-09 19:44:16 +08:00
Hongbo Wu
e7fc2c4567 reset failed_at to null if a feed is refreshed successfully after failure 2024-04-09 18:39:17 +08:00
Hongbo Wu
77c11fb80f
Merge pull request #3767 from omnivore-app/fix/verification 
Enqueue emails
 2024-04-08 12:30:37 +08:00
Hongbo Wu
ec309f5b0d verify captcha in email-login and forgot-password api 2024-04-08 12:27:29 +08:00
Hongbo Wu
7abb396032 remove stub 2024-04-05 22:53:03 +08:00
Hongbo Wu
af140cc8b4 remove user name from email template 2024-04-05 16:59:23 +08:00
Hongbo Wu
8af2bb076c rename send mail functions 2024-04-05 16:59:20 +08:00
Hongbo Wu
c37b31dcb6 fix tests 2024-04-05 16:57:02 +08:00
Hongbo Wu
10dcb922f2 Enqueue confirmation email 2024-04-05 16:57:02 +08:00
Hongbo Wu
207241934d show label_attached instead of label_created in rules 2024-04-05 16:51:11 +08:00
Hongbo Wu
25fb9fd698 deduplicate labels when there are case-insensitive identical names 2024-04-05 16:27:17 +08:00
Hongbo Wu
6f03b214b5 allow delete internal labels 2024-04-05 15:58:39 +08:00
Hongbo Wu
9f4f1d07ca do not create label if label name case-insensitively matches existing label name 2024-04-05 15:50:01 +08:00
Hongbo Wu
906b6c1ea5 fix web 2024-04-04 18:53:10 +08:00
Hongbo Wu
58708a507e use old event format for the compatibility with the old webhooks 2024-04-04 18:22:18 +08:00
Hongbo Wu
45a5382d7c do not send bad_request error if db failure 2024-04-04 12:19:18 +08:00
Hongbo Wu
5e7a82562b resolve conflicts after rebasing 2024-04-04 12:19:17 +08:00
Hongbo Wu
b2cd43136f hide webhooks page 2024-04-04 12:19:16 +08:00
Hongbo Wu
0fab3c7904 Add highlight_updated event 2024-04-04 12:17:28 +08:00
Hongbo Wu
88a7e8d85b fix tests 2024-04-04 12:17:15 +08:00
Hongbo Wu
81f21141fe optimize ts-node-dev 2024-04-04 12:17:15 +08:00
Hongbo Wu
841a0026ff do not send page_update event for thumbnailer 2024-04-04 12:17:15 +08:00
Hongbo Wu
1b24a24f7c remove slug from highlight event 2024-04-04 12:17:15 +08:00
Hongbo Wu
d67aea6d8a deduplicate notion page by omnivore id 2024-04-04 12:17:13 +08:00
Hongbo Wu
0428da3b71 fix notion syncing error 2024-04-04 12:16:45 +08:00
Hongbo Wu
6954fffd24 fix readwise syncing error 2024-04-04 12:16:26 +08:00
Hongbo Wu
ff78a4b266 fix readwise syncing error 2024-04-04 12:16:12 +08:00
Hongbo Wu
f35ac036eb add UI for export in rules 2024-04-04 12:15:28 +08:00
Hongbo Wu
ad2e8ee002 trigger export in rules 2024-04-04 12:14:38 +08:00
Hongbo Wu
58511049a2 rename LABEL_CREATED to LABEL_ATTACHED in webhook data 2024-04-04 12:02:29 +08:00
Hongbo Wu
e2b3235046 delete unneccessary properties from webhook data 2024-04-04 12:02:29 +08:00
Hongbo Wu
88d6455222 default filter is in:all 2024-04-04 12:02:29 +08:00
Hongbo Wu
c551d9fe6b allow send data to webhook with rules 2024-04-04 12:02:29 +08:00
Hongbo Wu
07cee98ff9 allow label and highlight events to trigger rules 2024-04-04 12:02:29 +08:00
Hongbo Wu
e28dd73f9c
Merge pull request #3780 from omnivore-app/fix/notion 
sync items and highlights to user created database in notion
 2024-04-04 11:58:59 +08:00
Hongbo Wu
10cca7f05d find the title property and update its name to Title 2024-04-04 10:43:39 +08:00
Jackson Harper
c45a9d365b Also use mailjet for password changes 2024-04-03 22:25:31 +08:00
Hongbo Wu
491611b05f sync items and highlights to user created database in notion 2024-04-03 22:25:19 +08:00
Jackson Harper
3c1abb964b Also use mailjet for reset password emails 2024-04-03 22:07:28 +08:00
Hongbo Wu
f794cf707a
Merge pull request #3769 from omnivore-app/fix/notion 
export last 100 items in one job
 2024-04-03 10:22:56 +08:00
Hongbo Wu
08fb92d599 fix: site filter in rules matching both site name and url 2024-04-02 18:32:36 +08:00
Hongbo Wu
cc07d2bfbf remove comment 2024-04-02 17:56:46 +08:00
Hongbo Wu
d83dc17651 fix tests 2024-04-02 17:49:29 +08:00
Hongbo Wu
e19d6c86d8 export last 100 items in one job 2024-04-02 15:51:02 +08:00
Jackson Harper
aea354c3f2 No longer make captcha optional on web signup 2024-04-01 19:07:21 +08:00
Jackson Harper
3a59317217 Force captcha on sign up 2024-04-01 17:06:45 +08:00
Hongbo Wu
6ca092fbce use different prefix for different rate limiter 2024-04-01 13:44:10 +08:00
Hongbo Wu
a97542d4ba use redis for hourly rate limiter 2024-04-01 13:01:02 +08:00
Jackson Harper
788460c144
Merge pull request #3761 from omnivore-app/feat/signup-captcha 
Add recaptcha on the web sign up flow
 2024-04-01 12:24:39 +08:00
Jackson Harper
23f5377b62 Clean up isRecaptchaResponse function 2024-04-01 12:03:36 +08:00
Jackson Harper
0fe471da82 Dont require score on challenge recaptcha 2024-04-01 11:32:10 +08:00
Jackson Harper
822fcb1e74 Add some logging 2024-04-01 11:17:15 +08:00
Jackson Harper
9a159084c8 Update the isValidSignUp to check for URLs 2024-04-01 11:17:15 +08:00
Jackson Harper
442475d684 Make this optional for self hosters 2024-04-01 11:17:15 +08:00
Jackson Harper
7518e164a5 Utils for recaptcha 2024-04-01 11:15:19 +08:00
Hongbo Wu
5dd308d78e store the rate limit data in redis if redis is connected 2024-03-31 20:13:27 +08:00
Jackson Harper
2f6ef5e447 Return false if can not send with Mailjet 2024-03-31 19:22:56 +08:00
Jackson Harper
cfe4315fe2 Add an email fallback to Mailjet 2024-03-31 19:03:26 +08:00
Jackson Harper
200354d181 Add url present check on email sign up 2024-03-31 09:55:53 +08:00
Jackson Harper
ede99dcd33 Better matching for youtube video URLs 2024-03-30 10:42:01 +08:00
Jackson Harper
b69ba150c5 Remove debug 2024-03-29 12:28:26 +08:00
Jackson Harper
3a352a9705 Dont need to chunk this anymore 2024-03-29 12:05:32 +08:00
Jackson Harper
08ed061756 Change model for youtube transcripts 2024-03-29 11:31:27 +08:00
Hongbo Wu
f677f1167f get feature list on web 2024-03-28 15:27:37 +08:00
Hongbo Wu
cc2fe04568 add featureList to the User type which contains a list of feature type 2024-03-28 14:42:08 +08:00
Hongbo Wu
1afec1ddf3 check if feature is granted when process youtube video 2024-03-28 14:41:30 +08:00
Hongbo Wu
45417efd0c bump youtube users to 500 2024-03-28 12:15:36 +08:00
Hongbo Wu
8d8c7e8a26 fix: allow max 1000 users to use notion feature 2024-03-28 12:13:28 +08:00
Jackson Harper
b5bea17ffe Export the age of the oldest job in the queue 2024-03-28 10:25:44 +08:00
Jackson Harper
82168fe439
Merge pull request #3739 from omnivore-app/fix/youtube-queue 
Only check for summaries if the updated entity is a page item
 2024-03-28 09:40:44 +08:00
Hongbo Wu
d4ed994914
Merge pull request #3740 from omnivore-app/fix/logger 
replace logger.warn with warning
 2024-03-27 16:00:08 +08:00
Jackson Harper
38f959685f Change warn to warning 2024-03-27 15:41:15 +08:00
Hongbo Wu
92abfed2df replace logger.warn with warning 2024-03-27 15:41:13 +08:00
Jackson Harper
d23a7d56c0 Only check for summaries if the updated entity is a page item 2024-03-27 13:54:54 +08:00
Jackson Harper
5b736c1c2e
Merge pull request #3738 from omnivore-app/fix/api-youtube-logging 
Add more logging to youtube jobs
 2024-03-27 11:44:37 +08:00
Hongbo Wu
7c281359e8 fix recommendations not filtered correctly by rules 2024-03-27 11:29:47 +08:00
Jackson Harper
28222dcd56 Add more logging to youtube jobs 2024-03-27 11:25:46 +08:00
Hongbo Wu
c022cf0a22 label match wildcard 2024-03-27 10:54:24 +08:00
Hongbo Wu
3544dd1709 remove feed content from job data 2024-03-27 10:38:51 +08:00
Hongbo Wu
af78b6f4f0 append newsletter labels to avoid labels attached by rules being replaced 2024-03-27 10:37:04 +08:00
Hongbo Wu
41b90cbfd1 fix some undefined error 2024-03-26 17:53:48 +08:00
Hongbo Wu
d02d9444a9 remove content from update event too 2024-03-26 17:46:38 +08:00
Hongbo Wu
c3dba7e9da fix typo 2024-03-26 17:44:05 +08:00
Hongbo Wu
4c458c7118 fallback to search query if implicit fields are in the filter 2024-03-26 17:34:57 +08:00
Hongbo Wu
37459f64c7 remove content from job data 2024-03-26 16:49:26 +08:00
Hongbo Wu
209b08e2ff add notion to the features list 2024-03-26 12:32:01 +08:00
Hongbo Wu
f7c25fe05a do not create a new connection for the discover api 2024-03-23 10:03:06 +08:00
Hongbo Wu
282ed75a5f
Merge pull request #3698 from omnivore-app/fix/notion 
Fix: notion
 2024-03-21 14:08:16 +08:00
Hongbo Wu
445d2846b5 export last 100 items 2024-03-21 14:06:56 +08:00
Hongbo Wu
390d12eea1 add export to integration and get task api 2024-03-21 11:14:23 +08:00
Hongbo Wu
b1e3811e4f
Merge pull request #3707 from omnivore-app/fix/rule 
send labels and highlight annotations in page_create event
 2024-03-21 10:52:15 +08:00
Hongbo Wu
6385b40e38 revert batch size change 2024-03-20 23:07:46 +08:00
Hongbo Wu
3d986a9036 send labels and highlight annotations in page_create event 2024-03-20 22:33:24 +08:00
Jackson Harper
59cc954618 Add 12hrs option, fix linting issue 2024-03-20 15:36:53 +08:00
Jackson Harper
82ef57a36a Add last24hrs query for time based search queries to get recently saved/changed items 2024-03-20 15:15:18 +08:00
Jackson Harper
014ccb1eae Set published date from youtube data 
its not clear how consistent the formatting is on these dates,
the one i tested seems like it was a UTC formatted dd M, yyyy string
 2024-03-20 13:10:41 +08:00
Hongbo Wu
689eeaeacd disable sending pubsub events for rss feed temporarily to resolve conflicts 2024-03-20 10:46:13 +08:00
Hongbo Wu
2e374557bf send labels in the page_created events 2024-03-20 10:36:36 +08:00
Hongbo Wu
abf9706539 resolve conflicts after rebasing 2024-03-20 10:36:21 +08:00
Hongbo Wu
7f224cc1e1 running text search on all text fields 2024-03-20 10:36:21 +08:00
Hongbo Wu
edeafc9a71 fix bug 2024-03-20 10:36:21 +08:00
Hongbo Wu
de28a9ad5f fix bug 2024-03-20 10:36:21 +08:00
Hongbo Wu
f1eff130af filter data locally with liqe 2024-03-20 10:36:21 +08:00
Hongbo Wu
e87e6dd102 filter data locally 2024-03-20 10:36:21 +08:00
Hongbo Wu
a49366b4cc remove unnecessary fields from item events 2024-03-20 10:36:19 +08:00
Hongbo Wu
208a5895ef remove unnecessary fields from item events 2024-03-20 10:35:58 +08:00
Hongbo Wu
9af4235233 skip failed to parse rules and show the timestamp in the UI 2024-03-20 10:35:58 +08:00
Hongbo Wu
c05721c87e validate the filter before creating rules 2024-03-20 10:35:57 +08:00
Hongbo Wu
76f3321680 revalidate integration once updated 2024-03-20 10:32:32 +08:00
Hongbo Wu
bf59fbf26e add integration api 2024-03-20 10:32:31 +08:00
Hongbo Wu
7e04b25497
Merge pull request #3696 from omnivore-app/fix/thumbnail 
fix/thumbnail
 2024-03-20 10:26:34 +08:00
Hongbo Wu
315c625482 fix title not fetched correctly when using preview content in rss feeds 2024-03-19 18:07:27 +08:00
Jackson Harper
e47e169402
Merge branch 'main' into discover-no-ts-changes 2024-03-19 15:50:51 +08:00
Hongbo Wu
ad32aa82ce Export highlight notes as children of highlights block in notion 2024-03-19 11:17:42 +08:00
Jackson Harper
21d4efc2b5 Allow opting into the YouTube transcript feature 2024-03-18 13:56:37 +08:00
Jackson Harper
4e50c51598
Merge pull request #3661 from omnivore-app/feat/job-youtube-transcript-generator 
Pull metadata from YouTube for library items
 2024-03-17 09:40:28 +08:00
Jackson Harper
596ab5a7aa Feature flag the transcripts 2024-03-15 20:28:02 +08:00
Jackson Harper
b87073b1fb Add youtube-transcripts beta feature 2024-03-15 20:25:44 +08:00
Jackson Harper
d3aa671200 Prettier 2024-03-15 10:44:40 +08:00
Jackson Harper
fdc8055b79 Prettier 2024-03-15 10:43:16 +08:00
Hongbo Wu
17e66aa010 append new highlights only 2024-03-14 22:13:27 +08:00
Hongbo Wu
1085dcc824 read settings 2024-03-14 21:36:47 +08:00
Hongbo Wu
69e74d432d append highlights to the existing page 2024-03-14 19:50:26 +08:00
Jackson Harper
f2d23626a5 Improve error logging / fix linting 2024-03-14 17:39:13 +08:00
Jackson Harper
dc6c047aec Add GCS cache 2024-03-14 16:12:06 +08:00
Jackson Harper
629c044273 Queue the transcript processing as a separate job 
Handle YouTube in two steps, first get metadata then
get the transcript.
 2024-03-14 15:09:53 +08:00
Hongbo Wu
1ea09e20bd fix: subscription not updated correctly after rss feed refreshed 
* sort the shallow copy of the userIds array so the original array not mutated
 2024-03-14 12:56:42 +08:00
Jackson Harper
fef28d1c6a Linting fix 2024-03-14 12:48:56 +08:00
Jackson Harper
3ee6787e39 Improve transcript generation 2024-03-14 12:48:56 +08:00
Jackson Harper
39dfa920b5 More fixes to youtube processor 2024-03-14 12:48:56 +08:00
Jackson Harper
6bcdbfa8f0 Linting 2024-03-14 12:48:56 +08:00
Jackson Harper
7c3d15e31a Add some scrolling on youtube videos 2024-03-14 12:48:56 +08:00
Jackson Harper
2dbd16a61e Pull duration and description from YouTube metadata 2024-03-14 12:48:56 +08:00
Jackson Harper
f7225b298a Rebase 2024-03-14 12:48:56 +08:00
Hongbo Wu
81e489021e add popular reads to the continue reading section of new user library 2024-03-14 11:14:37 +08:00
Hongbo Wu
6310e5f83b update form 2024-03-13 15:55:34 +08:00
Hongbo Wu
ff157bd453
Merge pull request #3620 from omnivore-app/feature/notion-integration 
Feature: Notion Integration
 2024-03-12 22:37:51 +08:00
Jackson Harper
3266d6a9a3 Linting fixes 2024-03-12 21:34:20 +08:00
Hongbo Wu
8eaf754538 do not get content from db when exporting item to integrations 2024-03-12 15:37:19 +08:00
Jackson Harper
d2c22eaea2
Merge pull request #3649 from Podginator/discover-clean 
Discover clean
 2024-03-12 10:11:00 +08:00
Jackson Harper
e15ae29961 Use folders when handling push notifications 2024-03-08 21:11:16 +08:00
Jackson Harper
e093738dcc Send libraryItemId with push notifications 2024-03-08 17:24:02 +08:00
Jackson Harper
7d77291f11 Push configuration on iOS 2024-03-08 17:15:59 +08:00
Jackson Harper
6bc100942f Send libraryItemId and thumbnail with push notifications 2024-03-08 16:40:49 +08:00
Thomas Rogers
7d291f084b Merge all changes from main, update theming of Discover 2024-03-07 17:39:57 +01:00
Hongbo Wu
44178c1c1e Add author 2024-03-07 13:24:02 +08:00
Hongbo Wu
39778feb88 export highlights and labels 2024-03-06 20:23:42 +08:00
Hongbo Wu
d41459a5cd log unhandled error 2024-03-06 12:04:16 +08:00
Hongbo Wu
289f1cf5e2 better error message 2024-03-06 11:54:41 +08:00
Hongbo Wu
c346ba8703 do not throw error if failed to find tokens 2024-03-06 11:47:17 +08:00
Hongbo Wu
ec32945ba2 update block 2024-03-05 21:00:39 +08:00
Hongbo Wu
dfa5512512 get parentPageId from settings 2024-03-05 16:18:15 +08:00
Hongbo Wu
1edf801b02 add settings jsonb column to the integrations 2024-03-05 16:13:50 +08:00
Hongbo Wu
f1b1f2c4c1 send created event only 2024-03-05 15:39:09 +08:00
Hongbo Wu
741f3213d9 integrate with notion api client 2024-03-05 12:43:18 +08:00
Hongbo Wu
e20908a0c2 add export to notion method 2024-03-04 18:44:25 +08:00
Hongbo Wu
d36c937765 fix redirect_uri error 2024-03-04 17:49:59 +08:00
Hongbo Wu
94db3e0de7 Add authorization with notion 2024-03-04 17:30:05 +08:00
Hongbo Wu
4a70291898 add directionality to the SearchItem and Article Grahql type in the schema 2024-03-04 13:23:52 +08:00
Hongbo Wu
0e46dc2302 save dir in the database 2024-03-04 12:28:51 +08:00
Hongbo Wu
08e5caddc2 retry export-item job if fails 2024-03-01 15:03:01 +08:00
Hongbo Wu
e33cf465fe throw error if export-item job fails 2024-03-01 15:00:28 +08:00
Hongbo Wu
a337e0e39d reduce export-item job priority to 50 2024-03-01 14:52:49 +08:00
Jackson Harper
a74377556b
Merge pull request #3596 from omnivore-app/fix/web-layout-fixes 
Small layout fixes for mobile
 2024-02-29 18:20:33 +08:00
Hongbo Wu
2636524642 fix typo 2024-02-29 16:35:52 +08:00
Hongbo Wu
c3e8173e5e fix search results for archived items which state is not archived 2024-02-29 16:34:55 +08:00
Jackson Harper
34ee75c555 Dont re-run summarize job 2024-02-29 15:50:25 +08:00
Jackson Harper
531f23fc4c
Merge pull request #3591 from omnivore-app/feat/summaries 
Summaries
 2024-02-29 14:35:12 +08:00
Hongbo Wu
3af62d5d89 fix typo 2024-02-28 17:13:11 +08:00
Hongbo Wu
7567e3d0f6 * return only count if size is 0 because limit 0 is not allowed in typeorm 2024-02-28 16:57:15 +08:00
Jackson Harper
6861d271bd Clean up assertions in summary job 2024-02-28 15:49:24 +08:00
Jackson Harper
0382fb4913 Fix comparison of library item state 2024-02-28 15:34:36 +08:00
Jackson Harper
692a992e77 Remove some debug, clean up summaries 2024-02-28 15:18:23 +08:00
Jackson Harper
c20d8e6b4d Add summaries, display accont features 2024-02-28 15:14:32 +08:00
Jackson Harper
bbe6204b91 summaries 2024-02-28 15:12:09 +08:00
Hongbo Wu
0369a39f08 replace $set with $set_once 2024-02-26 18:14:10 +08:00
Hongbo Wu
41aa976625 flush remaining events when server exits 2024-02-26 17:28:28 +08:00
Hongbo Wu
d26a2f403d replace segment with posthog 2024-02-26 16:47:04 +08:00
Hongbo Wu
c13896aeb2
Merge pull request #3567 from omnivore-app/feature/feed-summary 
feature/feed summary
 2024-02-26 12:19:57 +08:00
Jackson Harper
33fa36710f
Merge pull request #3568 from omnivore-app/feat/rules-delete-item-rule 
Add a rule to delete items
 2024-02-26 12:13:40 +08:00
Hongbo Wu
7db7099822 fix tests 2024-02-23 19:43:43 +08:00
Hongbo Wu
6b0cefe190 add representation column to the highlight table and gql schema which is either CONTENT or FEED_CONTENT 2024-02-23 17:14:52 +08:00
Hongbo Wu
67f54a1372 add fetchContent resolver to the function list 2024-02-23 15:44:53 +08:00
Hongbo Wu
49d1ae5037 fix bug 2024-02-23 15:32:41 +08:00
Hongbo Wu
510cd18b17 use json_agg to aggregate enum in db 2024-02-23 14:54:37 +08:00
Jackson Harper
90c049506d Add a rule to delete items 2024-02-23 14:42:10 +08:00
Hongbo Wu
df0bda9cb2 fix a typo 2024-02-23 14:21:34 +08:00
Hongbo Wu
f4b6e470f2 replace fetch_content with fetch_content_type as a enum: ALWAYS, NEVER, WHEN_EMPTY 2024-02-23 14:19:22 +08:00
Hongbo Wu
08e4bed146 rename preview_content to feed_content in library_item table and gql schema 2024-02-23 12:53:39 +08:00
Hongbo Wu
61e9ab3a17
Merge pull request #3566 from omnivore-app/perf/optimize-slow-query 
perf/optimize slow query
 2024-02-23 12:53:05 +08:00
Hongbo Wu
8ce7550582 create an index for upload_file_id in library_item table 2024-02-23 11:33:48 +08:00
Hongbo Wu
7c7d594b91 use low priority queue if user saved 5 items in the last minute. previously it was created 2024-02-23 10:17:18 +08:00
Hongbo Wu
b947100278 increase batch size to 100 2024-02-22 20:23:37 +08:00
Hongbo Wu
f541b8f8ca export at most 1000 most recent items 2024-02-22 20:22:37 +08:00
Hongbo Wu
6af6f8ac6c delete log 2024-02-22 15:40:09 +08:00
Hongbo Wu
f47f4c1c47 fix tests 2024-02-22 15:23:53 +08:00
Hongbo Wu
fb576a58a0 initial export 2024-02-22 15:20:24 +08:00
Hongbo Wu
f94267ee1a enqueue export job 2024-02-22 13:38:51 +08:00
Hongbo Wu
34db7bb2d8 create a job for exporting item to integrations 2024-02-22 11:55:51 +08:00
Hongbo Wu
649ccc692b log unknown job when getting priority 2024-02-22 09:23:32 +08:00
Hongbo Wu
ce19218e15 get queue priority from a table 2024-02-21 22:01:04 +08:00
Hongbo Wu
927394e07c fix: save url operation is delayed 2024-02-21 17:54:08 +08:00
Hongbo Wu
2fff9a5720
Merge pull request #3380 from omnivore-app/fix/recreate-item 
fix/recreate item
 2024-02-21 15:49:28 +08:00
Hongbo Wu
8f39985f0f fix deleted labels and highlights are still searchable 2024-02-21 15:06:04 +08:00
Hongbo Wu
51c1d33a09 fix tests 2024-02-21 14:55:49 +08:00
Hongbo Wu
af24856603 resolve conflicts after rebasing 2024-02-21 14:29:21 +08:00
Hongbo Wu
b1c45599f6 resolve conflicts after rebasing 2024-02-21 14:29:15 +08:00
Hongbo Wu
eb895b49bf delete labels and highlights attached to the recreated item if the item was deleted before 2024-02-21 14:18:06 +08:00
Hongbo Wu
abfa15e56f revert soft deletes all the labels and highlights attached to the deleted item 2024-02-21 14:16:21 +08:00
Hongbo Wu
38ee6c1331 merge labels and highlights when saving a duplicate item 2024-02-21 14:16:20 +08:00
Hongbo Wu
242e6e0fa3 delete all the labels and highlights attached to the item when item was deleted 2024-02-21 13:52:39 +08:00
Hongbo Wu
cd311d6454 do not update db options in queue-processor 2024-02-20 19:09:33 +08:00
Hongbo Wu
59d4d78f78 connection idle timeout in 10 seconds 2024-02-20 18:33:21 +08:00
Hongbo Wu
942b2374b8 get postgres connection pool size from env 2024-02-20 18:08:55 +08:00
Hongbo Wu
d0500f8524
Merge pull request #3542 from omnivore-app/fix/duplicate-key-violation 
fix/duplicate key violation
 2024-02-19 12:46:57 +08:00
Hongbo Wu
26d05a2e6e delete the new item if it is different from the existing one 2024-02-19 11:47:09 +08:00
Hongbo Wu
afd4d2a397 fix: duplicate key value violates unique constraint "library_item_pkey" by getting the item by user_id and md5(url) for update before upserting the item 2024-02-19 11:35:43 +08:00
Jackson Harper
9c493d29db More graceful shutdown for the api server 2024-02-19 11:25:22 +08:00
Jackson Harper
e0ce598be5 Move apollo prom init into apollo.ts file 2024-02-12 11:29:08 +08:00
Jackson Harper
a52e8f7295 Add service label 2024-02-10 17:21:28 +08:00
Jackson Harper
5c29a6c185 Add http metrics 2024-02-10 16:05:35 +08:00
Jackson Harper
545d618c11
Merge pull request #3524 from omnivore-app/feat/library-navigation-updates 
Updated library layout
 2024-02-09 14:17:01 +08:00
Jackson Harper
11463ba6f7 Colour clean ups 2024-02-08 15:01:40 +08:00
Hongbo Wu
5d201521e1
Merge pull request #3518 from omnivore-app/fix/library-item 
fix: duplicate key value violates unique constraint "library_item_pkey"
 2024-02-08 13:18:22 +08:00
Hongbo Wu
03f89d890e fix tests 2024-02-08 10:42:37 +08:00
Hongbo Wu
c1ba9b8017 fix tests 2024-02-08 10:15:05 +08:00
Hongbo Wu
86f4553dd1 fix: duplicate key value violates unique constraint "library_item_pkey" 2024-02-07 20:41:34 +08:00
Hongbo Wu
485caec803 append job version to the job id 2024-02-07 18:42:10 +08:00
Hongbo Wu
ef6c6349de add remove_on_complete and remove_on_fail to the unique job 2024-02-07 18:03:16 +08:00
Hongbo Wu
8c23cff17b remove limit on the sync reading progress job 2024-02-07 11:09:30 +08:00
Hongbo Wu
d7a81483cf resolve conflicts 2024-02-06 17:02:00 +08:00
Hongbo Wu
1b60694579 lint 2024-02-06 16:54:44 +08:00
Hongbo Wu
efd8119e02 reduce sql error 2024-02-06 16:53:32 +08:00
Hongbo Wu
7e022e3294 reduce find user query 2024-02-06 16:53:32 +08:00
Hongbo Wu
a37bfab348 update create article api 2024-02-06 16:53:32 +08:00
Hongbo Wu
4eba7df84c always upsert library items 2024-02-06 16:53:32 +08:00
Hongbo Wu
468f835f9e
Merge pull request #3502 from omnivore-app/fix/bullmq-job-retry-backoff 
increase the seed valud of backoff delay to 2 seconds
 2024-02-06 15:09:40 +08:00
Hongbo Wu
95fabe1dbf keep completed jobs for up to 24 hours and failed jobs for 7 days 2024-02-06 14:21:00 +08:00
Hongbo Wu
a939180dc9 fix: dedupe update-db job by using library_item_id as job_id 2024-02-06 14:10:53 +08:00
Hongbo Wu
3477da59bb
Merge pull request #3503 from omnivore-app/fix/force-update-reading-progress 
do not update db immediately for force update of reading progress
 2024-02-06 13:12:23 +08:00
Hongbo Wu
d77fad804a fix tests 2024-02-05 18:29:58 +08:00
Hongbo Wu
37c75865d1 do not update db immediately for force update of reading progress 2024-02-05 16:56:48 +08:00
Hongbo Wu
53f63b90d7 increase the seed valud of backoff delay to 2 seconds 2024-02-05 16:26:05 +08:00
Jackson Harper
2d6bd0779f Dont fetch from rsshub 2024-02-03 19:08:25 +08:00
Jackson Harper
0a38cd6422 More attempts now that lock timeouts are lower 2024-02-03 19:08:13 +08:00
Jackson Harper
dde80091b4
Merge pull request #3494 from omnivore-app/fix/rss-more-content-fetch-blocks 
Add rss.arxiv and daringfireball to content fetch block lists
 2024-02-03 10:57:04 +08:00
Jackson Harper
7fffa8ce75 Add some logging 2024-02-03 10:04:40 +08:00
Jackson Harper
46affc0508 Add xkcd 2024-02-03 10:02:25 +08:00
Jackson Harper
36915ec1ec Add rss.arxiv and daringfireball to content fetch block lists 2024-02-03 10:01:18 +08:00
Jackson Harper
7937c73de1 Remove old warning thats no longer needed 2024-02-03 09:54:54 +08:00
Jackson Harper
9102fc9ae4
Merge pull request #3491 from omnivore-app/fix/dont-sync-readingtime-too-soon 
Only sync reading times if they are more than a minute old
 2024-02-03 09:09:52 +08:00
Jackson Harper
a745e72ce4 Allow setting postgres extra options via envvar 2024-02-03 08:57:41 +08:00
Jackson Harper
9cc046e3a2 Only sync reading times if they are more than a minute old 
this should hopefully reduce contention with other rules running
 2024-02-02 20:41:14 +08:00
Hongbo Wu
b68adc5983 fix tests 2024-02-02 17:05:09 +08:00
Hongbo Wu
df2dbb2916 fix: find library item by url not using correct index 2024-02-02 16:46:13 +08:00
Hongbo Wu
74783313da fix query 2024-02-02 16:18:41 +08:00
Hongbo Wu
1742ffda24 change priority to 5 2024-02-02 16:18:41 +08:00
Hongbo Wu
25e374f6ff fix: replace webhook endpoint with a bullmq job 2024-02-02 16:18:40 +08:00
Hongbo Wu
5e49131fc2
Merge pull request #3470 from omnivore-app/fix/async-bulk-action 
async bulk action on item count > 100 and create batch jobs
 2024-02-02 16:00:12 +08:00
Hongbo Wu
54a4022409 reduce size 2024-02-02 15:45:11 +08:00
Hongbo Wu
e980be7771 skip test 2024-02-02 14:14:41 +08:00
Hongbo Wu
26b7103f0b improve batch update 2024-02-02 13:13:18 +08:00
Jackson Harper
d339c5e8f1 Use the md5 index when looking up library items by original url 2024-02-02 13:08:00 +08:00
Hongbo Wu
2d1eebfad8 remove size 2024-02-02 12:21:08 +08:00
Hongbo Wu
9c28726a9b add parent config 2024-02-02 11:49:02 +08:00
Hongbo Wu
1bccf33320 add job_id and parent for tracking 2024-02-02 10:50:07 +08:00
Jackson Harper
ed7ac6c1b4 Fix typo in lifecycle endpoint path 2024-02-01 20:25:52 +08:00
Hongbo Wu
2a0e6f8fa4 fix tests 2024-02-01 17:44:32 +08:00
Hongbo Wu
702bd6c3c6 async bulk action on item count > 100 and create batch jobs 2024-02-01 17:44:30 +08:00
Jackson Harper
7f4f7c6a80
Merge pull request #3479 from omnivore-app/fix/queue-processor-lifecycle-hook 
Implement a lifecycle hook for queue processor so it can shut down gracefully
 2024-02-01 17:01:11 +08:00
Hongbo Wu
8ac696e87d
Merge pull request #3480 from omnivore-app/fix/mark-read-action 
fix: update reading progress in cache when mark as read action was triggered by a rule
 2024-02-01 16:56:21 +08:00
Hongbo Wu
252443f1e4 fix: update reading progress in cache when mark as read action was triggered in a rule 2024-02-01 16:45:08 +08:00
Jackson Harper
0e9e331e18 Implement a lifecycle hook for queue processor so it can shut down gracefully 2024-02-01 16:43:57 +08:00
Jackson Harper
3feef7ca66 Use the DB stored reading progress when calculating maxes 2024-02-01 16:23:21 +08:00
Jackson Harper
f03b7182b3
Merge pull request #3460 from omnivore-app/feat/reading-progress-write-behind 
Write behind cache for reading progress
 2024-02-01 14:43:45 +08:00
Hongbo Wu
abaf726044 revert the change of priority 2024-02-01 13:22:06 +08:00
Hongbo Wu
19fe60d27a change the update-db job priority to 5 and use exponential backoff strategy for retrying 2024-02-01 13:14:01 +08:00
Hongbo Wu
e13c418389 remove the inner join with highlights table in the sql of update-label job 2024-02-01 13:09:50 +08:00
Jackson Harper
02c28be403 reduce unction is no longer async 2024-02-01 11:24:13 +08:00
Jackson Harper
792f13edfc Remove async from non-async function 2024-02-01 10:49:48 +08:00
Jackson Harper
1b44cb860b Function for cron jobs dont include queue in the read position metrics since its not part of that queue 2024-02-01 10:33:51 +08:00
Jackson Harper
7d6645b87a Rebase 2024-02-01 10:02:38 +08:00
Jackson Harper
570b7a4b78 Use a set in redis and push items to the set 
This lets us grab all available items, apply them, and then
remove them from the set. Once completed the items that were
applied can be removed from the set, if any new items were
added during that time they wont be effected.
 2024-02-01 09:59:42 +08:00
Jackson Harper
0e37bfa03d We need to update readAt because that isnt resolved from the cache 2024-02-01 09:59:42 +08:00
Jackson Harper
c47a720fe4 Do not cache on write, so the most updated item is always available on read 
This is mostly an issue on tests where all the queries can be run
in a single process, but if we cache on write, it means the next
read will just have the cached value, instead of the calculated
maximum value.
 2024-02-01 09:59:42 +08:00
Jackson Harper
bdf01ddaab Clear cache on force, return updated read position values 2024-02-01 09:59:42 +08:00
Jackson Harper
a060aaf496 Make reading position parsing safer 2024-02-01 09:59:42 +08:00
Jackson Harper
9dc070243a Revert temp 2024-02-01 09:59:41 +08:00
Jackson Harper
d8ea4ff1c5 Add reading position messages to exported metrics for queue processor 2024-02-01 09:59:41 +08:00
Jackson Harper
7bb3718a8f Add job to sync read position data 2024-02-01 09:59:41 +08:00
Jackson Harper
a40f5bed55 Create a service for interacting with cached read positions 2024-02-01 09:59:41 +08:00
Jackson Harper
fbfa934479 Write behind reading progress as a list that can be reduced at commit or read time 2024-02-01 09:59:39 +08:00
Jackson Harper
c2327781e1 Remove empty ctor 2024-02-01 09:57:11 +08:00
Jackson Harper
86c80d991f Write behing cache for reading progress 2024-02-01 09:57:11 +08:00
Hongbo Wu
c3276fdaf1
Merge pull request #3466 from omnivore-app/fix/rewrite-trigger-functions-in-db 
create a job for db trigger
 2024-01-31 19:47:45 +08:00
Hongbo Wu
6d5ce451ac fix bulk action 2024-01-31 19:24:18 +08:00
Hongbo Wu
1b8111b907 wait until all jobs are done in test 2024-01-31 19:10:57 +08:00
Hongbo Wu
5e883cb2ba running worker in the test 2024-01-31 17:45:49 +08:00
Hongbo Wu
cf898299e4 some lint 2024-01-31 14:18:34 +08:00
Hongbo Wu
5646413b59 add a job for updating highlight annotations in library_item 2024-01-31 14:15:33 +08:00
Hongbo Wu
43855247b8 update bulk action to add labels 2024-01-31 13:56:56 +08:00
Hongbo Wu
5ed35bbeff create a job for update labels in library item 2024-01-31 13:32:37 +08:00
Hongbo Wu
ddddf8234e create a job for db trigger 2024-01-30 23:53:05 +08:00
Hongbo Wu
36d2c21466 remove thumbnail job on complete or failure 2024-01-30 18:24:48 +08:00
Hongbo Wu
2eedba1522 fix: assign the rule job with priority 1 and do not retry 2024-01-30 18:14:34 +08:00
Hongbo Wu
e724bbd465 fallback to worker redis client if fetched content not found in cache 2024-01-30 12:27:45 +08:00
Hongbo Wu
8d8309d3bb fix: empty client request id in save-page job data when saving pdf in rss item 2024-01-29 21:22:26 +08:00
Hongbo Wu
5faa24556e fix: string interpolation in logs 2024-01-29 19:11:26 +08:00
Hongbo Wu
809c572ca7 reduce the size of the log entry by truncating any string values to 500 characters 2024-01-29 14:32:45 +08:00
Hongbo Wu
d5d581fc54 remove redundant logs 2024-01-29 14:32:43 +08:00
Hongbo Wu
0181a2bfcd
Merge pull request #3446 from omnivore-app/fix/refresh-feed 
fix: add rss label and feed url to the pdf in rss feed item
 2024-01-29 14:21:08 +08:00
Hongbo Wu
58d384f4f6 fix pdf saving 2024-01-29 14:11:07 +08:00
Hongbo Wu
4f061546bf remove api call to upload file request 2024-01-29 12:55:19 +08:00
Hongbo Wu
a2bdddf8c9 fix tests 2024-01-29 12:30:31 +08:00
Hongbo Wu
738f85923c
Merge pull request #3442 from omnivore-app/fix/rule-handler 
move trigger rule into a job
 2024-01-29 12:20:21 +08:00
Hongbo Wu
ac5acf7f21 fix: add rss label and feed url to the pdf in rss feed item 2024-01-26 22:34:55 +08:00
Hongbo Wu
fd80724d79 fallback to guid if link of rss item is not available 2024-01-26 22:09:53 +08:00
Hongbo Wu
30fef30384 resolve conflicts 2024-01-26 20:32:56 +08:00
Hongbo Wu
e03895c8b7 enqueue trigger rule job once page is created or updated 2024-01-26 17:46:17 +08:00
Hongbo Wu
7556c6da5f move trigger rule into a job 2024-01-26 17:45:40 +08:00
Hongbo Wu
52419c39da resolve conflicts 2024-01-26 13:01:53 +08:00
Hongbo Wu
7e89235806 update content-fetch 2024-01-26 13:01:53 +08:00
Hongbo Wu
be0a2e8c52 use different redis server for mq and cache 2024-01-26 13:01:53 +08:00
Hongbo Wu
4c765378e3 do not retry if find_thumbnail job fails 2024-01-26 13:01:53 +08:00
Hongbo Wu
e54c691c09 create a job for finding thumbnail 2024-01-26 13:01:51 +08:00
Jackson Harper
f3c1b9d9a0
Merge pull request #3430 from omnivore-app/fix/queue-processor-custom-metrics 
Expose custom metrics for the queue processor
 2024-01-25 21:17:49 +08:00
Hongbo Wu
fd7c2ffb49 fix code blocks not formatted correctly in articles from wechat official accounts 2024-01-25 16:30:59 +08:00
Hongbo Wu
dae90aaffa fix code blocks not formatted correctly 2024-01-25 16:30:59 +08:00
Hongbo Wu
8834efabd4 throw error if save-page job failed 2024-01-25 16:30:59 +08:00
Hongbo Wu
5e239d2568 run readability in save-page instead of puppeteer 2024-01-25 16:30:59 +08:00
Jackson Harper
a639a2fa10 Add types to exported metrics 2024-01-25 12:37:02 +08:00
Jackson Harper
5f9be385b7 Remove unneeded escapes 2024-01-24 21:34:51 +08:00
Jackson Harper
c581372a93 Improve exported metrics naming convention 2024-01-24 21:13:13 +08:00
Hongbo Wu
7bb63b4ecd for campability with old clients, use refreshedAt to replace lastFetchedAt 2024-01-24 15:32:43 +08:00
Hongbo Wu
3f3a96297f update entity 2024-01-24 14:55:50 +08:00