prehistoric-systems/omnivore
1
0
Fork 0
mirror of https://github.com/omnivore-app/omnivore.git synced 2026-03-11 08:54:26 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
12057 commits 201 branches 31 tags 432 MiB
Commit graph

2536 commits

Author SHA1 Message Date
Hongbo Wu
01ebcbb16b add bulk upload original content job 2024-05-10 14:37:05 +08:00
Jackson Harper
e85bb9aa6a Only allow eligible users to enable digest 2024-05-10 14:34:52 +08:00
Hongbo Wu
4c48a44512 format summary of digest in library and email 2024-05-09 17:38:21 +08:00
Hongbo Wu
ab85805e24
Merge pull request #3926 from omnivore-app/feature/delete-task-api 
feature/delete task api
 2024-05-09 15:15:33 +08:00
Hongbo Wu
bd92d0847b add some logs 2024-05-09 15:15:22 +08:00
Hongbo Wu
922adbae42 add delete digest api 2024-05-09 14:02:10 +08:00
Hongbo Wu
431dc420a9 add delete task api 2024-05-09 12:43:37 +08:00
Hongbo Wu
17708915d9 wrap content in <html><body> element 2024-05-09 11:46:01 +08:00
Hongbo Wu
e1990258f1 send analytic events for each digest channel 2024-05-09 11:38:45 +08:00
Hongbo Wu
5a730f12e7 only include the "Read in Omnivore" button if the digest is saved in library 2024-05-09 10:53:57 +08:00
Hongbo Wu
90b0845909 add "Update digest preference" button to the digest email 2024-05-09 10:45:13 +08:00
Jackson Harper
279874f319
Merge pull request #3917 from omnivore-app/fix/usernames-add-posts 
Add posts and digest(s) to reserved names
 2024-05-08 21:07:37 +08:00
Hongbo Wu
edf21c8078 add digest to the reserved username list 2024-05-08 17:08:53 +08:00
Hongbo Wu
42b8aede0e randomly pick the qualified thumbnail 2024-05-08 17:04:19 +08:00
Hongbo Wu
ee2ca1ec97 add a "read in omnivore" button to the digest email 2024-05-08 16:49:12 +08:00
Jackson Harper
4169aa0f71 Add posts and digest(s) to reserved names 2024-05-08 16:14:57 +08:00
Hongbo Wu
5f4ed6ad11 add move digest to library api 2024-05-08 15:58:29 +08:00
Hongbo Wu
5d04537bd7 fix tests 2024-05-08 14:36:34 +08:00
Hongbo Wu
9259913344 allow saving digest in the library as a channel 2024-05-08 14:32:16 +08:00
Hongbo Wu
d880a6c37d send json response 2024-05-08 14:06:40 +08:00
Hongbo Wu
5c91e66b47 catch db error in digest job and only send notifications when digest created 2024-05-07 19:17:40 +08:00
Hongbo Wu
31233d8348 get client from user-agent 2024-05-07 14:53:29 +08:00
Hongbo Wu
23eae7871a send client info to the analytics 2024-05-07 12:30:51 +08:00
Hongbo Wu
bc802adc93 feat: send digestId with the push notification for deep link 2024-05-06 16:44:21 +08:00
Hongbo Wu
2fd3b930fe use the shortened title as subject 2024-05-03 16:06:50 +08:00
Hongbo Wu
e013963143 update email content 2024-05-03 15:56:19 +08:00
Hongbo Wu
6a0ea1b078 update gcs upload filename 2024-05-03 15:15:11 +08:00
Hongbo Wu
9bac4798b1 put omnivore url in each chapter 2024-05-03 14:56:45 +08:00
Hongbo Wu
7b2e336eca log unknown channel 2024-05-03 14:45:01 +08:00
Hongbo Wu
a9dd2d0d4c dedupe channels 2024-05-03 14:43:18 +08:00
Hongbo Wu
0692ab0454 add chapters to the digest email 2024-05-03 14:39:12 +08:00
Hongbo Wu
20408277d2 add digestConfig to the userPersonalization API request data 2024-05-03 14:35:55 +08:00
Hongbo Wu
7060ae771c send notifications to the channels defined in digest config 2024-05-03 14:17:22 +08:00
Hongbo Wu
3e84c2f1df fix tests 2024-05-03 12:17:07 +08:00
Hongbo Wu
0639a52676 add digestConfig to the userPersonalization entity 2024-05-03 11:13:24 +08:00
Hongbo Wu
6e554e2bca update upload file name 2024-05-02 19:10:02 +08:00
Hongbo Wu
4610c81092 upload summaries to gcs 2024-05-02 19:00:21 +08:00
Hongbo Wu
298a50fba2 allow random model 2024-05-02 18:33:41 +08:00
Hongbo Wu
49bdf0a6f4 select model from prompt and defaults to openai 2024-05-02 18:30:04 +08:00
Jackson Harper
9a9147652f We dont need the newsletter classification email anymore 2024-05-02 13:40:45 +08:00
Hongbo Wu
d589c18b68 generate a unique id for each scheduled digest to avoid duplication 2024-04-30 12:48:19 +08:00
Hongbo Wu
8edfce596b revert to batch api call 2024-04-30 12:48:19 +08:00
Hongbo Wu
83ed971b28 improve logs 2024-04-30 12:48:19 +08:00
Hongbo Wu
aa30b592b5 reduce debug logs 2024-04-30 12:48:19 +08:00
Hongbo Wu
debb396b44 better error handling 2024-04-30 12:48:19 +08:00
Hongbo Wu
ea227b6462 use claude 2024-04-30 12:48:19 +08:00
Hongbo Wu
0a2f9dba25 filter out summary if longer than 1000 words 2024-04-30 12:48:19 +08:00
Hongbo Wu
cbf1f97630 fix timer on get candidates function 2024-04-30 12:48:19 +08:00
Hongbo Wu
9a6ec7a58b filter out summary if shorter than 200 2024-04-30 12:48:19 +08:00
Hongbo Wu
3954fa309b randomly select at most 25 candidates and time each step 2024-04-30 12:48:19 +08:00
Hongbo Wu
32be8126df reduce number of db calls for search library items 2024-04-30 12:48:19 +08:00
Hongbo Wu
14e91d338d do not create user profile and topics 2024-04-30 12:48:19 +08:00
Hongbo Wu
d0d088aae3 fix: remove redundant user query 2024-04-30 12:48:19 +08:00
Hongbo Wu
cfc1d12835
Merge pull request #3866 from omnivore-app/feature/data-loader 
feat: use dataloader for labels and highlights
 2024-04-29 15:20:53 +08:00
Jackson Harper
4632d08aa0
Merge pull request #3863 from omnivore-app/fix/fetch_feed 
increase rss parser timeout to 20 seconds
 2024-04-28 22:45:55 -07:00
Jackson Harper
c70c36ec7d
Merge pull request #3826 from kimlimjustin/feat/increase-daily-upload-limit 
feat: increase daily upload limit
 2024-04-28 22:45:10 -07:00
Hongbo Wu
8549c16ceb fix import 2024-04-28 21:55:09 +08:00
Hongbo Wu
ef2ff36f0e fetch api key once 2024-04-28 21:51:30 +08:00
Hongbo Wu
a78d438e21 fix: findOne with relations will execute query twice 2024-04-28 15:58:55 +08:00
Hongbo Wu
7ef6973a47 load labels when getting item 2024-04-28 14:25:11 +08:00
Hongbo Wu
89d6564d74 increase log metadata size to 10000 characters 2024-04-26 15:18:34 +08:00
Hongbo Wu
342db3928c batching upload_files 2024-04-26 14:44:17 +08:00
Hongbo Wu
01246207cb fix logger 2024-04-26 12:06:20 +08:00
Hongbo Wu
66d7649620 fix tests 2024-04-26 11:54:55 +08:00
Hongbo Wu
2231f03948 reduce joins 2024-04-26 11:36:16 +08:00
Hongbo Wu
364f8f102f use enum 2024-04-26 11:24:07 +08:00
Hongbo Wu
6493b8e2f6 batching recommendations 2024-04-26 11:17:52 +08:00
Hongbo Wu
fbe816d2d2 convert content if requested 2024-04-26 11:07:41 +08:00
Hongbo Wu
c7ecbd53f8 put dataloaders in apollo context 2024-04-25 23:26:55 +08:00
Hongbo Wu
2904c321e9 batching highlights 2024-04-25 23:03:08 +08:00
Hongbo Wu
1445dee933 batching labels 2024-04-25 22:20:56 +08:00
Hongbo Wu
83526e1b56 increase rss parser timeout to 20 seconds 2024-04-25 10:55:27 +08:00
Jackson Harper
636fa17712 Default to 5 items 2024-04-24 16:39:46 -07:00
Jackson Harper
ae8790b545
Merge pull request #3848 from omnivore-app/feat/ios-digest-view 
iOS Explain and Digest views
 2024-04-24 16:13:50 -07:00
Jackson Harper
3dcc514be7
Merge pull request #3835 from omnivore-app/fix/guess-user-email 
return UNKNOWN as error code if email exists when user signs up
 2024-04-24 16:06:46 -07:00
Jackson Harper
5a6d57e3da Remove uneeded 2024-04-24 15:58:39 -07:00
Hongbo Wu
0c0a95a79c fix newsletter dir not saved correctly 2024-04-24 21:10:13 +08:00
Hongbo Wu
e788c71eff rename variables 2024-04-24 16:49:46 +08:00
Hongbo Wu
e680202b7e better error message 2024-04-24 16:17:05 +08:00
Hongbo Wu
824b256d20 fix memory leak from axios error 2024-04-24 15:55:54 +08:00
Jackson Harper
e349f6c675
Merge pull request #3853 from omnivore-app/fix/android-obsidian-cors 
Add localhost to allow android Obsidian to query the API
 2024-04-23 08:01:37 -07:00
Jackson Harper
dfdb04af92 Add localhost to allow android Obsidian to query the API 2024-04-23 07:57:41 -07:00
Hongbo Wu
3fb7193fb5 reduce getTextNodesBetween timeout to 60 seconds 2024-04-23 21:28:02 +08:00
Hongbo Wu
002e455bbd fix tests 2024-04-23 17:54:52 +08:00
Hongbo Wu
0faafa0786 fix rebase conflicts 2024-04-23 16:04:59 +08:00
Hongbo Wu
2f7c18b363 add more logs 2024-04-23 15:50:15 +08:00
Hongbo Wu
1b6f4976d3 remove timeout 2024-04-23 15:50:15 +08:00
Hongbo Wu
de258ae0bf timeout if job takes more than 10 minutes 2024-04-23 15:50:15 +08:00
Hongbo Wu
a77ade0890 fix: export all items job stuck 2024-04-23 15:50:14 +08:00
Hongbo Wu
201a5817ef
Merge pull request #3843 from omnivore-app/fix/newsletter-language 
fix language and dir not saved for newsletters
 2024-04-23 14:45:36 +08:00
Jackson Harper
2449c872bf Add Obsidian's capacitor origin to our CORs config 2024-04-22 19:15:20 -07:00
Jackson Harper
cd5940b840 Feature flag explain 2024-04-22 19:04:10 -07:00
Jackson Harper
f0be6a41a9 Clean up, remove old stubbed stuff 2024-04-22 17:40:50 -07:00
Jackson Harper
7f0a95b454 Add iOS digest views 2024-04-22 15:52:28 -07:00
Hongbo Wu
dcf7c570c4 fix tests 2024-04-22 10:57:48 +08:00
Hongbo Wu
9f65588ba0 fix language and dir not saved for newsletters 2024-04-21 10:41:29 +08:00
Hongbo Wu
8501509564 include content in updateSince api response by default for offline use 2024-04-19 14:26:17 +08:00
Hongbo Wu
fa4b3bf462 fix cron schedule bug with bull 2024-04-19 14:25:18 +08:00
Hongbo Wu
ac775ae8f2 Drain httpServer when apollo server stops 2024-04-19 14:25:18 +08:00
Hongbo Wu
273ff9d4be increase max ai-digest beta users to 10 2024-04-19 14:25:18 +08:00
Hongbo Wu
07475e69a4 change cron to schedule at 10:30 2024-04-19 14:25:18 +08:00
Hongbo Wu
a69d1240c8 add urlsToAudio back 2024-04-19 14:25:18 +08:00
Hongbo Wu
efa930a31a send push notifications when digest is ready 2024-04-19 14:25:18 +08:00
Hongbo Wu
7871154160 mark failed task 2024-04-19 14:25:18 +08:00
Hongbo Wu
b93573460c filter out summary if the length is longer than the original content 2024-04-19 14:25:18 +08:00
Hongbo Wu
099310ed95 reuse existing candidates if there is no one 2024-04-19 14:25:18 +08:00
Hongbo Wu
0d8b17f6aa remove urlsToAudio from digest 2024-04-19 14:25:18 +08:00
Hongbo Wu
1da9b51dec do not reuse candidates after refreshing 2024-04-19 14:25:18 +08:00
Hongbo Wu
09ce46e7ea include readable content in search api response by default 2024-04-19 11:30:09 +08:00
Hongbo Wu
22850bb6ab return UNKNOWN if email exists when user signs up 2024-04-18 14:04:38 +08:00
Jackson Harper
9ed94b4571
Merge pull request #3796 from omnivore-app/feature/captcha 
feature: add captcha to email-login and reset-password pages
 2024-04-18 09:48:08 +08:00
Hongbo Wu
26579d9cbb use ### for title in the digest content 2024-04-17 18:26:48 +08:00
Hongbo Wu
3e51aa7c03 generate description 2024-04-17 16:49:19 +08:00
Hongbo Wu
0eb6460bec fix tests 2024-04-17 16:35:29 +08:00
Hongbo Wu
aeda4f5d5d add libraryItemIds to the payload 2024-04-17 16:20:00 +08:00
Hongbo Wu
57cb897839 generate content 2024-04-17 15:45:05 +08:00
Hongbo Wu
af184ded22 dedupe repeated jobs 2024-04-17 15:36:16 +08:00
Hongbo Wu
6222d567a4 add byline to the digest 2024-04-17 14:32:07 +08:00
Hongbo Wu
448e432807 runs at 1pm pst 2024-04-17 13:03:48 +08:00
Hongbo Wu
b2a55a78d3 check if feature is granted 2024-04-17 12:53:29 +08:00
Hongbo Wu
259b778aba add schedule and other parameters 2024-04-17 12:16:15 +08:00
Hongbo Wu
6a4faf6f3c add description to the digest 2024-04-17 10:04:52 +08:00
Hongbo Wu
47fd4d9663 save id, url, thumbnail and createdAt in the digest 2024-04-17 09:58:11 +08:00
Hongbo Wu
6bfb4842e2 fix idx 2024-04-16 21:27:20 +08:00
Hongbo Wu
4bc1246a47 fixing html 2024-04-16 21:00:53 +08:00
Hongbo Wu
71477b1712 implement create digest job 2024-04-16 21:00:51 +08:00
Jackson Harper
daaf493d4b Update stubbed data 2024-04-16 21:00:27 +08:00
Jackson Harper
bcddf556d5 Write speech files to redis 2024-04-16 21:00:27 +08:00
Jackson Harper
fbaa9ce663 Digest job 2024-04-16 21:00:27 +08:00
Jackson Harper
d2ea93a7dd Stub out some changes to the digest response 2024-04-16 21:00:27 +08:00
Hongbo Wu
e30fa13a2e add env to the posthog event 2024-04-16 21:00:27 +08:00
Hongbo Wu
03e497b489 use a separate queue for digest job 2024-04-16 21:00:26 +08:00
Hongbo Wu
cdb0b7525e cleanup 2024-04-16 21:00:10 +08:00
Hongbo Wu
bc955fbeeb remove stub job 2024-04-16 21:00:10 +08:00
Hongbo Wu
111dcd1562 send ranking models and summary models to posthog and log comment 2024-04-16 21:00:10 +08:00
Hongbo Wu
4342cdeb3e expire digest in one day 2024-04-16 21:00:10 +08:00
Hongbo Wu
521698646d update digest mock data 2024-04-16 21:00:10 +08:00
Hongbo Wu
a47a5a7d9c mock saving digest in redis 2024-04-16 21:00:10 +08:00
Hongbo Wu
e37bcd7408 send feedback api 2024-04-16 21:00:10 +08:00
Hongbo Wu
1edb3419f8 get digest api 2024-04-16 21:00:10 +08:00
Hongbo Wu
12aa7d4447 create digest job api 2024-04-16 21:00:10 +08:00
Hongbo Wu
b4911639c7 stub create digest job 2024-04-16 21:00:09 +08:00
Jackson Harper
be57f81bf1 Temp re-add features attribute to User 2024-04-16 13:59:04 +08:00
Hongbo Wu
8e31ac4fea fix tests 2024-04-16 12:24:20 +08:00
Hongbo Wu
d79cc1c39f wrap select in a auth transaction 2024-04-16 11:26:29 +08:00
Hongbo Wu
a9779a51e8 fix: LABEL_ALREADY_EXIST bug occured during label creation 
* comparing both user_id and label name when looking for an existing matching label
 2024-04-16 09:56:57 +08:00
Justin Maximillian Kimlim
56b87672fd
feat: increase daily upload limit 2024-04-16 00:49:26 +07:00
Hongbo Wu
9ccd1e90fe save received email as non-article by default 2024-04-15 18:54:15 +08:00
Hongbo Wu
4346746e0a save empty url for email attachment in upload_file table as it is required 2024-04-15 17:36:42 +08:00
Hongbo Wu
cbddc83829
Merge pull request #3784 from omnivore-app/feature/reply-to-email 
add reply to email API which replies Okay to the sender
 2024-04-15 15:24:32 +08:00
Hongbo Wu
e23d3d5933 update readable content instead 2024-04-12 13:26:11 +08:00
Hongbo Wu
f95642be2b remove redundant field 2024-04-11 18:47:19 +08:00
Hongbo Wu
448c78a87e add dotenv 2024-04-11 18:25:24 +08:00
Hongbo Wu
54e7c52b5e always update the state of the pdf item to succeeded/failed, even if we fail to parse the content 2024-04-11 18:07:53 +08:00
Hongbo Wu
f66d3fa36c migrate inbound-email tasks from pubsub to bullmq 2024-04-11 11:08:22 +08:00
Hongbo Wu
cd6594a2fd add logs 2024-04-11 11:08:22 +08:00
Hongbo Wu
67657773cf track the usage of the api 2024-04-11 11:08:22 +08:00
Hongbo Wu
cd6f3e6bbe enforce usage limits for the API 2024-04-11 11:08:22 +08:00
Hongbo Wu
5f239d2dcb update graceful shutdown 2024-04-11 11:08:22 +08:00
Hongbo Wu
f2f18eb12b reply only allowed text 2024-04-11 11:08:22 +08:00
Hongbo Wu
e7fc1316dd enqueue reply email job 2024-04-11 11:08:22 +08:00
Hongbo Wu
74f5b922dd update received email reply 2024-04-11 11:08:22 +08:00
Hongbo Wu
dce0b15f4f send to the reply-to address if it exists or the from address 2024-04-11 11:08:22 +08:00
Hongbo Wu
764b05fff4 save reply-to in recent emails 2024-04-11 11:08:22 +08:00
Hongbo Wu
5cc5c559ac add reply to email API which replies Okay to the sender 2024-04-11 11:08:22 +08:00
Hongbo Wu
a59c850c5f fix telegram channel 2024-04-09 19:44:16 +08:00
Hongbo Wu
e7fc2c4567 reset failed_at to null if a feed is refreshed successfully after failure 2024-04-09 18:39:17 +08:00
Hongbo Wu
77c11fb80f
Merge pull request #3767 from omnivore-app/fix/verification 
Enqueue emails
 2024-04-08 12:30:37 +08:00
Hongbo Wu
ec309f5b0d verify captcha in email-login and forgot-password api 2024-04-08 12:27:29 +08:00
Hongbo Wu
7abb396032 remove stub 2024-04-05 22:53:03 +08:00
Hongbo Wu
af140cc8b4 remove user name from email template 2024-04-05 16:59:23 +08:00
Hongbo Wu
8af2bb076c rename send mail functions 2024-04-05 16:59:20 +08:00
Hongbo Wu
c37b31dcb6 fix tests 2024-04-05 16:57:02 +08:00
Hongbo Wu
10dcb922f2 Enqueue confirmation email 2024-04-05 16:57:02 +08:00
Hongbo Wu
207241934d show label_attached instead of label_created in rules 2024-04-05 16:51:11 +08:00
Hongbo Wu
25fb9fd698 deduplicate labels when there are case-insensitive identical names 2024-04-05 16:27:17 +08:00
Hongbo Wu
6f03b214b5 allow delete internal labels 2024-04-05 15:58:39 +08:00
Hongbo Wu
9f4f1d07ca do not create label if label name case-insensitively matches existing label name 2024-04-05 15:50:01 +08:00
Hongbo Wu
906b6c1ea5 fix web 2024-04-04 18:53:10 +08:00
Hongbo Wu
58708a507e use old event format for the compatibility with the old webhooks 2024-04-04 18:22:18 +08:00
Hongbo Wu
45a5382d7c do not send bad_request error if db failure 2024-04-04 12:19:18 +08:00
Hongbo Wu
5e7a82562b resolve conflicts after rebasing 2024-04-04 12:19:17 +08:00
Hongbo Wu
b2cd43136f hide webhooks page 2024-04-04 12:19:16 +08:00
Hongbo Wu
0fab3c7904 Add highlight_updated event 2024-04-04 12:17:28 +08:00
Hongbo Wu
88a7e8d85b fix tests 2024-04-04 12:17:15 +08:00
Hongbo Wu
81f21141fe optimize ts-node-dev 2024-04-04 12:17:15 +08:00
Hongbo Wu
841a0026ff do not send page_update event for thumbnailer 2024-04-04 12:17:15 +08:00
Hongbo Wu
1b24a24f7c remove slug from highlight event 2024-04-04 12:17:15 +08:00
Hongbo Wu
d67aea6d8a deduplicate notion page by omnivore id 2024-04-04 12:17:13 +08:00
Hongbo Wu
0428da3b71 fix notion syncing error 2024-04-04 12:16:45 +08:00
Hongbo Wu
6954fffd24 fix readwise syncing error 2024-04-04 12:16:26 +08:00
Hongbo Wu
ff78a4b266 fix readwise syncing error 2024-04-04 12:16:12 +08:00
Hongbo Wu
f35ac036eb add UI for export in rules 2024-04-04 12:15:28 +08:00
Hongbo Wu
ad2e8ee002 trigger export in rules 2024-04-04 12:14:38 +08:00
Hongbo Wu
58511049a2 rename LABEL_CREATED to LABEL_ATTACHED in webhook data 2024-04-04 12:02:29 +08:00
Hongbo Wu
e2b3235046 delete unneccessary properties from webhook data 2024-04-04 12:02:29 +08:00
Hongbo Wu
88d6455222 default filter is in:all 2024-04-04 12:02:29 +08:00
Hongbo Wu
c551d9fe6b allow send data to webhook with rules 2024-04-04 12:02:29 +08:00
Hongbo Wu
07cee98ff9 allow label and highlight events to trigger rules 2024-04-04 12:02:29 +08:00
Hongbo Wu
e28dd73f9c
Merge pull request #3780 from omnivore-app/fix/notion 
sync items and highlights to user created database in notion
 2024-04-04 11:58:59 +08:00
Hongbo Wu
10cca7f05d find the title property and update its name to Title 2024-04-04 10:43:39 +08:00
Jackson Harper
c45a9d365b Also use mailjet for password changes 2024-04-03 22:25:31 +08:00
Hongbo Wu
491611b05f sync items and highlights to user created database in notion 2024-04-03 22:25:19 +08:00
Jackson Harper
3c1abb964b Also use mailjet for reset password emails 2024-04-03 22:07:28 +08:00
Hongbo Wu
f794cf707a
Merge pull request #3769 from omnivore-app/fix/notion 
export last 100 items in one job
 2024-04-03 10:22:56 +08:00
Hongbo Wu
08fb92d599 fix: site filter in rules matching both site name and url 2024-04-02 18:32:36 +08:00
Hongbo Wu
cc07d2bfbf remove comment 2024-04-02 17:56:46 +08:00
Hongbo Wu
d83dc17651 fix tests 2024-04-02 17:49:29 +08:00
Hongbo Wu
e19d6c86d8 export last 100 items in one job 2024-04-02 15:51:02 +08:00
Jackson Harper
aea354c3f2 No longer make captcha optional on web signup 2024-04-01 19:07:21 +08:00
Jackson Harper
3a59317217 Force captcha on sign up 2024-04-01 17:06:45 +08:00
Hongbo Wu
6ca092fbce use different prefix for different rate limiter 2024-04-01 13:44:10 +08:00
Hongbo Wu
a97542d4ba use redis for hourly rate limiter 2024-04-01 13:01:02 +08:00
Jackson Harper
788460c144
Merge pull request #3761 from omnivore-app/feat/signup-captcha 
Add recaptcha on the web sign up flow
 2024-04-01 12:24:39 +08:00
Jackson Harper
23f5377b62 Clean up isRecaptchaResponse function 2024-04-01 12:03:36 +08:00
Jackson Harper
0fe471da82 Dont require score on challenge recaptcha 2024-04-01 11:32:10 +08:00
Jackson Harper
822fcb1e74 Add some logging 2024-04-01 11:17:15 +08:00
Jackson Harper
9a159084c8 Update the isValidSignUp to check for URLs 2024-04-01 11:17:15 +08:00
Jackson Harper
442475d684 Make this optional for self hosters 2024-04-01 11:17:15 +08:00
Jackson Harper
7518e164a5 Utils for recaptcha 2024-04-01 11:15:19 +08:00
Hongbo Wu
5dd308d78e store the rate limit data in redis if redis is connected 2024-03-31 20:13:27 +08:00
Jackson Harper
2f6ef5e447 Return false if can not send with Mailjet 2024-03-31 19:22:56 +08:00
Jackson Harper
cfe4315fe2 Add an email fallback to Mailjet 2024-03-31 19:03:26 +08:00
Jackson Harper
200354d181 Add url present check on email sign up 2024-03-31 09:55:53 +08:00
Jackson Harper
ede99dcd33 Better matching for youtube video URLs 2024-03-30 10:42:01 +08:00
Jackson Harper
b69ba150c5 Remove debug 2024-03-29 12:28:26 +08:00
Jackson Harper
3a352a9705 Dont need to chunk this anymore 2024-03-29 12:05:32 +08:00
Jackson Harper
08ed061756 Change model for youtube transcripts 2024-03-29 11:31:27 +08:00
Hongbo Wu
f677f1167f get feature list on web 2024-03-28 15:27:37 +08:00
Hongbo Wu
cc2fe04568 add featureList to the User type which contains a list of feature type 2024-03-28 14:42:08 +08:00
Hongbo Wu
1afec1ddf3 check if feature is granted when process youtube video 2024-03-28 14:41:30 +08:00
Hongbo Wu
45417efd0c bump youtube users to 500 2024-03-28 12:15:36 +08:00
Hongbo Wu
8d8c7e8a26 fix: allow max 1000 users to use notion feature 2024-03-28 12:13:28 +08:00
Jackson Harper
b5bea17ffe Export the age of the oldest job in the queue 2024-03-28 10:25:44 +08:00
Jackson Harper
82168fe439
Merge pull request #3739 from omnivore-app/fix/youtube-queue 
Only check for summaries if the updated entity is a page item
 2024-03-28 09:40:44 +08:00
Hongbo Wu
d4ed994914
Merge pull request #3740 from omnivore-app/fix/logger 
replace logger.warn with warning
 2024-03-27 16:00:08 +08:00
Jackson Harper
38f959685f Change warn to warning 2024-03-27 15:41:15 +08:00
Hongbo Wu
92abfed2df replace logger.warn with warning 2024-03-27 15:41:13 +08:00
Jackson Harper
d23a7d56c0 Only check for summaries if the updated entity is a page item 2024-03-27 13:54:54 +08:00
Jackson Harper
5b736c1c2e
Merge pull request #3738 from omnivore-app/fix/api-youtube-logging 
Add more logging to youtube jobs
 2024-03-27 11:44:37 +08:00
Hongbo Wu
7c281359e8 fix recommendations not filtered correctly by rules 2024-03-27 11:29:47 +08:00
Jackson Harper
28222dcd56 Add more logging to youtube jobs 2024-03-27 11:25:46 +08:00
Hongbo Wu
c022cf0a22 label match wildcard 2024-03-27 10:54:24 +08:00
Hongbo Wu
3544dd1709 remove feed content from job data 2024-03-27 10:38:51 +08:00
Hongbo Wu
af78b6f4f0 append newsletter labels to avoid labels attached by rules being replaced 2024-03-27 10:37:04 +08:00
Hongbo Wu
41b90cbfd1 fix some undefined error 2024-03-26 17:53:48 +08:00
Hongbo Wu
d02d9444a9 remove content from update event too 2024-03-26 17:46:38 +08:00
Hongbo Wu
c3dba7e9da fix typo 2024-03-26 17:44:05 +08:00
Hongbo Wu
4c458c7118 fallback to search query if implicit fields are in the filter 2024-03-26 17:34:57 +08:00
Hongbo Wu
37459f64c7 remove content from job data 2024-03-26 16:49:26 +08:00
Hongbo Wu
209b08e2ff add notion to the features list 2024-03-26 12:32:01 +08:00
Hongbo Wu
f7c25fe05a do not create a new connection for the discover api 2024-03-23 10:03:06 +08:00
Hongbo Wu
282ed75a5f
Merge pull request #3698 from omnivore-app/fix/notion 
Fix: notion
 2024-03-21 14:08:16 +08:00
Hongbo Wu
445d2846b5 export last 100 items 2024-03-21 14:06:56 +08:00
Hongbo Wu
390d12eea1 add export to integration and get task api 2024-03-21 11:14:23 +08:00
Hongbo Wu
b1e3811e4f
Merge pull request #3707 from omnivore-app/fix/rule 
send labels and highlight annotations in page_create event
 2024-03-21 10:52:15 +08:00
Hongbo Wu
6385b40e38 revert batch size change 2024-03-20 23:07:46 +08:00
Hongbo Wu
3d986a9036 send labels and highlight annotations in page_create event 2024-03-20 22:33:24 +08:00
Jackson Harper
59cc954618 Add 12hrs option, fix linting issue 2024-03-20 15:36:53 +08:00
Jackson Harper
82ef57a36a Add last24hrs query for time based search queries to get recently saved/changed items 2024-03-20 15:15:18 +08:00
Jackson Harper
014ccb1eae Set published date from youtube data 
its not clear how consistent the formatting is on these dates,
the one i tested seems like it was a UTC formatted dd M, yyyy string
 2024-03-20 13:10:41 +08:00
Hongbo Wu
689eeaeacd disable sending pubsub events for rss feed temporarily to resolve conflicts 2024-03-20 10:46:13 +08:00
Hongbo Wu
2e374557bf send labels in the page_created events 2024-03-20 10:36:36 +08:00
Hongbo Wu
abf9706539 resolve conflicts after rebasing 2024-03-20 10:36:21 +08:00
Hongbo Wu
7f224cc1e1 running text search on all text fields 2024-03-20 10:36:21 +08:00
Hongbo Wu
edeafc9a71 fix bug 2024-03-20 10:36:21 +08:00
Hongbo Wu
de28a9ad5f fix bug 2024-03-20 10:36:21 +08:00
Hongbo Wu
f1eff130af filter data locally with liqe 2024-03-20 10:36:21 +08:00
Hongbo Wu
e87e6dd102 filter data locally 2024-03-20 10:36:21 +08:00
Hongbo Wu
a49366b4cc remove unnecessary fields from item events 2024-03-20 10:36:19 +08:00
Hongbo Wu
208a5895ef remove unnecessary fields from item events 2024-03-20 10:35:58 +08:00
Hongbo Wu
9af4235233 skip failed to parse rules and show the timestamp in the UI 2024-03-20 10:35:58 +08:00
Hongbo Wu
c05721c87e validate the filter before creating rules 2024-03-20 10:35:57 +08:00
Hongbo Wu
76f3321680 revalidate integration once updated 2024-03-20 10:32:32 +08:00
Hongbo Wu
bf59fbf26e add integration api 2024-03-20 10:32:31 +08:00
Hongbo Wu
7e04b25497
Merge pull request #3696 from omnivore-app/fix/thumbnail 
fix/thumbnail
 2024-03-20 10:26:34 +08:00
Hongbo Wu
315c625482 fix title not fetched correctly when using preview content in rss feeds 2024-03-19 18:07:27 +08:00
Jackson Harper
e47e169402
Merge branch 'main' into discover-no-ts-changes 2024-03-19 15:50:51 +08:00
Hongbo Wu
ad32aa82ce Export highlight notes as children of highlights block in notion 2024-03-19 11:17:42 +08:00
Jackson Harper
21d4efc2b5 Allow opting into the YouTube transcript feature 2024-03-18 13:56:37 +08:00
Jackson Harper
4e50c51598
Merge pull request #3661 from omnivore-app/feat/job-youtube-transcript-generator 
Pull metadata from YouTube for library items
 2024-03-17 09:40:28 +08:00
Jackson Harper
596ab5a7aa Feature flag the transcripts 2024-03-15 20:28:02 +08:00
Jackson Harper
b87073b1fb Add youtube-transcripts beta feature 2024-03-15 20:25:44 +08:00
Jackson Harper
d3aa671200 Prettier 2024-03-15 10:44:40 +08:00
Jackson Harper
fdc8055b79 Prettier 2024-03-15 10:43:16 +08:00
Hongbo Wu
17e66aa010 append new highlights only 2024-03-14 22:13:27 +08:00
Hongbo Wu
1085dcc824 read settings 2024-03-14 21:36:47 +08:00
Hongbo Wu
69e74d432d append highlights to the existing page 2024-03-14 19:50:26 +08:00
Jackson Harper
f2d23626a5 Improve error logging / fix linting 2024-03-14 17:39:13 +08:00
Jackson Harper
dc6c047aec Add GCS cache 2024-03-14 16:12:06 +08:00
Jackson Harper
629c044273 Queue the transcript processing as a separate job 
Handle YouTube in two steps, first get metadata then
get the transcript.
 2024-03-14 15:09:53 +08:00
Hongbo Wu
1ea09e20bd fix: subscription not updated correctly after rss feed refreshed 
* sort the shallow copy of the userIds array so the original array not mutated
 2024-03-14 12:56:42 +08:00
Jackson Harper
fef28d1c6a Linting fix 2024-03-14 12:48:56 +08:00
Jackson Harper
3ee6787e39 Improve transcript generation 2024-03-14 12:48:56 +08:00
Jackson Harper
39dfa920b5 More fixes to youtube processor 2024-03-14 12:48:56 +08:00
Jackson Harper
6bcdbfa8f0 Linting 2024-03-14 12:48:56 +08:00
Jackson Harper
7c3d15e31a Add some scrolling on youtube videos 2024-03-14 12:48:56 +08:00
Jackson Harper
2dbd16a61e Pull duration and description from YouTube metadata 2024-03-14 12:48:56 +08:00
Jackson Harper
f7225b298a Rebase 2024-03-14 12:48:56 +08:00
Hongbo Wu
81e489021e add popular reads to the continue reading section of new user library 2024-03-14 11:14:37 +08:00
Hongbo Wu
6310e5f83b update form 2024-03-13 15:55:34 +08:00
Hongbo Wu
ff157bd453
Merge pull request #3620 from omnivore-app/feature/notion-integration 
Feature: Notion Integration
 2024-03-12 22:37:51 +08:00
Jackson Harper
3266d6a9a3 Linting fixes 2024-03-12 21:34:20 +08:00
Hongbo Wu
8eaf754538 do not get content from db when exporting item to integrations 2024-03-12 15:37:19 +08:00
Jackson Harper
d2c22eaea2
Merge pull request #3649 from Podginator/discover-clean 
Discover clean
 2024-03-12 10:11:00 +08:00
Jackson Harper
e15ae29961 Use folders when handling push notifications 2024-03-08 21:11:16 +08:00
Jackson Harper
e093738dcc Send libraryItemId with push notifications 2024-03-08 17:24:02 +08:00
Jackson Harper
7d77291f11 Push configuration on iOS 2024-03-08 17:15:59 +08:00
Jackson Harper
6bc100942f Send libraryItemId and thumbnail with push notifications 2024-03-08 16:40:49 +08:00
Thomas Rogers
7d291f084b Merge all changes from main, update theming of Discover 2024-03-07 17:39:57 +01:00
Hongbo Wu
44178c1c1e Add author 2024-03-07 13:24:02 +08:00
Hongbo Wu
39778feb88 export highlights and labels 2024-03-06 20:23:42 +08:00
Hongbo Wu
d41459a5cd log unhandled error 2024-03-06 12:04:16 +08:00
Hongbo Wu
289f1cf5e2 better error message 2024-03-06 11:54:41 +08:00
Hongbo Wu
c346ba8703 do not throw error if failed to find tokens 2024-03-06 11:47:17 +08:00
Hongbo Wu
ec32945ba2 update block 2024-03-05 21:00:39 +08:00
Hongbo Wu
dfa5512512 get parentPageId from settings 2024-03-05 16:18:15 +08:00
Hongbo Wu
1edf801b02 add settings jsonb column to the integrations 2024-03-05 16:13:50 +08:00
Hongbo Wu
f1b1f2c4c1 send created event only 2024-03-05 15:39:09 +08:00
Hongbo Wu
741f3213d9 integrate with notion api client 2024-03-05 12:43:18 +08:00
Hongbo Wu
e20908a0c2 add export to notion method 2024-03-04 18:44:25 +08:00
Hongbo Wu
d36c937765 fix redirect_uri error 2024-03-04 17:49:59 +08:00
Hongbo Wu
94db3e0de7 Add authorization with notion 2024-03-04 17:30:05 +08:00
Hongbo Wu
4a70291898 add directionality to the SearchItem and Article Grahql type in the schema 2024-03-04 13:23:52 +08:00
Hongbo Wu
0e46dc2302 save dir in the database 2024-03-04 12:28:51 +08:00
Hongbo Wu
08e5caddc2 retry export-item job if fails 2024-03-01 15:03:01 +08:00
Hongbo Wu
e33cf465fe throw error if export-item job fails 2024-03-01 15:00:28 +08:00
Hongbo Wu
a337e0e39d reduce export-item job priority to 50 2024-03-01 14:52:49 +08:00
Jackson Harper
a74377556b
Merge pull request #3596 from omnivore-app/fix/web-layout-fixes 
Small layout fixes for mobile
 2024-02-29 18:20:33 +08:00
Hongbo Wu
2636524642 fix typo 2024-02-29 16:35:52 +08:00
Hongbo Wu
c3e8173e5e fix search results for archived items which state is not archived 2024-02-29 16:34:55 +08:00
Jackson Harper
34ee75c555 Dont re-run summarize job 2024-02-29 15:50:25 +08:00
Jackson Harper
531f23fc4c
Merge pull request #3591 from omnivore-app/feat/summaries 
Summaries
 2024-02-29 14:35:12 +08:00
Hongbo Wu
3af62d5d89 fix typo 2024-02-28 17:13:11 +08:00
Hongbo Wu
7567e3d0f6 * return only count if size is 0 because limit 0 is not allowed in typeorm 2024-02-28 16:57:15 +08:00
Jackson Harper
6861d271bd Clean up assertions in summary job 2024-02-28 15:49:24 +08:00
Jackson Harper
0382fb4913 Fix comparison of library item state 2024-02-28 15:34:36 +08:00
Jackson Harper
692a992e77 Remove some debug, clean up summaries 2024-02-28 15:18:23 +08:00
Jackson Harper
c20d8e6b4d Add summaries, display accont features 2024-02-28 15:14:32 +08:00
Jackson Harper
bbe6204b91 summaries 2024-02-28 15:12:09 +08:00
Hongbo Wu
0369a39f08 replace $set with $set_once 2024-02-26 18:14:10 +08:00
Hongbo Wu
41aa976625 flush remaining events when server exits 2024-02-26 17:28:28 +08:00
Hongbo Wu
d26a2f403d replace segment with posthog 2024-02-26 16:47:04 +08:00
Hongbo Wu
c13896aeb2
Merge pull request #3567 from omnivore-app/feature/feed-summary 
feature/feed summary
 2024-02-26 12:19:57 +08:00
Jackson Harper
33fa36710f
Merge pull request #3568 from omnivore-app/feat/rules-delete-item-rule 
Add a rule to delete items
 2024-02-26 12:13:40 +08:00
Hongbo Wu
7db7099822 fix tests 2024-02-23 19:43:43 +08:00
Hongbo Wu
6b0cefe190 add representation column to the highlight table and gql schema which is either CONTENT or FEED_CONTENT 2024-02-23 17:14:52 +08:00
Hongbo Wu
67f54a1372 add fetchContent resolver to the function list 2024-02-23 15:44:53 +08:00
Hongbo Wu
49d1ae5037 fix bug 2024-02-23 15:32:41 +08:00
Hongbo Wu
510cd18b17 use json_agg to aggregate enum in db 2024-02-23 14:54:37 +08:00
Jackson Harper
90c049506d Add a rule to delete items 2024-02-23 14:42:10 +08:00
Hongbo Wu
df0bda9cb2 fix a typo 2024-02-23 14:21:34 +08:00
Hongbo Wu
f4b6e470f2 replace fetch_content with fetch_content_type as a enum: ALWAYS, NEVER, WHEN_EMPTY 2024-02-23 14:19:22 +08:00
Hongbo Wu
08e4bed146 rename preview_content to feed_content in library_item table and gql schema 2024-02-23 12:53:39 +08:00
Hongbo Wu
61e9ab3a17
Merge pull request #3566 from omnivore-app/perf/optimize-slow-query 
perf/optimize slow query
 2024-02-23 12:53:05 +08:00
Hongbo Wu
8ce7550582 create an index for upload_file_id in library_item table 2024-02-23 11:33:48 +08:00
Hongbo Wu
7c7d594b91 use low priority queue if user saved 5 items in the last minute. previously it was created 2024-02-23 10:17:18 +08:00
Hongbo Wu
b947100278 increase batch size to 100 2024-02-22 20:23:37 +08:00
Hongbo Wu
f541b8f8ca export at most 1000 most recent items 2024-02-22 20:22:37 +08:00
Hongbo Wu
6af6f8ac6c delete log 2024-02-22 15:40:09 +08:00
Hongbo Wu
f47f4c1c47 fix tests 2024-02-22 15:23:53 +08:00
Hongbo Wu
fb576a58a0 initial export 2024-02-22 15:20:24 +08:00
Hongbo Wu
f94267ee1a enqueue export job 2024-02-22 13:38:51 +08:00
Hongbo Wu
34db7bb2d8 create a job for exporting item to integrations 2024-02-22 11:55:51 +08:00
Hongbo Wu
649ccc692b log unknown job when getting priority 2024-02-22 09:23:32 +08:00
Hongbo Wu
ce19218e15 get queue priority from a table 2024-02-21 22:01:04 +08:00
Hongbo Wu
927394e07c fix: save url operation is delayed 2024-02-21 17:54:08 +08:00
Hongbo Wu
2fff9a5720
Merge pull request #3380 from omnivore-app/fix/recreate-item 
fix/recreate item
 2024-02-21 15:49:28 +08:00
Hongbo Wu
8f39985f0f fix deleted labels and highlights are still searchable 2024-02-21 15:06:04 +08:00
Hongbo Wu
51c1d33a09 fix tests 2024-02-21 14:55:49 +08:00
Hongbo Wu
af24856603 resolve conflicts after rebasing 2024-02-21 14:29:21 +08:00
Hongbo Wu
b1c45599f6 resolve conflicts after rebasing 2024-02-21 14:29:15 +08:00
Hongbo Wu
eb895b49bf delete labels and highlights attached to the recreated item if the item was deleted before 2024-02-21 14:18:06 +08:00
Hongbo Wu
abfa15e56f revert soft deletes all the labels and highlights attached to the deleted item 2024-02-21 14:16:21 +08:00
Hongbo Wu
38ee6c1331 merge labels and highlights when saving a duplicate item 2024-02-21 14:16:20 +08:00
Hongbo Wu
242e6e0fa3 delete all the labels and highlights attached to the item when item was deleted 2024-02-21 13:52:39 +08:00
Hongbo Wu
cd311d6454 do not update db options in queue-processor 2024-02-20 19:09:33 +08:00
Hongbo Wu
59d4d78f78 connection idle timeout in 10 seconds 2024-02-20 18:33:21 +08:00
Hongbo Wu
942b2374b8 get postgres connection pool size from env 2024-02-20 18:08:55 +08:00
Hongbo Wu
d0500f8524
Merge pull request #3542 from omnivore-app/fix/duplicate-key-violation 
fix/duplicate key violation
 2024-02-19 12:46:57 +08:00
Hongbo Wu
26d05a2e6e delete the new item if it is different from the existing one 2024-02-19 11:47:09 +08:00
Hongbo Wu
afd4d2a397 fix: duplicate key value violates unique constraint "library_item_pkey" by getting the item by user_id and md5(url) for update before upserting the item 2024-02-19 11:35:43 +08:00
Jackson Harper
9c493d29db More graceful shutdown for the api server 2024-02-19 11:25:22 +08:00
Jackson Harper
e0ce598be5 Move apollo prom init into apollo.ts file 2024-02-12 11:29:08 +08:00
Jackson Harper
a52e8f7295 Add service label 2024-02-10 17:21:28 +08:00
Jackson Harper
5c29a6c185 Add http metrics 2024-02-10 16:05:35 +08:00
Jackson Harper
545d618c11
Merge pull request #3524 from omnivore-app/feat/library-navigation-updates 
Updated library layout
 2024-02-09 14:17:01 +08:00
Jackson Harper
11463ba6f7 Colour clean ups 2024-02-08 15:01:40 +08:00
Hongbo Wu
5d201521e1
Merge pull request #3518 from omnivore-app/fix/library-item 
fix: duplicate key value violates unique constraint "library_item_pkey"
 2024-02-08 13:18:22 +08:00
Hongbo Wu
03f89d890e fix tests 2024-02-08 10:42:37 +08:00
Hongbo Wu
c1ba9b8017 fix tests 2024-02-08 10:15:05 +08:00
Hongbo Wu
86f4553dd1 fix: duplicate key value violates unique constraint "library_item_pkey" 2024-02-07 20:41:34 +08:00
Hongbo Wu
485caec803 append job version to the job id 2024-02-07 18:42:10 +08:00
Hongbo Wu
ef6c6349de add remove_on_complete and remove_on_fail to the unique job 2024-02-07 18:03:16 +08:00
Hongbo Wu
8c23cff17b remove limit on the sync reading progress job 2024-02-07 11:09:30 +08:00
Hongbo Wu
d7a81483cf resolve conflicts 2024-02-06 17:02:00 +08:00
Hongbo Wu
1b60694579 lint 2024-02-06 16:54:44 +08:00
Hongbo Wu
efd8119e02 reduce sql error 2024-02-06 16:53:32 +08:00
Hongbo Wu
7e022e3294 reduce find user query 2024-02-06 16:53:32 +08:00
Hongbo Wu
a37bfab348 update create article api 2024-02-06 16:53:32 +08:00