prehistoric-systems/omnivore
1
0
Fork 0
mirror of https://github.com/omnivore-app/omnivore.git synced 2026-03-11 08:54:26 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
12057 commits 201 branches 31 tags 432 MiB
Commit graph

2536 commits

Author SHA1 Message Date
Hongbo Wu
0dada2d708 fix: digest score api timeout = 5 seconds 2024-06-28 16:38:00 +08:00
Hongbo Wu
a7ba02e063
Merge pull request #4117 from omnivore-app/fix/use-read-replica 
fix/use read replica for RLS enabled transactions
 2024-06-28 15:13:26 +08:00
Hongbo Wu
a06f1a243c fix tests 2024-06-28 12:37:45 +08:00
Hongbo Wu
a3ba8573b7 fix: validate ownership of library item before assigning labels to the item 2024-06-28 11:53:20 +08:00
Hongbo Wu
3ad826392e use digest-score service 2024-06-27 20:55:04 +08:00
Hongbo Wu
822f249fb1 cont 2024-06-27 19:10:50 +08:00
Hongbo Wu
2ed480bb49 use read replica for selecting library items, labels and highlights 2024-06-27 19:02:15 +08:00
Hongbo Wu
fb3d2eada6 use primary and replica instead of master and slave 2024-06-27 18:42:42 +08:00
Hongbo Wu
0830f0b312 allow defining replica mode in the auth transaction 2024-06-27 18:38:43 +08:00
Hongbo Wu
17eb5efa37 resolve conflicts 2024-06-27 17:16:17 +08:00
Hongbo Wu
45986184c4 tidy 2024-06-27 17:12:45 +08:00
Hongbo Wu
1168d12018 use read replica for raw query 2024-06-27 17:10:58 +08:00
Hongbo Wu
84953198fe
Merge pull request #4114 from omnivore-app/fix/read-replica 
fix/read replica
 2024-06-26 17:38:53 +08:00
Hongbo Wu
329dd13ed2 return empty array if no candidates for home feed selection 2024-06-26 17:19:08 +08:00
Hongbo Wu
a6ab96155d by default use info level for loggers 2024-06-26 16:19:13 +08:00
Hongbo Wu
818ad5a0bb log isReplicated for each query 2024-06-26 16:11:52 +08:00
Hongbo Wu
bc6a09798b fix read replica not being used 2024-06-26 16:11:34 +08:00
Hongbo Wu
57e0002cae rename slave to replica in db config 2024-06-26 11:34:45 +08:00
Hongbo Wu
9f16e5fede add "canMove" field to the HomeItem type in graphql api which indicates if the item is in following folder 2024-06-25 16:40:12 +08:00
Hongbo Wu
a1cd611fa7 use read replica for "read" 2024-06-21 14:39:40 +08:00
Hongbo Wu
b0d6876e56 add replication config 2024-06-21 13:31:50 +08:00
Hongbo Wu
14a375db83 use real scoring service 2024-06-21 12:01:34 +08:00
Hongbo Wu
977a1a90e5 Skip scoring RSS items for now 2024-06-21 12:00:53 +08:00
Hongbo Wu
8c84fc58b8 add more subscription features to the score api payload 2024-06-20 17:36:57 +08:00
Hongbo Wu
40182421f7 show archived highlights 2024-06-20 16:31:05 +08:00
Hongbo Wu
2e26a04cd0 remove redundant context type 2024-06-20 15:47:52 +08:00
Jackson Harper
4ab0497034 Get uid from claims 2024-06-19 18:09:00 +08:00
Jackson Harper
54f608d23c Use context uid 2024-06-19 17:53:08 +08:00
Jackson Harper
e2e0774e7e Add user field to the ArticleSavingRequest resolver 2024-06-19 17:44:44 +08:00
Hongbo Wu
6df8c71035 reduce bullmq worker concurrency to 2 2024-06-18 17:28:19 +08:00
Hongbo Wu
2ed63af9a4 stub score client 2024-06-18 12:10:03 +08:00
Hongbo Wu
47c6488834 do not update library item score in database for now 2024-06-18 11:25:44 +08:00
Hongbo Wu
a26e8a3972
Merge pull request #4041 from omnivore-app/feature/folder-policy-api 
feat: folder expiration policy api
 2024-06-14 14:44:21 +08:00
Hongbo Wu
c0947ddd48 fix batch delete query does not use index 2024-06-14 12:21:20 +08:00
Hongbo Wu
aea1650626 do not show move/archived/deleted item in home feed 2024-06-14 12:15:17 +08:00
Hongbo Wu
be8185d79c
Merge pull request #4059 from omnivore-app/fix/rss 
fix: fetch old items for existing subscribed feeds
 2024-06-14 09:04:37 +08:00
Hongbo Wu
31f04344fc resolve rebase conflicts 2024-06-13 18:41:35 +08:00
Hongbo Wu
88e3d648c8 add a REST API to trigger the expire folder job 2024-06-13 18:36:33 +08:00
Hongbo Wu
6f496b9336 add folder expiration jobs 2024-06-13 18:30:27 +08:00
Hongbo Wu
12fa6ca0b4 add delete folder policy api 2024-06-13 18:29:21 +08:00
Hongbo Wu
6138e078b1 add update folder policy api 2024-06-13 18:29:21 +08:00
Hongbo Wu
2d757a4896 add create folder policy api 2024-06-13 18:29:21 +08:00
Hongbo Wu
7940c3f4ea add list folder policy api 2024-06-13 18:29:21 +08:00
Hongbo Wu
25ca4c7791 create folder_policy table 2024-06-13 18:29:21 +08:00
Hongbo Wu
e3d70fc27d
Merge pull request #4052 from omnivore-app/feature/prune-trash-items 
feature/prune trash items
 2024-06-13 17:58:18 +08:00
Hongbo Wu
9ef91ff1ca add an REST API to trigger the prune trash job 2024-06-13 14:48:14 +08:00
Hongbo Wu
c82efb31d3 fetch old items for existing subscribed feeds 2024-06-13 14:15:19 +08:00
Jackson Harper
c7b1794591 Implement more of Recently Added section 2024-06-12 21:00:14 +08:00
Jackson Harper
e9f3009111
Merge pull request #4040 from omnivore-app/feat/web-layout-navigation 
Update layout navigation with left menu
 2024-06-12 20:02:42 +08:00
Hongbo Wu
71d07caf94 run prune trash job at 3am UTC daily 2024-06-12 19:27:42 +08:00
Hongbo Wu
46aa405552 remove timeout in highlight generator 2024-06-12 19:13:09 +08:00
Hongbo Wu
f29592d671 worker can concurrently process 10 jobs 2024-06-12 18:40:08 +08:00
Hongbo Wu
a580fd7ebf catch sync reading progress query error 2024-06-12 18:40:07 +08:00
Hongbo Wu
fc5b152c43 update job name 2024-06-12 18:39:27 +08:00
Hongbo Wu
755f483ba9 create a function to batch delete items in trash 2024-06-12 18:38:14 +08:00
Jackson Harper
5f817bc336 Null handler as we know URL must be set for RSS subscriptions 2024-06-12 14:00:45 +08:00
Hongbo Wu
a5295feb91 remove console timer logs 2024-06-12 13:37:23 +08:00
Hongbo Wu
57695c370c monitor each job latency in prometheus 2024-06-12 13:35:04 +08:00
Hongbo Wu
f052ab55a0 monitor home feed job latency in prometheus 2024-06-12 13:01:36 +08:00
Hongbo Wu
bc7e710835 open youtube transcript to public 2024-06-11 10:55:56 +08:00
Jackson Harper
3f6e7c924a Work on new left menu w/shortcuts 2024-06-10 17:34:59 +08:00
Hongbo Wu
470413b704 add basic highlights web UI 2024-06-08 10:50:44 +08:00
Hongbo Wu
c826e034ab
Merge pull request #4034 from omnivore-app/fix/home-item-thumbnail 
fix: home item thumbnail not using proxy url
 2024-06-07 13:41:15 +08:00
Hongbo Wu
e05985b2e4 fix typo 2024-06-07 13:40:54 +08:00
Hongbo Wu
0db1108675 replace continuous spaces and new lines with space in preview content 2024-06-07 13:29:38 +08:00
Hongbo Wu
76383d8a33 feat: generate preview content if excerpt less than 180 chars 2024-06-06 19:08:41 +08:00
Hongbo Wu
c48013ed4b fix: home item thumbnail not using proxy url 2024-06-06 17:50:18 +08:00
Hongbo Wu
70bc136d15 batch get labels from highlight id 2024-06-06 17:26:01 +08:00
Hongbo Wu
24457d9036 search highlights by labels 2024-06-06 16:12:07 +08:00
Hongbo Wu
51efff8b8e allow returning library item information in highlight 2024-06-06 15:23:56 +08:00
Hongbo Wu
9259a9cfe3 fix tests 2024-06-06 14:53:14 +08:00
Hongbo Wu
166b324d63 add tests 2024-06-06 13:26:13 +08:00
Hongbo Wu
c620ef38a2 clean up helpers 2024-06-06 12:08:15 +08:00
Hongbo Wu
0013150c26 feat: highlights api 2024-06-05 20:44:18 +08:00
Hongbo Wu
1df84f7a42
Merge pull request #4022 from omnivore-app/fix/notion 
fix: always sync Last Updated in Notion
 2024-06-03 13:01:49 +08:00
Hongbo Wu
e1236dbe29 always sync Last Updated 2024-06-02 09:56:17 +08:00
Hongbo Wu
8eac9375a2 feat: add a hidden section in home feed and load on demand 2024-05-31 16:34:32 +08:00
Hongbo Wu
e932777347 fix ranking order 2024-05-31 12:58:01 +08:00
Hongbo Wu
a6d27bea8d rename for you to top picks 2024-05-31 12:23:11 +08:00
Hongbo Wu
aa4e091c4b exclude just added items from for you and quick links section 2024-05-31 12:00:30 +08:00
Hongbo Wu
1af6cfa549 rename long section to for you 2024-05-31 11:51:30 +08:00
Hongbo Wu
633a022825 add client side pagination 2024-05-30 17:36:18 +08:00
Hongbo Wu
d349c1867f add section title 2024-05-30 15:48:57 +08:00
Hongbo Wu
8dfd876392 limit to 5 just added items 2024-05-30 15:48:28 +08:00
Hongbo Wu
bc4951f376 do not fail score job if failed to enqueue updating home job 2024-05-30 15:39:04 +08:00
Hongbo Wu
3c27125645 trigger update home job once a new item is scored 2024-05-30 14:34:08 +08:00
Hongbo Wu
4bb33da27f add just added section to home 2024-05-30 13:43:12 +08:00
Hongbo Wu
5739ec2843 only score candidates without scores 2024-05-29 18:47:22 +08:00
Hongbo Wu
84a6530743 log job error 2024-05-29 18:29:48 +08:00
Hongbo Wu
c7de24e9f8 do not rank candidates and just return them all if less than 10 candidates 2024-05-29 18:22:16 +08:00
Hongbo Wu
d1d7f93fc3 time each steps in the update home job 2024-05-29 17:58:28 +08:00
Hongbo Wu
69dbbd7635 add score api url to env var 2024-05-29 17:45:51 +08:00
Hongbo Wu
b7e6888395 remove debugging logs 2024-05-29 17:42:30 +08:00
Hongbo Wu
69c555b8c3
Merge pull request #4005 from omnivore-app/feature/refresh-home-api 
feature/refresh home api
 2024-05-29 17:37:41 +08:00
Hongbo Wu
b6811ba957 delete home cache before recreating 2024-05-29 17:33:01 +08:00
Hongbo Wu
91c2ae8d97 add refresh home api 2024-05-29 17:24:37 +08:00
Hongbo Wu
a9ebcd33e4 add mark as seen to the bulk action 2024-05-29 17:07:19 +08:00
Hongbo Wu
f496664025 remove expired items before remove extra items 2024-05-29 16:42:16 +08:00
Jackson Harper
7466501c95 Parse the single score from the digest-score service 2024-05-29 16:40:44 +08:00
Hongbo Wu
9280703047
Merge pull request #4003 from omnivore-app/fix/add-score-to-item 
fix/add score to item
 2024-05-29 15:19:43 +08:00
Hongbo Wu
292d6b8149
Merge pull request #4001 from omnivore-app/feature/get-subscription-api 
feat: create an api to get subscription by id
 2024-05-29 15:17:31 +08:00
Hongbo Wu
8e8db02633 add score to search item 2024-05-29 15:07:22 +08:00
Hongbo Wu
974c910846 add score to home item 2024-05-29 15:02:10 +08:00
Hongbo Wu
7a34a03bdd fix subscription and public item orders too 2024-05-29 11:30:19 +08:00
Hongbo Wu
e66a1374be remove argument directive 2024-05-29 11:02:35 +08:00
Hongbo Wu
95539948ff feat: create an api to get subscription by id 2024-05-29 10:14:48 +08:00
Hongbo Wu
3b69f0ce0c make slug optional 2024-05-29 09:15:15 +08:00
Hongbo Wu
d04626a02c add slug 2024-05-28 21:59:37 +08:00
Hongbo Wu
9dff961bba fix null checking 2024-05-28 21:44:55 +08:00
Hongbo Wu
5257d95559 fix: home items order is not accurate 2024-05-28 21:29:35 +08:00
Hongbo Wu
4d5a26de2c add start and end cursor to the pageinfo 2024-05-28 17:08:13 +08:00
Hongbo Wu
2e57ceba3a fix removing new items from redis 2024-05-28 17:07:59 +08:00
Hongbo Wu
3d262a4d1f remove fake code 2024-05-28 15:44:53 +08:00
Hongbo Wu
a3d20c0395 add subscription to the feature 2024-05-28 15:05:44 +08:00
Hongbo Wu
6b835f0872 batch load sources 2024-05-28 15:02:22 +08:00
Hongbo Wu
e131afc183 update API 2024-05-28 14:45:33 +08:00
Hongbo Wu
88dcdbfb2d update score api client 2024-05-27 21:29:15 +08:00
Hongbo Wu
1c07e41efd update comment 2024-05-27 21:20:49 +08:00
Hongbo Wu
57b28c39bf store topic as a ltree 2024-05-27 21:17:54 +08:00
Hongbo Wu
efb7160373 rename just read feed to home 2024-05-27 21:11:08 +08:00
Hongbo Wu
5b9e34739d skip auth token for score api for now 2024-05-27 17:59:04 +08:00
Hongbo Wu
b24d8ca413 show long read items before quick links 2024-05-27 17:55:19 +08:00
Hongbo Wu
f603c84b27 store only feed item id and type in redis and fetch the details from DB when pagination API is called 2024-05-27 17:38:36 +08:00
Hongbo Wu
87ad71fbe9 add published_at to the score api request 2024-05-27 12:56:38 +08:00
Hongbo Wu
88c25844b4 set priority = 10 for score job 2024-05-27 12:50:57 +08:00
Hongbo Wu
fff3b4b523 add authorization to the score api client 2024-05-27 11:58:51 +08:00
Hongbo Wu
8f0bebf381 use pre-calculated score from library item when ranking 2024-05-27 11:55:44 +08:00
Hongbo Wu
41b9fef316 score library item after saved 2024-05-27 11:45:51 +08:00
Hongbo Wu
3cb93283b0 add score library item job 2024-05-27 11:37:59 +08:00
Hongbo Wu
f6424c3bd2 add score to library_item entity 2024-05-27 11:12:07 +08:00
Hongbo Wu
25a984dc6a check for existing job before enqueueing 2024-05-26 14:48:30 +08:00
Hongbo Wu
67dac9ba31 add public item source 2024-05-26 14:41:38 +08:00
Hongbo Wu
0a8cc13ac5 add mixing algorithm 2024-05-26 14:31:22 +08:00
Hongbo Wu
1ba6221bc4 add pagination 2024-05-25 11:47:45 +08:00
Hongbo Wu
880a907679 add public item stats 2024-05-24 15:48:24 +08:00
Hongbo Wu
6c9f0c81f6 convert language to code 2024-05-24 15:41:20 +08:00
Hongbo Wu
f2d2e852e1 grant db permission 2024-05-24 15:32:14 +08:00
Hongbo Wu
a48a614676 save just read feed in redis sorted set 2024-05-24 14:52:21 +08:00
Hongbo Wu
cb4fc23507 implement justReadFeed API 2024-05-23 22:07:19 +08:00
Hongbo Wu
b058952c2d implement updateJustReadFeed job 2024-05-23 21:10:01 +08:00
Hongbo Wu
06b89a88b3 update candidates selection 2024-05-23 15:48:30 +08:00
Hongbo Wu
43ebdc30a1 update just read api 2024-05-23 11:36:01 +08:00
Hongbo Wu
b6c150a5cb
Merge pull request #3966 from omnivore-app/feature/add-index-for-archived-at 
feature/add index for archived at and folder
 2024-05-20 12:08:27 +08:00
Hongbo Wu
a3fe526c2c remove debugging timers 2024-05-20 11:26:06 +08:00
Hongbo Wu
eaf5632d0a profile query time 2024-05-19 10:17:50 +08:00
Hongbo Wu
913215ab09 fix archived query 2024-05-18 22:05:36 +08:00
Hongbo Wu
be9e826446 use IN query for state 2024-05-18 21:24:56 +08:00
Hongbo Wu
6f2aa2e0cd add more logs 2024-05-17 17:19:55 +08:00
Hongbo Wu
5bf9d29967 enqueue uploading original content after saving page 2024-05-17 17:00:37 +08:00
Hongbo Wu
52ebf466e3 get content from cache first when saving url 2024-05-17 16:46:54 +08:00
Hongbo Wu
9769eab5dc fix youtube transcript 2024-05-16 11:23:36 +08:00
Hongbo Wu
8cfa24a847 allow downloading/uploading readable content 2024-05-15 21:46:22 +08:00
Hongbo Wu
1a973faadc make author optional 2024-05-15 18:12:54 +08:00
Hongbo Wu
396ada28b8 pre-cache thumbnail 2024-05-15 17:49:31 +08:00
Hongbo Wu
9f1babf787 select used columns 2024-05-15 17:42:53 +08:00
Hongbo Wu
505e3666b6 fix: add author to the chapter in digest 2024-05-15 17:38:24 +08:00
Hongbo Wu
6d0edbd61a
Merge pull request #3839 from omnivore-app/feature/upload-original-content 
feature/upload original content
 2024-05-15 16:47:46 +08:00
Hongbo Wu
950b42899d fix mock cloud storage 2024-05-15 15:53:42 +08:00
Jackson Harper
95796607d0
Merge pull request #3956 from omnivore-app/fix/update-openai 
Upgrade to gpt-4o
 2024-05-15 12:57:46 +08:00
Jackson Harper
74bf05cf06 Linting 2024-05-15 12:41:12 +08:00
Jackson Harper
f217c6f4ad Cast digestConfig to any to serialize properly 2024-05-15 12:08:29 +08:00
Jackson Harper
c6bf64a648 Upgrade to gpt-4o 2024-05-15 11:44:23 +08:00
Jackson Harper
89fb6255a5
Merge pull request #3952 from omnivore-app/fix/ios-digest-new-users 
Allow opt into digest on the library view
 2024-05-15 11:10:38 +08:00
Hongbo Wu
cca7a21884 fix tests 2024-05-15 11:02:18 +08:00
Jackson Harper
2cb1b8ed1e Add an extra test for clearing the user personalization data 2024-05-15 10:35:58 +08:00
Hongbo Wu
69608119a8 allow preflight api calls to the digest POST api 2024-05-15 09:48:42 +08:00
Jackson Harper
bbf3de1f86 Transform digest config to a JSON string before setting 2024-05-15 00:06:04 +08:00
Hongbo Wu
b7886b8d25 fix tests 2024-05-14 22:47:10 +08:00
Hongbo Wu
9dee510be1 fix rss 2024-05-14 20:18:18 +08:00
Jackson Harper
3c24dc2e78 GQL requires this be an input object 2024-05-14 18:01:18 +08:00
Hongbo Wu
dbd7b7932f cont 2024-05-14 17:23:56 +08:00
Hongbo Wu
d796503fec resolve rebase conflicts 2024-05-14 17:21:50 +08:00
Hongbo Wu
cce5f2463d still use redis for cache 2024-05-14 17:16:26 +08:00
Hongbo Wu
80968472d8 fix bug 2024-05-14 17:15:52 +08:00
Hongbo Wu
eddf9206d0 do not store original content in db 2024-05-14 17:15:51 +08:00
Hongbo Wu
6cfb06c226 normalize file url 2024-05-14 17:14:41 +08:00
Hongbo Wu
5bd157ca25 hash url as the key 2024-05-14 17:14:41 +08:00
Hongbo Wu
7a0b2f3d33 upload file only not exists 2024-05-14 17:14:41 +08:00
Hongbo Wu
9286174ec7 upload and download original content from GCS 2024-05-14 17:14:40 +08:00
Hongbo Wu
bc1c48da4b fix failed to get thumbnail from rss feed 2024-05-14 17:14:02 +08:00
Jackson Harper
c3c6083239 Use DigestConfig type when setting user personalization 2024-05-14 17:03:09 +08:00
Jackson Harper
2b9851cbdb Make digestconfig a proper type so Swift can interact with it 2024-05-14 16:49:16 +08:00
Hongbo Wu
cbba54111f
Merge pull request #3947 from omnivore-app/fix/content-api 
fix/content api
 2024-05-13 19:14:35 +08:00
Hongbo Wu
302efc119e fix tests 2024-05-13 19:13:55 +08:00
Hongbo Wu
0a96057bda fix: text not highlighted in highlightedMarkdown content 2024-05-13 18:35:24 +08:00
Jackson Harper
3514595d83
Merge pull request #3937 from omnivore-app/fix/digest-eligibility 
Fix logic on the digest eligibility test
 2024-05-13 16:29:00 +08:00
Jackson Harper
5dbed8c6c2
Merge pull request #3943 from omnivore-app/feat/api-digest-label 
Add a label to saved digests
 2024-05-13 16:28:45 +08:00
Hongbo Wu
e37abb47a1
Merge pull request #3942 from omnivore-app/fix/puppeteer-timeout 
fix: puppeteer protocol timeout
 2024-05-13 16:01:44 +08:00
Hongbo Wu
64e4258d89
Merge pull request #3935 from omnivore-app/feature/get-content-api 
feature/get content api
 2024-05-13 15:27:34 +08:00
Jackson Harper
92e3f23ec6 Add a label to saved digests 2024-05-13 15:16:30 +08:00
Hongbo Wu
34c214e7d8 fix docker error 2024-05-13 15:11:37 +08:00
Hongbo Wu
6bb81dd5c3 skip uploading if file already exists 2024-05-11 11:30:41 +08:00
Jackson Harper
7e5b0864b3 Fix logic on the digest eligibility test 2024-05-10 22:50:22 +08:00
Hongbo Wu
6f7c58289a fix cors 2024-05-10 17:48:47 +08:00
Hongbo Wu
8034e18825 fix tests 2024-05-10 16:40:55 +08:00
Hongbo Wu
6e7a436cea POST content API 2024-05-10 16:26:42 +08:00
Hongbo Wu
0f184c4c21 add get content api 2024-05-10 15:43:15 +08:00
Jackson Harper
e96d07fb40 Clean up digest email formatting 2024-05-10 15:09:45 +08:00
Jackson Harper
e749c8dc3a Add authtrx when checking user digest eligibility 2024-05-10 14:52:23 +08:00
Jackson Harper
500ec889c8 Add block around var definition 2024-05-10 14:48:27 +08:00