prehistoric-systems/omnivore
1
0
Fork 0
mirror of https://github.com/omnivore-app/omnivore.git synced 2026-03-11 08:54:26 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
12057 commits 201 branches 31 tags 432 MiB
Commit graph

195 commits

Author SHA1 Message Date
Hongbo Wu
ae504a970b save instances in redis as a sorted set 2023-06-23 12:17:41 +08:00
Hongbo Wu
4607754f58 reduce timeout to 20s 2023-06-23 11:36:30 +08:00
Hongbo Wu
6b203dde4c update instances pool to pick one from each country 2023-06-23 10:54:43 +08:00
Hongbo Wu
61542a34bd fix: create an instance pool to scrape nitter 2023-06-23 10:45:37 +08:00
Hongbo Wu
30b7c38e76 fix: increase twitter-handler timeout to 60 seconds 2023-06-22 21:16:55 +08:00
Hongbo Wu
12009d1d06 fix: tweet author not saved 2023-06-22 18:05:53 +08:00
Hongbo Wu
0a0d64ca4e
Merge pull request #2409 from omnivore-app/fix/nitter-scraper 
using axios to fetch html from nitter.net
 2023-06-22 15:01:28 +08:00
Hongbo Wu
3e2e16e915 skip running readability on scraped tweet 2023-06-22 15:00:54 +08:00
Hongbo Wu
0d7adcdecc using axios to fetch html from nitter.net 2023-06-22 14:32:18 +08:00
Jackson Harper
a0556055c5
Merge pull request #2389 from Podginator/feat/atlantic-handler 
Add The Atlantic handler to avoid paywall and correctly format
 2023-06-22 10:21:13 +08:00
Jackson Harper
de2ca58b37 Linting fix 2023-06-22 09:52:43 +08:00
Hongbo Wu
6aceadc819 fix attachment url 2023-06-21 17:46:05 +08:00
Hongbo Wu
f0e6458e4a filter out replies 2023-06-21 16:01:09 +08:00
Hongbo Wu
a3ad9375bf replace url in tweets 2023-06-21 15:50:55 +08:00
Hongbo Wu
d2094981ad increase timeout value to 60 seconds 2023-06-21 15:21:40 +08:00
Hongbo Wu
dde104b95e fix video attachment not saved 2023-06-21 15:07:21 +08:00
Hongbo Wu
a109d0383f go to the next thread page to scrape long thread 2023-06-20 19:17:34 +08:00
Hongbo Wu
89f89875d6 fix date parsing 2023-06-20 17:35:59 +08:00
Thomas Rogers
7293078068 Remove Audio section for Feature Articles 2023-06-20 10:35:04 +02:00
Thomas Rogers
44dd44dd58 update comments on tests 2023-06-20 10:26:53 +02:00
Thomas Rogers
c0f34270ee Remove Related content links from article (IE: Read article X) 2023-06-20 10:24:41 +02:00
Thomas Rogers
55fd02b67a Fix Tests 2023-06-20 09:20:43 +02:00
Hongbo Wu
ff9fcef08d disable twitter-handler 2023-06-20 14:59:14 +08:00
Hongbo Wu
65758f3469 feat: add nitter handler to scrape tweets from twitter.com and nitter.net 2023-06-20 14:56:54 +08:00
Jackson Harper
8e13284795 Linting fixes 2023-06-20 13:48:18 +08:00
Thomas Rogers
0cae728a43 Add missing files 2023-06-20 02:42:09 +02:00
Thomas Rogers
a08af0610a Add The Atlantic handler to avoid paywall and correctly format 2023-06-20 02:38:24 +02:00
Hongbo Wu
fb2c70a6fd change the page viewport when scraping twitter thread 2023-06-15 18:33:26 +08:00
Hongbo Wu
d69e732579 fix: dedupe url if failed to save 2023-06-01 16:11:35 +08:00
Hongbo Wu
8ddde61285 fix: increase twitter-handler timeout to 60 seconds 2023-06-01 16:11:35 +08:00
Hongbo Wu
4d422402f8 fix: convert published time to GMT +8 in the parsed wechat blogs 2023-05-24 10:55:54 +08:00
Hongbo Wu
94aead9988 Add content handler for weixin.qq.com 2023-05-16 14:46:44 +08:00
Jackson Harper
89c85e5627
Merge pull request #2109 from omnivore-app/feat/tweet-titles 
Better tweet titles
 2023-04-25 14:17:33 +08:00
Jackson Harper
6560a6e694 Add type definitions 2023-04-25 12:23:37 +08:00
Jackson Harper
155fad5b9c Truncate tweet text used in titles using lodash 2023-04-25 11:39:55 +08:00
Jackson Harper
53e81cc06e Better formatting of tweet titles 2023-04-25 09:47:52 +08:00
Hongbo Wu
187ad654e7 Add debug logs 2023-04-22 12:29:46 +08:00
Hongbo Wu
2760ce60f0 Fix list-unsubscribe header undefined 2023-04-22 11:07:15 +08:00
Hongbo Wu
89ac9c46cc Remove other places that use rfc2047 decoder 2023-04-20 20:53:06 +08:00
Hongbo Wu
06733227d6 Remove new "restack" and "share" buttons from SubStack newsletters 2023-04-18 14:53:27 +08:00
Jackson Harper
6c27c5385f Remove footnote links in wikipedia articles 2023-04-13 12:10:07 +08:00
Jackson Harper
e03301e294
Merge pull request #2026 from omnivore-app/fix/embedded-quote-tweets 
Handle embedded quote tweets
 2023-04-12 16:23:38 +08:00
Jackson Harper
c366cd6472 Fix test value, there is only one quote tweet 2023-04-12 15:37:26 +08:00
Jackson Harper
ae45067a32 Add test data file 2023-04-12 15:16:11 +08:00
Jackson Harper
344514bd0f Handle embedded quote tweets 2023-04-12 14:50:20 +08:00
Hongbo Wu
4969391d4c Replace picture with image 2023-04-12 12:15:53 +08:00
Hongbo Wu
eb58bf11ba Force to use content handler of piped.video when saving from extensions 2023-04-10 20:52:09 +08:00
Hongbo Wu
bfbd80e8f1 Get the published date 2023-04-06 16:41:21 +08:00
Hongbo Wu
70f4068d89 Add handler for piped video 2023-04-06 16:31:06 +08:00
Hongbo Wu
29776359f2 Fix a typo in the url 2023-03-10 18:27:58 +08:00
Hongbo Wu
687771f676 Use YouTube as site name 2023-03-10 18:26:21 +08:00
Hongbo Wu
56ce70dbd4 Revert using youtube data api 2023-03-10 18:21:51 +08:00
Hongbo Wu
98d5f92ad4 Call youtube data api to fetch published date 2023-03-10 17:38:03 +08:00
Hongbo Wu
a0aab94e8b Add site_name and published date to saved twitter 2023-03-10 16:42:40 +08:00
Hongbo Wu
de9524c3e8 Fix confusing unsubscribing url with email address 2023-02-27 12:06:46 +08:00
Hongbo Wu
729b8159f9
Merge pull request #1818 from omnivore-app/fix/energy-world-newsletter 
Preparse the newsletter content
 2023-02-17 22:41:20 +08:00
Hongbo Wu
31eb67efb1 Fix sometimes there is extra spaces between sender and email address by checking only the email address 2023-02-17 22:05:35 +08:00
Hongbo Wu
7f761049d7 Preparse the newsletter content 2023-02-17 09:49:19 +08:00
Hongbo Wu
cc8b1cefdb Preserve <pre> elements with prism- class and identity them as code blocks 2023-02-14 12:33:59 +08:00
Hongbo Wu
db1a194d11 Add support for india times newsletter 2023-02-03 18:18:44 +08:00
Hongbo Wu
d36b608919 Add support for energy-world newsletter 2023-02-03 17:23:51 +08:00
Hongbo Wu
b6c41b9c73 Add support for every.io newsletter 2023-02-03 16:59:27 +08:00
Hongbo Wu
363489f5be Add generic newsletter handler 2023-02-03 14:46:24 +08:00
Hongbo Wu
39b87e2aaf Add support for Time To Sign Off newsletter 2023-02-02 17:01:25 +08:00
Hongbo Wu
ba8b6796f3
Merge pull request #1746 from omnivore-app/fix/save-youtube-playlist 
Fix incorrect Youtube playlist author name and description
 2023-01-31 11:06:25 +08:00
Hongbo Wu
61bf89b9ad Add tests 2023-01-30 19:10:27 +08:00
Hongbo Wu
4f12580b1c Use email header x-beehiiv-type to detect beehiiv newsletter and get the newsletter url from the header x-newsletter 2023-01-30 19:03:15 +08:00
Hongbo Wu
88f6e20bfc Pass email headers to the content-handler 2023-01-30 18:52:38 +08:00
Hongbo Wu
ec7344b923 Get playlist id from the URL and retrieve the information of the playlist 2023-01-30 16:53:19 +08:00
Hongbo Wu
1b3d4bbffa Do not escape the title of youtube video or tweet which is saved in elastic 2023-01-18 19:08:36 +08:00
Jackson Harper
6133d0865a Set title for YouTube based on oembed data 2023-01-16 20:03:23 +08:00
Hongbo Wu
145a8b9730 Fix last two tweets not saved in a long thread by waiting for document loaded after scrolling 2023-01-12 22:03:29 +08:00
Hongbo Wu
1243e2ace2 Fix the error when no name in subscription by using the email address as the author when no author found in the newsletter 2023-01-10 11:17:22 +08:00
Hongbo Wu
ab16447c14 Fix not saving all the tweets in a thread 2023-01-03 21:26:34 +08:00
Hongbo Wu
3a3a06b5d6 Fix vote count 2022-11-29 16:18:09 +08:00
Hongbo Wu
1fe86b6230 Parse owner 2022-11-29 15:48:45 +08:00
Hongbo Wu
f8e0efad05 fix bug 2022-11-29 15:40:22 +08:00
Hongbo Wu
8ee19a98fa parse authors 2022-11-29 12:26:37 +08:00
Hongbo Wu
c993fa11be Add stack-overflow content-handler 2022-11-29 11:39:23 +08:00
Jackson Harper
db4f225b5c
Merge pull request #1426 from omnivore-app/fix/github-handler-titles 
Use greedy matching on the GitHub title regex
 2022-11-17 13:42:23 +08:00
Hongbo Wu
c18217e697 Fix logRecord being shared by requests 2022-11-17 11:28:46 +08:00
Hongbo Wu
d9237723c5 Share puppeteer browser instance between content handlers 2022-11-17 11:28:46 +08:00
Hongbo Wu
e866541ae1 Fix puppeteer launch in head mode 2022-11-17 11:28:46 +08:00
Hongbo Wu
8bdbd86e97 Make headless = false in local testing 2022-11-17 11:27:51 +08:00
Jackson Harper
0d83ecb068 Use greedy matching on the GitHub title regex 
This fixes issues with titles that have slashes (/) in their title
as the regex match would look for the last possible matching
slash.
 2022-11-16 13:52:58 +08:00
Hongbo Wu
c8febe900f Fix prettier error 2022-11-16 10:34:52 +08:00
Jackson Harper
726f69484f Better content extraction of title and description for GitHub URLs 2022-11-15 18:43:55 +08:00
Hongbo Wu
9d753e3c0e Forward convertkit confirmation email 2022-11-02 15:27:30 +08:00
Hongbo Wu
ae3ff6f34b Add test case 2022-11-01 17:48:39 +08:00
Hongbo Wu
67082a478e Add support for hey world newsletters 2022-11-01 16:41:42 +08:00
Jackson Harper
8712a3efef
Merge pull request #1373 from omnivore-app/fix/substack-embedded-tweets 
Handle embedded tweets in substack emails
 2022-11-01 15:04:03 +08:00
Jackson Harper
e1d7eeed01 Make test verify preparse is true 2022-11-01 10:46:31 +08:00
Jackson Harper
9b29a3c261 querySelectorAll returns a NodeList which we can iterate 2022-11-01 10:34:18 +08:00
Jackson Harper
392ac15a62 Treat all tweets as elements so we dont need to cast 2022-11-01 10:31:12 +08:00
Jackson Harper
a4f0698974 Handle older style substack image embeds as well as new 2022-10-31 21:57:29 +08:00
Jackson Harper
734c4337df Add missing test file 2022-10-31 21:51:22 +08:00
Jackson Harper
2622437831 Fix linting 2022-10-31 21:42:31 +08:00
Jackson Harper
cc91e43572 Handle embedded tweets in substack emails 
This does a few things:
- tags static tweets found in substack emails with a special class
- upgrades readability to ignore special class names
- reduces some readability debug output
 2022-10-31 21:28:36 +08:00
Hongbo Wu
a7b208fd26 Skip non-author replies in twitter thread 2022-10-31 16:51:20 +08:00
Hongbo Wu
1067747cc2 Add twitter author when saved 2022-10-27 19:02:16 +08:00