prehistoric-systems/omnivore
1
0
Fork 0
mirror of https://github.com/omnivore-app/omnivore.git synced 2026-03-11 08:54:26 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
3439 commits 201 branches 31 tags 432 MiB
Commit graph

58 commits

Author SHA1 Message Date
Hongbo Wu
e52013ccb1 It seems to have some issue with disabling puppeteer timeout by setting it to 0. 
So I set the timeout to 2 minutes which should be enough and it works in my local env
 2022-08-26 11:54:33 +08:00
dependabot[bot]
d12f3642e6
Bump puppeteer-core from 15.3.2 to 16.1.0 
Bumps [puppeteer-core](https://github.com/puppeteer/puppeteer) from 15.3.2 to 16.1.0.
- [Release notes](https://github.com/puppeteer/puppeteer/releases)
- [Changelog](https://github.com/puppeteer/puppeteer/blob/main/CHANGELOG.md)
- [Commits](https://github.com/puppeteer/puppeteer/compare/v15.3.2...v16.1.0)

---
updated-dependencies:
- dependency-name: puppeteer-core
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-08-11 02:15:16 +00:00
Jackson Harper
f17ee64676 Use ScrapingBee for some hosts 2022-07-16 14:09:45 -07:00
Hongbo Wu
11f20ab64a Revert "close browser when request finished" 
This reverts commit 7e68ad5237.
 2022-07-15 15:35:07 +08:00
Hongbo Wu
7e68ad5237 close browser when request finished 2022-07-15 15:23:02 +08:00
Hongbo Wu
b2238ce7f2 revert no-sandbox 2022-07-15 14:43:17 +08:00
Hongbo Wu
ed09d78980 remove no-sandbox 2022-07-15 14:32:02 +08:00
Hongbo Wu
d9bb664fc0 remove puppeteer dependency in docker 2022-07-15 14:15:31 +08:00
Hongbo Wu
9191f5710c remove single-process arg 2022-07-15 14:04:41 +08:00
Hongbo Wu
4929bae81b close context if encounter error 2022-07-15 11:49:36 +08:00
Hongbo Wu
610c790a7e do not use puppeteer-extra plugin 2022-07-15 11:09:57 +08:00
dependabot[bot]
bb7ea78e8f
Bump puppeteer-core from 13.7.0 to 15.3.2 
Bumps [puppeteer-core](https://github.com/puppeteer/puppeteer) from 13.7.0 to 15.3.2.
- [Release notes](https://github.com/puppeteer/puppeteer/releases)
- [Changelog](https://github.com/puppeteer/puppeteer/blob/main/CHANGELOG.md)
- [Commits](https://github.com/puppeteer/puppeteer/compare/v13.7.0...v15.3.2)

---
updated-dependencies:
- dependency-name: puppeteer-core
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-07-11 21:35:51 +00:00
Jackson Harper
c94d5db259
Merge pull request #889 from omnivore-app/dependabot/npm_and_yarn/axios-0.27.2 
Bump axios from 0.26.0 to 0.27.2
 2022-07-08 13:48:27 -07:00
Jackson Harper
01353add63 Shorten the timeout requesting pages 
I believe our process is sometimes being terminated before this
timeout is hit, which means we then don't have time to fetch
with a fallback.
 2022-07-05 11:16:11 -07:00
Jackson Harper
9554f8f6ba Create a scrapingbee url when using the fallback 
Javascript hoists variables to the top of scope, so `url` here
refers to the `url` variable defined lower in the block.
 2022-07-05 08:41:34 -07:00
Hongbo Wu
3a79710dbf Always fall back to scrapingbee if there is an exception 2022-07-05 21:48:58 +08:00
Hongbo Wu
37075f076e Remove userDataDir 2022-06-29 22:56:14 +08:00
dependabot[bot]
e91f25e58c
Bump axios from 0.26.0 to 0.27.2 
Bumps [axios](https://github.com/axios/axios) from 0.26.0 to 0.27.2.
- [Release notes](https://github.com/axios/axios/releases)
- [Changelog](https://github.com/axios/axios/blob/v0.27.2/CHANGELOG.md)
- [Commits](https://github.com/axios/axios/compare/v0.26.0...v0.27.2)

---
updated-dependencies:
- dependency-name: axios
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-06-27 21:31:14 +00:00
Jackson Harper
feb197c731 Fix a crash when parsing content fetches that are blocked 2022-06-22 14:55:37 -07:00
Hongbo Wu
a9b3a5c925
Merge pull request #805 from omnivore-app/fix/duplicate-content 
Remove duplicate content
 2022-06-19 10:56:10 +08:00
Jackson Harper
1d99bfaa10 Use a different Dockerfile for content-fetch with App Engine and docker-compose 2022-06-17 17:12:33 -07:00
Jackson Harper
ddaac82653 Fix content-fetch on docker compose 2022-06-17 14:59:42 -07:00
Jackson Harper
58814e1854 Run the content-fetch service in docker compose 2022-06-17 14:19:06 -07:00
Jackson Harper
71f8834477 Fix detection of medium subdomains 2022-06-17 09:25:42 -07:00
Hongbo Wu
486f22a594 remove redundant async 2022-06-15 22:31:55 +08:00
Hongbo Wu
2aafd39650 add fastcompany.com to the non-script hosts list 2022-06-15 22:27:25 +08:00
Hongbo Wu
e028e2e440 generate test page for fast company 2022-06-15 21:22:25 +08:00
Jackson Harper
486f3c930b Remove PROXY_URL from content-fetch 2022-06-14 20:30:02 -07:00
Jackson Harper
ec5bbb8350 Return URL as a string 2022-06-14 16:23:07 -07:00
Jackson Harper
be2801477b Add some extra debugging 2022-06-14 16:13:43 -07:00
Jackson Harper
159a7f8950 Fallback to scrapingbee if a page cant fetch content 2022-06-14 16:06:01 -07:00
Jackson Harper
814f6098a3 Log proxy url 2022-06-14 14:27:19 -07:00
Jackson Harper
a4ad78652a Allow specifying a proxy url when launching puppeteer 2022-06-14 13:30:32 -07:00
Jackson Harper
b94215f1fc Allow selectively disabling javascript on some hosts 
Some hosts readability is improved by disabling javascript
 2022-06-10 13:25:14 -07:00
Jackson Harper
cb98a9cf86 Make clients opt into creating a page when uploading a file 2022-05-26 21:40:40 -07:00
Hongbo Wu
0cc7e84a82 Fix content not getting parsed by linkedom properly without <html> tag by replacing innerHtml with outerHtml 2022-05-18 15:52:16 +08:00
Hongbo Wu
8f0447ed3f Stop blocking images and css file 2022-05-18 15:50:52 +08:00
Hongbo Wu
629aa54c58 Fix youtube handler 2022-05-18 11:28:33 +08:00
Hongbo Wu
ca662964e6 Fix not getting youtube video id from url 2022-05-17 21:51:03 +08:00
Hongbo Wu
745f55a843 Set headless=true 2022-05-14 10:47:15 +08:00
Hongbo Wu
80c14cd6ca Remove single-process from chromium args 2022-05-14 10:37:06 +08:00
Jackson Harper
7bfb8cfee4
Merge pull request #597 from omnivore-app/remove-chrome-aws-lambda 
Optimize puppeteer and remove chrome-aws-lambda dependencies
 2022-05-13 16:12:24 -07:00
Jackson Harper
87b11277d1 Add token verification to the content-fetch service 2022-05-13 15:14:09 -07:00
Hongbo Wu
6f09a4b31a Fix missing variable name in medium handler 2022-05-13 17:47:21 +08:00
Hongbo Wu
f5003c1370 Stop blocking script 2022-05-13 12:17:19 +08:00
Hongbo Wu
37e55add98 Stop blocking stylesheet and media 2022-05-13 12:09:05 +08:00
Hongbo Wu
ad99f933e5 Fix tests cont 2022-05-12 17:53:28 +08:00
Hongbo Wu
60bbbb6cf3 Block requests to 'font', 'image', 'stylesheet', 'script', 'media' in puppeteer 2022-05-12 17:10:38 +08:00
Hongbo Wu
b766e17189 Remove jsdom in content-fetch 2022-05-12 16:48:59 +08:00
Hongbo Wu
9606cd6b28 Remove chrome-aws-lambda dependencies 2022-05-12 16:32:22 +08:00