Commit graph

53 commits

Author SHA1 Message Date
Tom Rogers
4e582fb55d
Improving Self-Hosting and Removing 3rd Party dependencies. (#4513)
Some checks failed
Build Docker Images / Build docker images (push) Has been cancelled
Build Self-Hosting Docker Images / Build self-host docker images (push) Has been cancelled
Run tests / Run Codebase tests (push) Has been cancelled
Run tests / Build docker images (push) Has been cancelled
* fix: Library Header layout shift

* Bump Github Actions versions.

* Self-Hosting Changes

* Fix Minio Environment Variable

* Just make pdfs successful, due to lack of PDFHandler

* Fix issue where flag was set wrong

* Added an NGINX Example file

* Add some documentation for self-hosting via Docker Compose

* Make some adjustments to Puppeteer due to failing sites.

* adjust timings

* Add start of Mail Service

* Fix Docker Files

* More email service stuff

* Add Guide to use Zapier for Email-Importing.

* Ensure that if no env is provided it uses the old email settings

* Add some instructions for self-hosted email

* Add SNS Endpoints for Mail Watcher

* Add steps and functionality for using SES and SNS for email

* Uncomment a few jobs.

* Added option for Firefox for parser. Was having issues with Chromium on Docker.

* Add missing space.

Co-authored-by: Russ Taylor <729694+russtaylor@users.noreply.github.com>

* Fix some wording on the Guide

* update browser extension to handle self-hosted instances

* add slight documentation to options page

* Fix MV

* Do raw handlers for Medium

* Fix images in Medium

* Update self-hosting/GUIDE.md

Co-authored-by: Mike Baker <1426795+mbaker3@users.noreply.github.com>

* Update Guide with other variables

* Add The Verge to JS-less handlers

* Update regex and image-proxy

* Update self-hosting/nginx/nginx.conf

Co-authored-by: Mike Baker <1426795+mbaker3@users.noreply.github.com>

* Update regex and image-proxy

* Update self-hosting/docker-compose/docker-compose.yml

Co-authored-by: Mike Baker <1426795+mbaker3@users.noreply.github.com>

* Fix Minio for Export

* Merge to main

* Update GUIDE with newer NGINX

* Update nginx config to include api/save route

* Enable Native PDF View for PDFS

* Enable Native PDF View for PDFS

* feat:lover packages test

* feat:working build

* feat:alpine build

* docs:api dockerfile docs

* Write a PDF.js wrapper to replace pspdfkit

* Revert changes for replication, set settings to have default mode

* build folder got removed due to gitignore on pdf

* Add Box shadow to pdf pages

* Add Toggle for Progress in PDFS, enabled native viewer toggle

* Update node version to LTS

* Update node version to LTS

* Fix Linting issues

* Fix Linting issues

* Make env variable nullable

* Add touchend listener for mobile

* Make changes to PDF for mobile

* fix(android): change serverUrl to selfhosted first

* feat:2 stage alpine content fetch

* feat:separated start script

* fix:changed to node 22

* Add back youtube functionality and add guide

* trigger build

* Fix cache issue on YouTube

* Allow empty AWS_S3_ENDPOINT

* Allow empty AWS_S3_ENDPOINT

* Add GCHR for all images

* Add GCHR For self hosting.

* Add GCHR For self hosting.

* Test prebuilt.

* Test prebuilt

* Test prebuilt...

* Fix web image

* Remove Web Image (For now)

* Move docker-compose to images

* Move docker-compose files to correct locations

* Remove the need for ARGS

* Update packages, and Typescript versions

* Fix

* Fix issues with build on Web

* Correct push

* Fix Linting issues

* Fix Trace import

* Add missing types

* Fix Tasks

* Add information into guide about self-build

* Fix issues with PDF Viewer

---------

Co-authored-by: keumky2 <keumky2@woowahan.com>
Co-authored-by: William Theaker <wtheaker@nvidia.com>
Co-authored-by: Russ Taylor <729694+russtaylor@users.noreply.github.com>
Co-authored-by: David Adams <david@dadams2.com>
Co-authored-by: Mike Baker <1426795+mbaker3@users.noreply.github.com>
Co-authored-by: m1xxos <66390094+m1xxos@users.noreply.github.com>
Co-authored-by: Adil <mr.adil777@gmail.com>
2025-01-27 13:33:16 +01:00
Hongbo Wu
066883a84d remove unused dependencies 2024-07-24 12:51:25 +08:00
Hongbo Wu
29a5b20d2c remove scrapingbee from content-fetch 2024-07-24 12:17:13 +08:00
Hongbo Wu
75338f5927 bypass cloudflare captcha 2024-07-10 14:43:47 +08:00
Hongbo Wu
c0f45e2411 upgrade puppeteer-core 2024-07-09 18:06:42 +08:00
Hongbo Wu
81fbaf9807 inject fingerprint 2024-07-05 18:52:20 +08:00
Hongbo Wu
a924c8448b capture content-fetch success and error events 2024-05-13 14:55:48 +08:00
Hongbo Wu
d23bccf459 upgrade puppeteer-core to prevent ProtocolTimeout and adding more debug logs 2024-05-13 14:28:26 +08:00
Hongbo Wu
a03eee5ef7 fix dependecies 2024-01-18 18:48:46 +08:00
Hongbo Wu
d9feb740cb convert content-fetch to typescript 2024-01-18 18:48:46 +08:00
Hongbo Wu
cd3402b98a rewrite puppeteer in typescript 2024-01-18 18:48:46 +08:00
Hongbo Wu
51e586ed3d separate content-fetch in puppeteer packages from saving page content 2024-01-18 18:48:46 +08:00
Luke Channings
c4773dc904
Landing page improvements and various supporting improvements 2023-10-24 09:43:39 +01:00
Hongbo Wu
792cf0b207 upgrade google cloud sdk 2023-08-11 13:55:29 +08:00
Hongbo Wu
63cbb3011e upgrade puppeteer and update chromium args 2023-08-02 15:33:15 +08:00
Hongbo Wu
8dc4c48f87 Add createImageProxyUrl callback to the readability in puppeteer-parse service 2022-12-28 16:51:23 +08:00
Hongbo Wu
6a49689d1e Add readability in puppeteer-parse 2022-12-27 21:22:49 +08:00
Hongbo Wu
e75e49a7b4 Remove logging dependecies in puppeteer-parse 2022-11-17 11:28:26 +08:00
Hongbo Wu
b3e73cd84b Add puppeteer-extra plugins 2022-11-17 11:27:51 +08:00
Hongbo Wu
c8575a203b Fix tests 2022-11-16 10:16:09 +08:00
Hongbo Wu
b18af10e75 Import puppeteer-parse in content-fetch 2022-11-16 10:15:49 +08:00
Hongbo Wu
00fed8a0fb Remove content-fetch-gcf and create a Dockerfile for the cloud function 2022-11-16 10:15:49 +08:00
Hongbo Wu
cb858484c6 Make puppeteer-parse a module 2022-11-16 10:15:49 +08:00
Hongbo Wu
53d6afe25f Fix tests 2022-10-04 10:47:58 +08:00
Hongbo Wu
78bce2d1e7 Add Dockerfile for cloud run 2022-10-04 09:51:42 +08:00
Hongbo Wu
726df26c15 Temporarily copy content-handler to cloud functions root dir when doing cloud build 2022-10-03 15:39:05 +08:00
Hongbo Wu
a9607adfd3 Import content-handler as local dependency 2022-10-03 11:11:24 +08:00
Hongbo Wu
cb609d893e Escape HTML entities in puppeteer-parse 2022-09-23 16:40:32 +08:00
dependabot[bot]
d12f3642e6
Bump puppeteer-core from 15.3.2 to 16.1.0
Bumps [puppeteer-core](https://github.com/puppeteer/puppeteer) from 15.3.2 to 16.1.0.
- [Release notes](https://github.com/puppeteer/puppeteer/releases)
- [Changelog](https://github.com/puppeteer/puppeteer/blob/main/CHANGELOG.md)
- [Commits](https://github.com/puppeteer/puppeteer/compare/v15.3.2...v16.1.0)

---
updated-dependencies:
- dependency-name: puppeteer-core
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-08-11 02:15:16 +00:00
dependabot[bot]
b86d9a08b9
Bump @google-cloud/logging-winston from 4.2.2 to 5.1.1
Bumps [@google-cloud/logging-winston](https://github.com/googleapis/nodejs-logging-winston) from 4.2.2 to 5.1.1.
- [Release notes](https://github.com/googleapis/nodejs-logging-winston/releases)
- [Changelog](https://github.com/googleapis/nodejs-logging-winston/blob/main/CHANGELOG.md)
- [Commits](https://github.com/googleapis/nodejs-logging-winston/compare/v4.2.2...v5.1.1)

---
updated-dependencies:
- dependency-name: "@google-cloud/logging-winston"
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-07-18 21:59:44 +00:00
Jackson Harper
33f35092ba Add puppeteer-core package 2022-07-15 11:19:44 -07:00
Jackson Harper
38b2b0c271 Remove puppeteer-core package in GCF 2022-07-15 11:01:15 -07:00
Hongbo Wu
1f1698ea81 sync changes to content-fetch-gcf 2022-07-15 15:11:41 +08:00
dependabot[bot]
bb7ea78e8f
Bump puppeteer-core from 13.7.0 to 15.3.2
Bumps [puppeteer-core](https://github.com/puppeteer/puppeteer) from 13.7.0 to 15.3.2.
- [Release notes](https://github.com/puppeteer/puppeteer/releases)
- [Changelog](https://github.com/puppeteer/puppeteer/blob/main/CHANGELOG.md)
- [Commits](https://github.com/puppeteer/puppeteer/compare/v13.7.0...v15.3.2)

---
updated-dependencies:
- dependency-name: puppeteer-core
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-07-11 21:35:51 +00:00
dependabot[bot]
e91f25e58c
Bump axios from 0.26.0 to 0.27.2
Bumps [axios](https://github.com/axios/axios) from 0.26.0 to 0.27.2.
- [Release notes](https://github.com/axios/axios/releases)
- [Changelog](https://github.com/axios/axios/blob/v0.27.2/CHANGELOG.md)
- [Commits](https://github.com/axios/axios/compare/v0.26.0...v0.27.2)

---
updated-dependencies:
- dependency-name: axios
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-06-27 21:31:14 +00:00
Jackson Harper
0e31a40331 Use chrome-aws-lambda in the puppeteer GCF 2022-05-13 16:48:51 -07:00
Hongbo Wu
b766e17189 Remove jsdom in content-fetch 2022-05-12 16:48:59 +08:00
Hongbo Wu
9606cd6b28 Remove chrome-aws-lambda dependencies 2022-05-12 16:32:22 +08:00
Hongbo Wu
0984dca183 Remove adblocker and block resources by url and also block mathJax script 2022-05-11 22:04:47 +08:00
Hongbo Wu
1dffbfc881 Add puppeteer-code dependencies 2022-05-09 13:45:45 +08:00
Hongbo Wu
21799b7b6d Add puppeteer-stealth and puppeteer-ad-block plugin and a user-data-dir to reduce processing time 2022-05-09 13:45:45 +08:00
Jackson Harper
a4198e34dd
Merge pull request #335 from omnivore-app/dependabot/npm_and_yarn/chrome-aws-lambda-10.1.0
Bump chrome-aws-lambda from 7.0.0 to 10.1.0
2022-04-04 15:36:06 -07:00
dependabot[bot]
3b4254a711
Bump luxon from 1.28.0 to 2.3.1
Bumps [luxon](https://github.com/moment/luxon) from 1.28.0 to 2.3.1.
- [Release notes](https://github.com/moment/luxon/releases)
- [Changelog](https://github.com/moment/luxon/blob/master/CHANGELOG.md)
- [Commits](https://github.com/moment/luxon/compare/1.28.0...2.3.1)

---
updated-dependencies:
- dependency-name: luxon
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-04-04 17:55:59 +00:00
dependabot[bot]
f356ebd781
Bump chrome-aws-lambda from 7.0.0 to 10.1.0
Bumps [chrome-aws-lambda](https://github.com/alixaxel/chrome-aws-lambda) from 7.0.0 to 10.1.0.
- [Release notes](https://github.com/alixaxel/chrome-aws-lambda/releases)
- [Commits](https://github.com/alixaxel/chrome-aws-lambda/compare/v7.0.0...v10.1.0)

---
updated-dependencies:
- dependency-name: chrome-aws-lambda
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-03-28 21:27:06 +00:00
Jackson Harper
5036924b21
Merge pull request #280 from omnivore-app/dependabot/npm_and_yarn/jsdom-19.0.0
Bump jsdom from 16.7.0 to 19.0.0
2022-03-25 09:08:55 -07:00
dependabot[bot]
6ac10c4780
Bump @google-cloud/functions-framework from 1.9.0 to 3.0.0
Bumps [@google-cloud/functions-framework](https://github.com/GoogleCloudPlatform/functions-framework-nodejs) from 1.9.0 to 3.0.0.
- [Release notes](https://github.com/GoogleCloudPlatform/functions-framework-nodejs/releases)
- [Changelog](https://github.com/GoogleCloudPlatform/functions-framework-nodejs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/GoogleCloudPlatform/functions-framework-nodejs/compare/v1.9.0...v3.0.0)

---
updated-dependencies:
- dependency-name: "@google-cloud/functions-framework"
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-03-23 18:51:39 +00:00
dependabot[bot]
3ec6f22199
Bump jsdom from 16.7.0 to 19.0.0
Bumps [jsdom](https://github.com/jsdom/jsdom) from 16.7.0 to 19.0.0.
- [Release notes](https://github.com/jsdom/jsdom/releases)
- [Changelog](https://github.com/jsdom/jsdom/blob/master/Changelog.md)
- [Commits](https://github.com/jsdom/jsdom/compare/16.7.0...19.0.0)

---
updated-dependencies:
- dependency-name: jsdom
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-03-23 18:51:08 +00:00
Jackson Harper
8ed16e0e8a Bump winston in the puppeteer service 2022-02-23 09:56:45 -08:00
dependabot[bot]
0f843809e3
Bump axios from 0.21.4 to 0.26.0
Bumps [axios](https://github.com/axios/axios) from 0.21.4 to 0.26.0.
- [Release notes](https://github.com/axios/axios/releases)
- [Changelog](https://github.com/axios/axios/blob/master/CHANGELOG.md)
- [Commits](https://github.com/axios/axios/compare/v0.21.4...v0.26.0)

---
updated-dependencies:
- dependency-name: axios
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-02-14 21:35:58 +00:00
Jackson Harper
42836b6b38 Simplify startup of the puppeteer service
- Run on port 9090 so we don't conflict with other services
- Route the docker-compose requests to the host network
- Dont require preview bucket information on startup
2022-02-11 14:44:32 -08:00