da28998130
next/react doesnt want child elements of the paragraphs Improve formatting Improve wording Use buttons in the subscribe directly blocks Simplify docs on setting up forwarding rules Add extra padding on bottom of help docs Remove unused style Add emails help page Improve formatting Prefetch page content on iOS Reduce the reader overly length now that items are precached Add invalidation when highlights are added to items fix missing index_settings.json file in api dockerfile for creating elastic index (#363) Handle full email address objects in the to param from sendgrid These come in a format like: "jacksonh-dfdf@inbox.omnivore.app" <jacksonh-dfdf@inbox-demo.omnivore.app> New IDs for short highlights because they dont cascade delete now Testing CI issues Simplify test CI test Use promises for async tests Temporarily remove test to debug CI Re-enable re-enable test, return error Specify a userId when looking up saved email pages create a unique url for newsletters without a URL Use 500ms on page test timeouts Increase timeout Dont use deep equal to match newsletter label Run just the labels API Run against just the newsletter emails Run without the page tests Fix Set the allow uncaught flag Remove highlight tests Remove newsletters tests more resolver tests Remove newsetter tests Comment out resolver tests Use nock for external requests in tests Specify puppeteer url for tests Comment out more tests uncomment tests re-enable re-enable email test Re-disable Re-enable one pdf attachment test Re-disable pdf attachment test Use promises on setTimeout tests rm label tests mv label tests into a context Comment out pdf tests Comment out pdf tests Async test Async wrappers Delay when creating test pages More debugging Unique short ids Remove potentially problematic test Fetch page before returning for test handler in before block more debugging More debugging Move errors checks into contexts Use a context when saving newsletters to force index refresh Prettier fix Fix newsletter label check, remove setTimeout Re-enable test timeout on pdf router handler Fix method call comment out PDF test Unique fake username Comment out PDF test Debugging signed urls Re-enable New email pdf test PDF tests Comment out pdf test Add nock stubs for email URLs Use full address for PDF test Remove debug Use full email addresses |
||
|---|---|---|
| .github | ||
| apple | ||
| imageproxy | ||
| packages | ||
| pkg | ||
| .dockerignore | ||
| .editorconfig | ||
| .eslintignore | ||
| .eslintrc | ||
| .gitignore | ||
| .prettierrc | ||
| .vercelignore | ||
| codegen.yml | ||
| docker-compose.yml | ||
| lerna.json | ||
| LICENSE | ||
| Makefile | ||
| package.json | ||
| README.md | ||
| tsconfig.json | ||
| vercel.json | ||
| yarn.lock | ||
Omnivore
Omnivore is a complete, open source read-it-later solution for people who like text.
We built Omnivore because we love reading and we want it to be more social. Join us!
- Highlighting, notes, search, and sharing
- Full keyboard navigation
- Automatically saves your place in long articles
- Add articles via email (with substack support!)
- PDF support
- Web app written in node and typescript
- Native iOS app
- Progressive web app for Android users
- Browser extensions for Chrome, Safari, Firefox, and Edge
- Tagging (coming soon!)
- Offline support (coming soon!)
Every single part is fully open source! Fork it, extend it, or deploy it to your own server.
We also have a free hosted version of Omnivore at omnivore.app -- try it now!
Join us on Discord! 💬
We're building our community on Discord. Join us!
Read more about Omnivore on our blog. https://blog.omnivore.app/p/getting-started-with-omnivore
Shoutouts 🎉
Omnivore takes advantage of some great open source software:
- TypeScript - Most of our backend and frontend are written in TypeScript.
- Next.js - Our frontend is a Next.JS app and is hosted on Vercel.
- SWR - We do all our data fetching on the web using SWR.
- Stitches - We use Stitches on the frontend to style our components.
- Mozilla Readability - We use Mozilla's Readability library to make pages easier to read.
- Swift GraphQL - We generate our GraphQL queries on iOS using Swift GraphQL.
- Radix - We use Raxix UI's components on our frontend.
- And many more awesome libraries, just checkout our package files to see what we are using.
Importing Libraries
If you have a library you'd like to import, @davidohlin has created a tool that imports a list of CSV URLs: omnivore-import
How to setup local development 💻
The easiest way to get started with local development is to use docker-compose up. This will start a postgres container, our web frontend, an API server, and our content fetching microservice.
Requirements for development
Omnivore is written in TypeScript and JavaScript.
Running the web and API services
1. Start docker-compose
git clone https://github.com/omnivore-app/omnivore
cd omnivore
docker-compose up
This will start postgres, initialize the database, and start the web and api services.
2. Open the browser
Open http://localhost:3000 and confirm Omnivore is running
3. Create a test account
Omnivore uses social login, but for testing there is an email + password option.
Go to http://localhost:3000/email-registration in your browser.
Frontend Development
If you want to work on just the frontend of Omnivore you can run the backend services with docker compose and the frontend locally:
docker-compose up api content-fetch
cd packages/web
cp .env.local .env
yarn dev
Running the puppeteer-parse service outside of Docker
To save pages you need to run the puppeteer-parse service.
1. Install and configure Chromium
brew install chromium --no-quarantine
export PUPPETEER_SKIP_CHROMIUM_DOWNLOAD=true
export CHROMIUM_PATH=`which chromium`
2. Navigate to the service directory, setup your env file, and install dependencies
cd packages/puppeteer-parse
cp .env.example .env
yarn
3. Start the service
yarn start
This will start the puppeteer-parse service on port 9090.
In your browser go to http://localhost:3000/home, click the Add Link button,
and enter a URL such as https://blog.omnivore.app/p/getting-started-with-omnivore.
You should see a Chromium window open and navigate to your link. When the service is done fetching your content you will see it in your library.
How to deploy to your own server
Omnivore was originally designed to be deployed on GCP and takes advantage of some of GCP's PaaS features. We are working to make Omnivore more portable so you can easily run the service on your own infrastructure. You can track progress here: https://github.com/omnivore-app/omnivore/issues/25
To deploy Omnivore on your own hardware you will need to deploy three dockerized services and configure access to a postgres service. To handle PDF documents you will need to configure access to a Google Cloud Storage bucket.
packages/api- the backend API servicepackages/web- the web frontend (can easily be deployed to vercel)packages/puppeteer-parse- the content fetching service (can easily be deployed as an AWS lambda or GCP Cloud Function)
Additionally, you will need to run our database migrations to initialize
your database. These are dockerized and can be run with the
packages/db service.
License
Omnivore and our extensions to Readability.js are under the AGPL-3.0 license.