e652a6ea8c
* Add elastic to our docker compose * add AND/OR/NOT search operations * add elastic and create article in elastic * change error code when elastic throws error * add search pages in elastic * add search by labels * Add elastic to GitHub Action * Update elastic version * Fix port for elastic * add url in search query * Set elastic features when running tests * add debug logs * Use localhost instead of service hostname * refresh elastic after create/update * update search labels query * add typescript support * search pages in elastic * fix search queries * use elastic for saving page * fix test failure * update getArticle api to use elastic * use generic get page function * add elastic migration python script * fix bulk helper param * save elastic page id in article_saving_request instead of postgres article_id * fix page archiving and deleting * add tests for deleteArticle * remove custom date type in elastic mappings which not exist in older version of elastic * fix timestamp format issue * add tests for save reading progress * add tests for save file * optimize search results * add alias to index * update migration script to receive env var as params * Add failing test to validate we don't decrease reading progress This test is failing with Elastic because we aren't fetching the reading progress from elastic here, and are fetching it from postgres. * Rename readingProgress to readingProgressPercent This is the name stored in elastic, so fixes issues pulling the value out. * Linting * Add failing test for creating highlights w/elastic This test fails because the highlight can't be looked up. Is there a different ID we should be passing in to query for highlights, or do we need to update the query to look for elastic_id? * add tests code coverage threshold * update nyc config * include more files in test coverage * change alias name * update updateContent to update pages in elastic * remove debug log * fix createhighlight test * search pages by alias in elastic * update set labels and delete labels in elastic * migration script enumeration * make BULK_SIZE an env var * fix pdf search indexing * debug github action exit issue * call pubsub when create/update/delete page in elastic * fix json parsing bug and reduce reading data from file * replace a depreciated pubsub api call * debug github action exit issue * debug github action exit issue * add handler to upload elastic page data to GCS * fix tests * Use http_auth instead of basic_auth * add index creation and existing postgres tables update in migration script * fix a typo to connect to elastic * rename readingProgress to readingProgressPercent * migrate elastic_page_id in highlights and article_saving_request tables * update migration script to include number of updated rows * update db migration query * read index mappings from file * fix upload pages to gcs * fix tests failure due to pageContext * fix upload file id not exist error * Handle savedAt & isArchived attributes w/out quering elastic * Fix prettier issues * fix content-type mismatching * revert pageId to linkId because frontend was not deployed yet * fix newsletters and attachment not saved in elastic * put linkId in article for setting labels * exclude orginalHtml in the result of searching to improve performace * exclude content in the result of searching to improve performace * remove score sorting * do not refresh immediately to reduce searching and indexing time * do not replace the backup data in gcs * fix no article id defined in articleSavingRequest * add logging of elastic api running time * reduce home feed pagination size to 15 * reduce home feed pagination size to 10 * stop revalidating first page * do not use a separate api to fetch reading progress * Remove unused comment * get reading progress if not exists * replace ngram tokenizer with standard tokenizer * fix tests * remove .env.local * add sort keyword in searching to sort by score Co-authored-by: Hongbo Wu <hongbo@omnivore.app> |
||
|---|---|---|
| .github | ||
| apple | ||
| imageproxy | ||
| packages | ||
| pkg | ||
| .dockerignore | ||
| .editorconfig | ||
| .eslintignore | ||
| .eslintrc | ||
| .gitignore | ||
| .prettierrc | ||
| .vercelignore | ||
| codegen.yml | ||
| docker-compose.yml | ||
| lerna.json | ||
| LICENSE | ||
| Makefile | ||
| package.json | ||
| README.md | ||
| tsconfig.json | ||
| vercel.json | ||
| yarn.lock | ||
Omnivore
Omnivore is a complete, open source read-it-later solution for people who like text.
We built Omnivore because we love reading and we want it to be more social. Join us!
- Highlighting, notes, search, and sharing
- Full keyboard navigation
- Automatically saves your place in long articles
- Add articles via email (with substack support!)
- PDF support
- Web app written in node and typescript
- Native iOS app
- Progressive web app for Android users
- Browser extensions for Chrome, Safari, Firefox, and Edge
- Tagging (coming soon!)
- Offline support (coming soon!)
Every single part is fully open source! Fork it, extend it, or deploy it to your own server.
We also have a free hosted version of Omnivore at omnivore.app -- try it now!
Join us on Discord! 💬
We're building our community on Discord. Join us!
Read more about Omnivore on our blog. https://blog.omnivore.app/p/getting-started-with-omnivore
Shoutouts 🎉
Omnivore takes advantage of some great open source software:
- TypeScript - Most of our backend and frontend are written in TypeScript.
- Next.js - Our frontend is a Next.JS app and is hosted on Vercel.
- SWR - We do all our data fetching on the web using SWR.
- Stitches - We use Stitches on the frontend to style our components.
- Mozilla Readability - We use Mozilla's Readability library to make pages easier to read.
- Swift GraphQL - We generate our GraphQL queries on iOS using Swift GraphQL.
- Radix - We use Raxix UI's components on our frontend.
- And many more awesome libraries, just checkout our package files to see what we are using.
Importing Libraries
If you have a library you'd like to import, @davidohlin has created a tool that imports a list of CSV URLs: omnivore-import
How to setup local development 💻
The easiest way to get started with local development is to use docker-compose up. This will start a postgres container, our web frontend, an API server, and our content fetching microservice.
Requirements for development
Omnivore is written in TypeScript and JavaScript.
Running the web and API services
1. Start docker-compose
git clone https://github.com/omnivore-app/omnivore
cd omnivore
docker-compose up
This will start postgres, initialize the database, and start the web and api services.
2. Open the browser
Open http://localhost:3000 and confirm Omnivore is running
3. Create a test account
Omnivore uses social login, but for testing there is an email + password option.
Go to http://localhost:3000/email-registration in your browser.
Frontend Development
If you want to work on just the frontend of Omnivore you can run the backend services with docker compose and the frontend locally:
docker-compose up api content-fetch
cd packages/web
cp .env.local .env
yarn dev
Running the puppeteer-parse service outside of Docker
To save pages you need to run the puppeteer-parse service.
1. Install and configure Chromium
brew install chromium --no-quarantine
export PUPPETEER_SKIP_CHROMIUM_DOWNLOAD=true
export CHROMIUM_PATH=`which chromium`
2. Navigate to the service directory, setup your env file, and install dependencies
cd packages/puppeteer-parse
cp .env.example .env
yarn
3. Start the service
yarn start
This will start the puppeteer-parse service on port 9090.
In your browser go to http://localhost:3000/home, click the Add Link button,
and enter a URL such as https://blog.omnivore.app/p/getting-started-with-omnivore.
You should see a Chromium window open and navigate to your link. When the service is done fetching your content you will see it in your library.
How to deploy to your own server
Omnivore was originally designed to be deployed on GCP and takes advantage of some of GCP's PaaS features. We are working to make Omnivore more portable so you can easily run the service on your own infrastructure. You can track progress here: https://github.com/omnivore-app/omnivore/issues/25
To deploy Omnivore on your own hardware you will need to deploy three dockerized services and configure access to a postgres service. To handle PDF documents you will need to configure access to a Google Cloud Storage bucket.
packages/api- the backend API servicepackages/web- the web frontend (can easily be deployed to vercel)packages/puppeteer-parse- the content fetching service (can easily be deployed as an AWS lambda or GCP Cloud Function)
Additionally, you will need to run our database migrations to initialize
your database. These are dockerized and can be run with the
packages/db service.
License
Omnivore and our extensions to Readability.js are under the AGPL-3.0 license.