Commit graph

151 commits

Author SHA1 Message Date
Jackson Harper
d33d383891
Merge pull request #422 from omnivore-app/fix/reminder-data-model-error
Fix reminders resolver getting pages from postgres
2022-04-14 14:10:48 -07:00
Jackson Harper
c80ce4da10 Consistent naming convention for events 2022-04-14 10:31:10 -07:00
Jackson Harper
2c9c2a2adf
Merge pull request #394 from omnivore-app/header-and-reader-improvements
Add margins on wider screens, keyboard commands to adjust
2022-04-13 21:26:24 -07:00
Hongbo Wu
c0966d908f
upload highlight and label to gcs if successfully updated in elastic (#420) 2022-04-14 12:14:15 +08:00
Hongbo Wu
88d8ca7b7d fix reminders resolver getting page from postgres 2022-04-14 12:09:17 +08:00
Jackson Harper
e282fc8316
Merge pull request #417 from omnivore-app/feature/upload-highlight-gcs
Upload highlights and labels data to GCS
2022-04-13 13:41:29 -07:00
Hongbo Wu
0b0b521fab use generic method for uploading 2022-04-13 22:21:20 +08:00
Hongbo Wu
a806f55233 upload highlight to gcs 2022-04-13 21:18:58 +08:00
Hongbo Wu
8ba1be456c fix sanitizer invalid name 2022-04-13 13:23:21 +08:00
Hongbo Wu
3b831fecb8 test both max length and pattern if declared 2022-04-13 12:20:41 +08:00
Hongbo Wu
928ddd69a9 validate color input in label as rgb hex value 2022-04-13 12:17:12 +08:00
Jackson Harper
73ef8fef09 Rename linkId to pageId in SetLabelsInput 2022-04-12 12:46:32 -07:00
Jackson Harper
d05f091876 Add debug logging for GQL errors 2022-04-12 12:23:55 -07:00
Jackson Harper
6c5e88cf12
Merge pull request #405 from omnivore-app/feature/label-max-length
Set label name max length = 64 char
2022-04-12 11:14:20 -07:00
Hongbo Wu
ae0d1dd2ee
Feature/search highlights backend (#395)
* add highlight mappings

* return highlight in resolvers

* temporarily skip highlight tests

* add test for getting highlights

* update merge highlight

* separate elastic methods

* roll back merge highlight test

* add highlight to elastic script

* update delete highlight in elastic

* migrate highlight data from postgres to elastic

* rescue not found exception when page is not found in the migration script

* exclude highlights in searching pages results

* search pages with highlights only with has:highlight query

* add search endpoint to search pages or highlights

* reduce code smell in search api

* fix rebase error

* fix tests

* add test for search highlight

* add test for new search endpoint

* add labels to search results

* update schema

* update search query

* fix update/share highlights

* fix rebase error

* fix tests

* add highlight model in elastic

* add savedAt and publishedAt date range in search query

* add sort by updated and recently read

* fix tests

* close db connection when tests are done

* test github action

* revert github action test

* fix rebase error

* add docker-compose for api-test

* remove unused env

* remove highlights with no page attached to

* allow get_articles resolver to search for query so we can merge it without web changes
2022-04-12 12:31:08 +08:00
Hongbo Wu
2ebdaba780
add generate api key api and test (#392)
* add generate api key api and test

* test if user can make api call with the api key
2022-04-12 12:11:45 +08:00
Hongbo Wu
bb665a704b make label name <= 64 char length 2022-04-12 12:08:02 +08:00
Hongbo Wu
b4420839bd
fix: fail to search by excluding labels with capital letter (#400)
* fix: fail to search by excluding labels with capital letter

* make all the excluded labels lowcased

* Revert "make all the excluded labels lowcased"

This reverts commit 866bed4080.
2022-04-11 14:05:21 +08:00
Jackson Harper
48ca13818f Prettier fixes 2022-04-06 10:45:01 -07:00
Jackson Harper
57f865c9a2 Update to typeorm 0.3 2022-04-06 10:37:13 -07:00
Jackson Harper
1cdb7753c3 Update typeorm usage 2022-04-06 09:56:32 -07:00
Jackson Harper
06360a5993
Merge branch 'main' into OMN-190 2022-04-06 09:24:49 -07:00
Hongbo Wu
d1269f815e identify a newsletter by checking both list-url and unsubscribe-url in headers and email sender address 2022-04-06 20:59:14 +08:00
Hongbo Wu
7569e988bf
upgrade typeorm to 3.0 (#359)
* upgrade typeorm to 3.0

* use new datasource object in typeorm 3

* fix tests

* fix tests

* migrate before creating connection

* fail the test if migration failed
2022-04-06 10:32:41 +08:00
Jackson Harper
9845a49040 Remove semicolon 2022-04-04 08:17:46 -07:00
Jackson Harper
c8ec7d636e Use require to import address parser function 2022-04-04 08:11:26 -07:00
Jackson Harper
92925b6808 Prettier fixes 2022-04-03 10:21:16 -07:00
Jackson Harper
da28998130 Pull ul list out of newsletter blurb
next/react doesnt want child elements of the paragraphs

Improve formatting

Improve wording

Use buttons in the subscribe directly blocks

Simplify docs on setting up forwarding rules

Add extra padding on bottom of help docs

Remove unused style

Add emails help page

Improve formatting

Prefetch page content on iOS

Reduce the reader overly length now that items are precached

Add invalidation when highlights are added to items

fix missing index_settings.json file in api dockerfile for creating elastic index (#363)

Handle full email address objects in the to param from sendgrid

These come in a format like:

"jacksonh-dfdf@inbox.omnivore.app" <jacksonh-dfdf@inbox-demo.omnivore.app>

New IDs for short highlights because they dont cascade delete now

Testing CI issues

Simplify test

CI test

Use promises for async tests

Temporarily remove test to debug CI

Re-enable

re-enable test, return error

Specify a userId when looking up saved email pages

create a unique url for newsletters without a URL

Use 500ms on page test timeouts

Increase timeout

Dont use deep equal to match newsletter label

Run just the labels API

Run against just the newsletter emails

Run without the page tests

Fix

Set the allow uncaught flag

Remove highlight tests

Remove newsletters tests

more resolver tests

Remove newsetter tests

Comment out resolver tests

Use nock for external requests in tests

Specify puppeteer url for tests

Comment out more tests

uncomment tests

re-enable

re-enable email test

Re-disable

Re-enable one pdf attachment test

Re-disable pdf attachment test

Use promises on setTimeout tests

rm label tests

mv label tests into a context

Comment out pdf tests

Comment out pdf tests

Async test

Async wrappers

Delay when creating test pages

More debugging

Unique short ids

Remove potentially problematic test

Fetch page before returning for test

handler in before block

more debugging

More debugging

Move errors checks into contexts

Use a context when saving newsletters to force index refresh

Prettier fix

Fix newsletter label check, remove setTimeout

Re-enable test

timeout on pdf router handler

Fix method call

comment out PDF test

Unique fake username

Comment out PDF test

Debugging signed urls

Re-enable

New email

pdf test

PDF tests

Comment out pdf test

Add nock stubs for email URLs

Use full address for PDF test

Remove debug

Use full email addresses
2022-04-02 16:56:24 -07:00
gitstart-omnivore
1978a92a1b Effected review corrections 2022-03-31 13:03:50 +00:00
Jackson Harper
a8faaf835c Remove debug sort on labels 2022-03-30 14:13:26 -07:00
Jackson Harper
2c15516d96 Order labels by createdAt so they maintain a consistent order 2022-03-30 13:28:37 -07:00
gitstart-omnivore
11db7be473 [Omn-190] - [Settings View] - Labels 2022-03-30 12:24:29 +00:00
Hongbo Wu
6d405432af
add site_name and site_icon to page model and return in resolver (#341)
* add site_name and site_icon to page model and return in resolver

* fix tests
2022-03-30 10:43:10 +08:00
Jackson Harper
2bada0b9df
Merge pull request #293 from omnivore-app/feature/priority-by-rate-limit
Use lower priority queue for large imports
2022-03-28 13:19:35 -07:00
Jackson Harper
d9af397b9c
Merge pull request #331 from omnivore-app/fix/update-page-after-deletion
fix document_missing_exception being thrown to sentry when updating page in elastic
2022-03-28 10:30:31 -07:00
Hongbo Wu
fd18affaec fix document_missing_exception being thrown to sentry when updating page in elastic 2022-03-28 21:16:33 +08:00
Hongbo Wu
b6c04f3508 fix saving report item by linking to elastic page 2022-03-28 20:38:54 +08:00
Jackson Harper
1cdbfaca26
Merge pull request #294 from omnivore-app/feature/add-created-or-updated_at-to-newsletter_email
add created_at and updated_at to newsletter emails and sort by create…
2022-03-25 09:10:54 -07:00
Hongbo Wu
27735f7310
add page id in updated page json file in gcs (#319) 2022-03-25 13:50:43 +08:00
Hongbo Wu
e7203bebb5
add a function_resolver to return originalArticleUrl from article.url (#317)
* add a function_resolver to return originalArticleUrl from article.url

* add a test
2022-03-25 11:25:30 +08:00
Hongbo Wu
6597cc37b6 add created_at and updated_at to newsletter emails and sort by created_at desc 2022-03-22 18:43:47 +08:00
Hongbo Wu
acec3b1acb Use lower priority queue for large imports
* add a count query in elastic
  * if priority is not set, check the rate limit of user's libary
    - if 5 articles added in the last minute: use low queue
    - default: use normal queue
2022-03-22 18:08:08 +08:00
Hongbo Wu
1893e36375
Fix uploading endpoint returns 504 (#289)
* add timeout = 60s for uploading to private bucket

* add debug logs

* send response to client to close connection

* add tests for upload

* add PUBSUB_VERIFICATION_TOKEN in .env.test
2022-03-22 14:20:40 +08:00
Hongbo Wu
3e9063c145
return empty content instead of null in search result (#291) 2022-03-22 13:34:44 +08:00
Hongbo Wu
bee3b3c6fa
fix article_id type mismatch (uuid => varchar) when saving highlights in postgres by setting article_id to be undefined (#285) 2022-03-22 11:48:01 +08:00
Hongbo Wu
4ca7c89622
format date string as yyyy-mm-dd (#288) 2022-03-22 11:46:39 +08:00
Hongbo Wu
c15efe6aad
Fix sentry log size error (#287)
* reduce sentry log size error by partially logging page created in elastic

* remove some debugging logs

* remove more debugging logs
2022-03-22 11:45:58 +08:00
Jackson Harper
f9bce1932c Print the logging level on startup 2022-03-21 09:31:05 -07:00
Hongbo Wu
77050905a5 retry max three times if update labels is conflicted 2022-03-18 10:39:11 +08:00
Hongbo Wu
9d8d2e40d7 reduce time to save reading progress in elastic update 2022-03-18 10:29:01 +08:00