diff --git a/docs/architecture/notebook-feature-analysis.md b/docs/architecture/notebook-feature-analysis.md new file mode 100644 index 000000000..d02bef595 --- /dev/null +++ b/docs/architecture/notebook-feature-analysis.md @@ -0,0 +1,463 @@ +# Notebook Feature Analysis + +**Date**: 2025-01-16 +**Context**: Understanding how Omnivore's "Notebook" feature works and how it fits into our architecture + +--- + +## What is the Notebook? + +The **Notebook** is a free-form note area attached to each library item, separate from highlights. Think of it as: + +- **Highlights** = Specific text selections from the article (micro-notes) +- **Notebook** = Free-form note about the entire document (macro-note) + +### Use Cases + +1. **Document Summary**: Write your own summary of the article +2. **Reactions**: "This article made me think about X" +3. **Questions**: "I need to research Y further" +4. **Connections**: "This relates to concept Z I learned in..." +5. **Action Items**: "Follow up with the author about..." +6. **Meta-notes**: Notes about the document that don't fit as highlights + +--- + +## How It's Currently Implemented (Legacy Omnivore) + +### Database Schema + +**Clever hack**: The notebook is stored in the `highlight` table with a special type: + +```typescript +// highlight entity +export enum HighlightType { + Highlight = 'HIGHLIGHT', // Normal text highlights + Redaction = 'REDACTION', // Remove text from page + Note = 'NOTE', // The "notebook" for the document +} +``` + +**Key fields for notebook**: +- `highlightType` = 'NOTE' +- `annotation` = The free-form note text +- `quote`, `prefix`, `suffix`, `patch` = NULL (not needed for document-level notes) +- `libraryItemId` = Which document this note belongs to +- One notebook per library item (enforced in application logic) + +### UI/UX + +**Location**: +- Accessed via "Notebook" button in reader view +- Opens sidebar or modal + +**Layout**: +``` +┌─────────────────────────────────────────┐ +│ NOTEBOOK (for entire document) │ +│ ┌─────────────────────────────────┐ │ +│ │ Free-form text area │ │ +│ │ "Add notes to this document..." │ │ +│ │ │ │ +│ │ [Auto-saves as you type] │ │ +│ └─────────────────────────────────┘ │ +│ │ +│ HIGHLIGHTS (text selections) │ +│ ┌─────────────────────────────────┐ │ +│ │ "Quoted text from article" │ │ +│ │ └─ Annotation: "My thoughts" │ │ +│ └─────────────────────────────────┘ │ +│ ┌─────────────────────────────────┐ │ +│ │ "Another quoted text" │ │ +│ │ └─ Annotation: "More thoughts" │ │ +│ └─────────────────────────────────┘ │ +└─────────────────────────────────────────┘ +``` + +**Features**: +- Auto-save (saves after typing stops for a few seconds) +- Shows "Saved" status +- Can delete entire notebook +- Notebook always appears at the top, highlights below + +--- + +## Design Considerations for Our Implementation + +### Option 1: Keep Current Design (Notebook = Special Highlight Type) + +**Pros**: +- ✅ Already works this way in legacy code +- ✅ Simpler migration (one less entity to create) +- ✅ Highlights and notebooks share the same table +- ✅ Less code to write + +**Cons**: +- ⚠️ Conceptually odd (a "note" is not really a "highlight") +- ⚠️ Entity comment says: "to be deleted in favor of note on library item" +- ⚠️ Mixing two different concepts in one table +- ⚠️ Query complexity (always need to filter by type) + +**Code comment from entity**: +```typescript +Note = 'NOTE', // to be deleted in favor of note on library item +``` +This suggests the Omnivore team was considering moving it! + +--- + +### Option 2: Move Notebook to LibraryItem Table (Cleaner Design) + +**Pros**: +- ✅ Cleaner separation of concerns (notebook is document-level, highlights are text-level) +- ✅ Simpler queries (no need to filter highlights by type) +- ✅ More intuitive data model +- ✅ Easier to understand for new developers +- ✅ Follows the original team's planned direction + +**Cons**: +- ⚠️ Need to migrate data (convert type='NOTE' highlights to library_item.notebook column) +- ⚠️ Slightly more complex initial implementation + +**Proposed Schema**: +```sql +ALTER TABLE omnivore.library_item +ADD COLUMN notebook TEXT, +ADD COLUMN notebook_updated_at TIMESTAMPTZ; +``` + +**Migration**: +```sql +-- Copy existing notebooks from highlights table +UPDATE omnivore.library_item li +SET + notebook = h.annotation, + notebook_updated_at = h.updated_at +FROM omnivore.highlight h +WHERE h.library_item_id = li.id + AND h.highlight_type = 'NOTE'; + +-- Delete the old notebook-type highlights +DELETE FROM omnivore.highlight +WHERE highlight_type = 'NOTE'; +``` + +--- + +## Recommendation: Option 2 (Move to LibraryItem) + +### Why? + +1. **Cleaner architecture**: Notebook is document-level metadata, not a text highlight +2. **Simpler queries**: No need to filter highlights by type everywhere +3. **Aligns with original team's intent**: The comment suggests they wanted to do this +4. **Better UX**: Makes it clearer that notebook is different from highlights +5. **Future-proof**: Easier to add more document-level fields later + +### Implementation Plan + +**Backend (NestJS)**: + +1. **Database Migration** (new migration 0192): + ```sql + -- Add notebook columns to library_item + ALTER TABLE omnivore.library_item + ADD COLUMN notebook TEXT, + ADD COLUMN notebook_updated_at TIMESTAMPTZ; + + -- Migrate existing notebooks + UPDATE omnivore.library_item li + SET + notebook = h.annotation, + notebook_updated_at = h.updated_at + FROM omnivore.highlight h + WHERE h.library_item_id = li.id + AND h.highlight_type = 'NOTE'; + + -- Clean up old notebooks + DELETE FROM omnivore.highlight + WHERE highlight_type = 'NOTE'; + ``` + +2. **Update LibraryItemEntity**: + ```typescript + @Entity({ name: 'library_item', schema: 'omnivore' }) + export class LibraryItemEntity { + // ... existing fields ... + + @Column('text', { nullable: true }) + notebook?: string | null; + + @Column('timestamp', { nullable: true }) + notebookUpdatedAt?: Date | null; + } + ``` + +3. **GraphQL Schema**: + ```graphql + type LibraryItem { + # ... existing fields ... + notebook: String + notebookUpdatedAt: Date + } + + input UpdateNotebookInput { + itemId: String! + notebook: String! + } + + type Mutation { + updateNotebook(input: UpdateNotebookInput!): LibraryItem! + } + ``` + +4. **LibraryService methods**: + ```typescript + async updateNotebook(userId: string, itemId: string, notebook: string) { + const item = await this.libraryItemRepository.findOne({ + where: { id: itemId, userId } + }); + + if (!item) throw new NotFoundException(); + + item.notebook = notebook; + item.notebookUpdatedAt = new Date(); + + return await this.libraryItemRepository.save(item); + } + ``` + +**Frontend (web-vite)**: + +1. **Update GraphQL queries** to include notebook field +2. **Create NotebookEditor component**: + - Text area for free-form notes + - Auto-save functionality (debounced) + - "Saved" indicator +3. **Add to ReaderPage**: + - Notebook button in reader toolbar + - Opens sidebar or modal with notebook at top, highlights below +4. **Mutation hooks**: + - `useUpdateNotebook()` hook + +--- + +## Comparison: Notebook vs. Highlights + +| Aspect | Notebook | Highlights | +|--------|----------|------------| +| **Scope** | Entire document | Specific text selection | +| **Quantity** | One per document | Many per document | +| **UI Location** | Top of sidebar | List below notebook | +| **Input Type** | Free-form text area | Selected text + optional note | +| **Use Case** | Document-level thoughts | Text-level annotations | +| **Data Model** | `library_item.notebook` | `highlight` table | +| **Export** | Include in document exports | Export as list | + +--- + +## User Workflow + +### Creating a Notebook + +1. Open article in reader +2. Click "Notebook" button in toolbar +3. Sidebar opens showing notebook area at top +4. Type free-form notes +5. Auto-saves after typing stops (2-3 seconds) +6. Shows "Saved" indicator + +### Viewing Notebook + Highlights Together + +1. Click "Notebook" button +2. See notebook (free-form notes) at top +3. Scroll down to see all highlights +4. Both are part of the same unified view + +### Exporting + +When exporting highlights to Obsidian/Notion: +```markdown +# Article Title + +## Notebook +[Free-form notes here] + +## Highlights +- "Quoted text" - Annotation +- "Another quote" - Annotation +``` + +--- + +## Integration with Our Architecture + +### Where It Fits + +**Notebook complements our knowledge capture strategy**: + +1. **During Reading**: + - Article text (main content) + - Highlights (specific insights) + - Notebook (overall thoughts) + +2. **After Reading**: + - Review notebook for summary + - Export notebook + highlights to Obsidian + - Search across notebooks (future: RAG over notebooks) + +3. **Knowledge Synthesis**: + - Notebooks become personal summaries + - Highlights are specific evidence + - Together = comprehensive capture + +### Future AI Integration + +**Potential AI features** (post-MVP): +- "Generate notebook summary from my highlights" +- "Compare my notebook to AI summary" (learning check) +- "Find similar notebooks across my library" (semantic search) +- "Synthesize insights from multiple notebooks" (RAG) + +--- + +## Implementation Priority + +### Short-term (ARC-010: Reading & Highlights) + +**MVP for notebook**: +- ✅ Database migration (add notebook column) +- ✅ Backend mutation (updateNotebook) +- ✅ Simple text area in reader +- ✅ Auto-save +- ❌ No fancy editor (plain text is fine) +- ❌ No AI features + +**Time estimate**: 2-3 days including migration and frontend + +### Medium-term (ARC-019: Unified Highlights) + +**Enhanced notebook**: +- Rich text editor (markdown support) +- Include notebook in highlights export +- Search across notebooks + +### Long-term (Future) + +**Advanced features**: +- AI-generated notebook suggestions +- Notebook templates +- Cross-notebook synthesis + +--- + +## Technical Decisions + +### Auto-save Strategy + +**Debounced save** (like Google Docs): +- Wait 2-3 seconds after user stops typing +- Show "Saving..." indicator +- Show "Saved" when complete +- Show error if save fails + +**Why not save on every keystroke?** +- Too many API calls +- Poor UX (network lag) +- Database write overhead + +**Implementation**: +```typescript +const [notebook, setNotebook] = useState(''); +const debouncedSave = useMemo( + () => debounce((text: string) => { + updateNotebookMutation.mutate({ itemId, notebook: text }); + }, 2000), + [itemId] +); + +const handleNotebookChange = (text: string) => { + setNotebook(text); + debouncedSave(text); +}; +``` + +### Data Migration Strategy + +**Safe migration**: +1. Add new columns to library_item +2. Migrate data from highlights table +3. Keep old data temporarily (don't delete immediately) +4. Test thoroughly +5. Once confirmed working, delete old notebook-type highlights + +**Rollback plan**: +- If issues found, old data still in highlights table +- Can restore from backup + +--- + +## Questions & Answers + +### Q: Should notebook be markdown or plain text? + +**A**: Start with plain text for MVP, add markdown support later if needed. + +**Reasoning**: +- Plain text is simpler (no editor library needed) +- Users can still write structured notes +- Can always upgrade later (plain text → markdown is easy) +- Markdown → plain text is lossy (better to start simple) + +### Q: Should we allow multiple notebooks per document? + +**A**: No, one notebook per document (matches Omnivore design). + +**Reasoning**: +- Keeps UI simple +- Matches user mental model (one document = one summary note) +- If users need multiple notes, they can use highlights with annotations +- Can always add "sections" within notebook later if needed + +### Q: How does notebook relate to AI digest? + +**A**: They're different features: +- **AI Digest** = Daily summary of NEW content (triage) +- **Notebook** = Personal notes while READING content (capture) + +**Workflow**: +1. AI Digest shows "Here's what came in" +2. Click into interesting article +3. Read article, write notebook note +4. Add highlights +5. Later: Review notebook + highlights for synthesis + +--- + +## Conclusion + +**Notebook is a crucial feature** for knowledge capture: +- Complements highlights (document-level vs. text-level) +- Provides space for personal synthesis +- Part of the "Content Inbox → Knowledge Base" workflow + +**Recommendation**: +- Implement as part of ARC-010 (Reading & Highlights) +- Move from highlight table to library_item table (cleaner design) +- Start simple (plain text, auto-save) +- Enhance later (markdown, AI features) + +**Priority**: **Medium-High** (include in Phase 1 completion) + +--- + +## Next Steps + +1. Add notebook to ARC-010 (Reading Progress & Highlights) +2. Create migration 0192 (add notebook columns) +3. Update LibraryItemEntity +4. Implement updateNotebook mutation +5. Create NotebookEditor component +6. Wire into ReaderPage +7. Test thoroughly +8. Update export functionality to include notebook diff --git a/docs/architecture/product-brief.md b/docs/architecture/product-brief.md new file mode 100644 index 000000000..40a5b2edb --- /dev/null +++ b/docs/architecture/product-brief.md @@ -0,0 +1,734 @@ +# Enhancing Omnivore: Multi-Content Support, Companion Tools & Ethical Monetization + +**Introduction:** + +Omnivore – a free, open-source read-it-later app – has a strong foundation for reading and annotating text articles + +[docs.omnivore.app](https://docs.omnivore.app/#:~:text=,Chrome%2C%20Safari%2C%20Firefox%2C%20and%20Edge) + +. To expand its appeal and longevity, Omnivore can evolve beyond text-only content. This strategic report outlines opportunities to support additional content types (audio, video, PDFs, etc.), proposes companion tools for seamless content capture, suggests premium features to sustain development, and recommends ethical monetization paths. The goal is to broaden Omnivore’s capabilities while honoring its open-source values and keeping the core experience free. + +## Expanding Core Functionality to Multiple Content Types + +Omnivore can become a true **“omnivore” of content** by handling media beyond web articles. Key expansion opportunities include: + +### 1. Audiobooks and Long-Form Audio + +**Opportunity:** Enable users to save and engage with audiobooks or other lengthy audio content in Omnivore, treating them as first-class citizens alongside text articles. This could involve letting users add audiobook files/links to their library, track listening progress, and capture notes or quotes from them. + +**Implementation Ideas:** + +- **Transcripts & Textual View:** Offer an AI-powered transcript for uploaded audiobook files (when legally permissible). Using open-source speech-to-text (e.g. Whisper) to generate text would let users read or search an audiobook’s content. For copyrighted books, perhaps limit transcript length or require the user to own the text (to avoid full unauthorized reproduction). + + [github.com](https://github.com/omnivore-app/omnivore/issues/3736#:~:text=I%20use%20PocketCasts,which%20opens%20a%20HTML%20page) + +- **Highlighting & Notes:** Allow time-based highlights in the audio (like bookmarks) which link to the corresponding transcript text. Users could select a snippet of the transcript and save it as a highlight with the timestamp. This mirrors how Snipd (a podcast app) lets listeners capture “snips” from audio for later review. + + [snipd.com](https://www.snipd.com/#:~:text=Save%20Key%20Insights%2C%20Automatically) + + [snipd.com](https://www.snipd.com/#:~:text=) + +- **Bi-Directional Sync with eBooks:** If a user has the eBook (text) version of an audiobook, Omnivore could sync positions between the audio and text. For example, as they listen to chapter 3 of an audiobook, the eBook text scrolls or marks that chapter. This would require aligning the audiobook with the text (perhaps via chapter timings or whisper alignment tools). +- **User Experience:** Omnivore’s interface can include a simple audio player (play, pause, 30-second skip) for audiobooks. Progress (current timestamp) should sync across devices just like reading position does for long articles. Users could toggle between **“Listen”** and **“Read”** modes. In **Read mode**, they see the transcript (if available) and can highlight text; in **Listen mode**, they get playback controls and maybe the scrolling text. + + [docs.omnivore.app](https://docs.omnivore.app/#:~:text=,Browser%20extensions%20for%20Chrome%2C%20Safari) + + +**Why it’s Valuable:** Many users consume books via audio for convenience. Supporting audiobooks would let Omnivore serve as a unified library for both what you *read* and what you *listen to*. It taps into the growing audiobook market while leveraging Omnivore’s strength in notes/highlights. Notably, the Snipd app has expanded beyond podcasts to allow users to **upload audiobooks or any audio file** and apply its AI transcription/highlighting features + +[snipd.com](https://www.snipd.com/#:~:text=Take%20Notes%20from%20Audiobooks%20and,YouTube) + +– indicating strong demand for integrating audiobooks into knowledge workflows. + +### 2. Podcasts and Audio Articles + +**Opportunity:** Integrate podcasts into Omnivore so users can save podcast episodes for later, read or search their transcripts, and capture key insights. This bridges the gap between long-form audio content and text-based knowledge management. + +**Implementation Ideas:** + +- **Podcast Feed Integration:** Allow users to subscribe to podcast RSS feeds in Omnivore (similar to how it supports RSS news feeds). New episodes would appear in their library (perhaps under a “Podcasts” or audio content filter). Users could also save individual episode links via the browser extension or share sheet. + + [blog.omnivore.app](https://blog.omnivore.app/p/updates-notion-youtube#:~:text=The%20iOS%20app%20received%20a,few%20improvements%20and%20changes) + + [blog.omnivore.app](https://blog.omnivore.app/p/updates-notion-youtube#:~:text=The%20Android%20app%20now%20has,the%20%E2%80%9CFollowing%E2%80%9D%20tab) + +- **Automated Transcription:** When a podcast episode is saved, Omnivore can fetch its audio (if accessible via the feed or a service) and generate a transcript. If the podcast provides an official transcript (some do, per the Podcast 2.0 spec), Omnivore should import that. Otherwise, use speech-to-text to create one. An Omnivore community member suggested exactly this: parsing the episode’s audio URL, transcribing it (e.g. via Whisper), optionally refining with an LLM for punctuation/formatting, and saving the result. This mirrors what Omnivore did with YouTube videos by retrieving transcripts and cleaning them up with AI. + + [github.com](https://github.com/omnivore-app/omnivore/issues/3736#:~:text=a%20book%20is%20totally%20different,transcripts%20as%20srt%20for%20example) + + [github.com](https://github.com/omnivore-app/omnivore/issues/3736#:~:text=I%20use%20PocketCasts,which%20opens%20a%20HTML%20page) + + [blog.omnivore.app](https://blog.omnivore.app/p/updates-notion-youtube#:~:text=Transcripts%20are%20not%20just%20the,into%20a%20nice%2C%20coherent%20article) + +- **In-App Audio Player:** Provide the ability to play the podcast within Omnivore while reading along with the transcript. Like Readwise Reader’s video feature, clicking on a transcript line could jump the audio to that point. Playback controls (speed, skip) would be included. This makes Omnivore a *podcast player* of sorts, optimized for learning. + + [docs.readwise.io](https://docs.readwise.io/reader/docs/faqs/videos#:~:text=If%20you%20save%20a%20YouTube,or%20using%20special%20keyboard%20controls) + + [docs.readwise.io](https://docs.readwise.io/reader/docs/faqs/videos#:~:text=time,or%20using%20special%20keyboard%20controls) + +- **Highlighting & Note-Taking:** Users should be able to highlight text from the podcast transcript just as they do for articles. Those highlights could be exported or synced to notes (e.g. Obsidian) the same way article highlights are. Each highlight might carry the timestamp, and potentially a link that plays that segment of audio – useful if reviewing later. + + [zapier.com](https://zapier.com/blog/best-bookmaking-read-it-later-app/#:~:text=The%204%20best%20read%20it,Obsidian%2C%20Logseq%2C%20Readwise%2C%20and%20Notion) + +- **Mobile Experience:** On mobile, saved podcasts could behave like a playlist. A user might even listen to a “queue” of podcast episodes via Omnivore’s text-to-speech if they prefer an AI voice summary (see Daily Digest below). For true hands-free use, see the **Voice Integration** section on how headphone controls or voice commands could trigger highlights. + +**Why it’s Valuable:** Many users discover ideas through podcasts and want to remember or reference them later. Bringing podcasts into Omnivore allows **searching and referencing audio content as easily as text**. It essentially *unlocks audio for deep reading*: “It’s 2025, your podcast app should be able to let you search transcripts of your podcasts” and Snipd currently offers the best example of this + +[latent.space](https://www.latent.space/p/snipd#:~:text=last%203%20years%2C%20I%20finally,bullet%20and%20switched%20to%20Snipd) + +. By transcribing podcasts, Omnivore turns them into readable, highlightable material. This is especially powerful for research or learning, where you might recall + +*hearing* + +something in a podcast – with Omnivore you could search your library and find the exact quote. + +*Ethical considerations:* Full-text transcripts of podcasts (and audiobooks) raise copyright questions. Omnivore should ensure this feature is for personal use and note-taking, not wide redistribution of someone else’s content. (Even Snipd does not allow exporting entire transcripts, likely due to copyright + +[github.com](https://github.com/omnivore-app/omnivore/issues/3736#:~:text=pinei%20%20%20commented%20,74) + +.) A possible approach is to store transcripts client-side or in the user’s account privately, and perhaps limit length or require confirmation that the user has rights to that content. Despite these concerns, offering transcripts is generally accepted for personal “time-shifting” of content, similar to how DVRs time-shift TV. + +### 3. YouTube Videos and Online Lectures + +**Opportunity:** Build on Omnivore’s early YouTube integration to fully support video content. The idea is to **“read” videos** – by extracting transcripts and enabling highlights – and optionally to watch or listen to videos within Omnivore. + +**Current Status:** Omnivore introduced a beta feature to save YouTube videos and retrieve transcripts (for videos under 30 minutes) + +[blog.omnivore.app](https://blog.omnivore.app/p/updates-notion-youtube#:~:text=YouTube%20transcripts%20and%20improvements) + +. The transcript text is cleaned up with AI (punctuation, formatting) so it reads like an article + +[blog.omnivore.app](https://blog.omnivore.app/p/updates-notion-youtube#:~:text=Transcripts%20are%20not%20just%20the,into%20a%20nice%2C%20coherent%20article) + +. In the Omnivore web app and iOS app, the video could be played in a small window alongside the text + +[blog.omnivore.app](https://blog.omnivore.app/p/updates-notion-youtube#:~:text=Since%20the%20first%20iteration%20of,window%20on%20the%20right%20side) + +. This was a great start, but it had limitations (length, English-only, no timestamp sync). + +**Enhancement Ideas:** + +- **Expand Length & Languages:** Remove or extend the 30-minute limit as technology allows. For longer lectures or talks, perhaps generate transcripts in chunks. Leverage multi-language transcription (YouTube’s API or Whisper models) so non-English videos are supported. Future versions could integrate translation tools (the team had “translation tools” on their roadmap of possible paid features). This means a user could save a foreign-language video and get a transcript translated to their preferred language – hugely expanding the content they can consume. + + [docs.omnivore.app](https://docs.omnivore.app/about/pricing.html#:~:text=Future%20Pricing%20Plans%20) + +- **Time-Synced Transcripts:** Implement *interactive transcripts* that scroll with the video and highlight the current sentence, akin to karaoke subtitles. Readwise’s Reader does this by time-syncing transcripts and even allows clicking a line to jump the video. Omnivore could achieve similar functionality using YouTube’s timestamped caption data (if available) or by aligning the AI transcript with the audio. This makes it easy to follow along or locate a specific moment in the video. + + [docs.readwise.io](https://docs.readwise.io/reader/docs/faqs/videos#:~:text=If%20you%20save%20a%20YouTube,or%20using%20special%20keyboard%20controls) + + [docs.readwise.io](https://docs.readwise.io/reader/docs/faqs/videos#:~:text=time,or%20using%20special%20keyboard%20controls) + +- **Video Player Integration:** Continue to refine the in-app video player. Users should be able to **watch inside Omnivore** (in a resizable window) or choose to just read the text. Autoscroll features (like a teleprompter mode) could advance the transcript as the video plays, with the option to toggle off autoscroll for independent reading. If the user is *only* reading the transcript, perhaps show a thumbnail or link to the video, but focus on text for a distraction-free experience. + + [docs.readwise.io](https://docs.readwise.io/reader/docs/faqs/videos#:~:text=What%20is%20,do%20I%20turn%20it%20off) + +- **Metadata and Organization:** Save video metadata like duration (already being captured in beta), channel name, and possibly tags derived from the video description. This can help in filtering content by length or source. For example, a user might filter their library to only show content >1 hour (long lectures) or to show all items from a certain YouTube channel. Subscribing to channels via Omnivore could also be offered (similar to subscribing to RSS feeds). Readwise enables adding all your YouTube channel subscriptions via OPML – Omnivore could allow subscribing to a channel URL and automatically ingest new videos with transcripts, creating a personal lecture library. + + [blog.omnivore.app](https://blog.omnivore.app/p/updates-notion-youtube#:~:text=Since%20the%20first%20iteration%20of,window%20on%20the%20right%20side) + + [docs.readwise.io](https://docs.readwise.io/reader/docs/faqs/videos#:~:text=How%20do%20I%20add%20my,YouTube%20subscriptions%20to%20Reader) + + +**Why it’s Valuable:** A huge amount of educational content is on YouTube and elsewhere (conference talks, documentaries, tutorials). Users often don’t have time to watch them immediately. Omnivore’s approach of turning videos into readable articles is a game changer – as one user said, *“The YouTube transcript is a game changer… making this awesome app even more awesome!”* + +[blog.omnivore.app](https://blog.omnivore.app/p/updates-notion-youtube#:~:text=Liked%20by%20Daniel%20Prindii) + +. By fully supporting videos, Omnivore lets users + +**consume video content in their preferred modality (text or audio)** + +. It also makes video content searchable and skimmable. For instance, you could save a 60-minute lecture and later + +*search within it* + +for topics of interest – something nearly impossible just by watching. This multi-modal flexibility aligns with the trend of treating saved content as a personal knowledge base. Andreessen Horowitz even noted how Omnivore’s prototype could make a daily podcast of saved articles, letting users consume content “in whatever modality they are in the mood for” + +[blog.omnivore.app](https://blog.omnivore.app/p/updates-notion-youtube#:~:text=Andreessen%20Horowitz%20mentioned%20our%20demo,their%20articles%20related%20to%20AI) + +– the same principle applies to consuming video content as text. + +### 4. PDFs and eBooks (Enhanced Document Support) + +**Opportunity:** Strengthen Omnivore’s support for PDFs and add eBook formats (e.g. EPUB). This would broaden its utility for researchers, students, and anyone reading longer documents or books. + +**Current Status:** Omnivore already allows saving and reading PDFs + +[docs.omnivore.app](https://docs.omnivore.app/#:~:text=,Labels%20%28aka%20tagging) + +, including highlighting text in them. However, PDFs can be cumbersome (especially scanned ones), and Omnivore does not yet list support for EPUB eBooks natively. + +**Enhancement Ideas:** + +- **EPUB Support:** Implement an EPUB reader in Omnivore. EPUB is a popular open eBook format that would allow users to import novels, manuals, or reports and read them with Omnivore’s clean interface. This likely involves using a rendering library to display EPUB (which is essentially HTML/CSS content). The benefit is reflowable text (better for mobile reading than PDF) and access to book metadata (title, author, chapters). Users could then highlight and annotate eBooks just like web articles. Open-source EPUB readers (like epub.js) could potentially be integrated to speed development. +- **Improved PDF Handling:** For PDFs, add features like text reflow (where content is re-formatted for smaller screens), if possible. Ensure that highlights in PDFs are extractable as text (when PDFs are text-based). For scanned/image PDFs, integrate OCR to make them searchable and highlightable – perhaps as a background task when a PDF is added. This could use an open-source OCR engine or an API, possibly offered as a premium service if it’s resource-intensive (see Premium section). +- **Document Navigation:** Add quality-of-life features for long documents: e.g. a sidebar with document outline (headings or PDF bookmarks), ability to jump to chapters or a page number, and remembering the last read position within each PDF/ebook (similar to how it saves your place in long articles). This ensures a good experience for reading books or lengthy reports over multiple sessions. + + [docs.omnivore.app](https://docs.omnivore.app/#:~:text=,Browser%20extensions%20for%20Chrome%2C%20Safari) + +- **Annotations & Export:** Many academic users might benefit from being able to add margin notes or comments on PDFs. While Omnivore supports highlights and notes, making sure this works seamlessly in PDFs is key. Also, allow exporting PDF highlights (perhaps in a format like Markdown or an annotation file) for use elsewhere. For EPUBs, since they’re text, perhaps allow exporting them with highlights included (or as separate summary). +- **Integration with eReaders:** A stretch goal could be to integrate with e-ink devices or Kindle. For example, Omnivore could send EPUBs to a Kindle (via email) for those who prefer e-ink reading, then import any notes back. Or integrate with services like Pocket’s Kindle send – though that may be complex and not core to Omnivore’s mission of *keeping you in the app*. At minimum, ensuring that Omnivore’s mobile apps work well on tablets or e-ink Android devices (for a more book-like reading experience) would be beneficial. + +**Why it’s Valuable:** Many users have a mix of content: articles, newsletters, **PDF research papers, reports, and eBooks**. Right now they might need one app for read-it-later and another for PDFs/ebooks. By handling all in one place, Omnivore becomes the go-to reading hub. For example, a student could save a web article, a journal PDF, and an open textbook EPUB all into Omnivore and have a unified highlighting and note-taking system. This is also appealing for those who want to own their data – Omnivore being open-source and self-hostable means your annotated book or paper collection isn’t locked in a proprietary app. Moreover, expanding PDF/ebook support aligns with Omnivore’s open-source peers: the app is already seen as a strong alternative to proprietary read-it-later services + +[forum.cloudron.io](https://forum.cloudron.io/topic/8852/omnivore-open-source-read-it-later-solution#:~:text=Omnivore%20is%20a%20complete%2C%20open,very%20good%20alternative%20to%20Wallabag) + +, and improving document support will attract more users (Wallabag, an open-source competitor, has basic PDF support but limited polish). This is a natural extension of “for people who love to read” beyond just web content + +[docs.omnivore.app](https://docs.omnivore.app/#:~:text=Omnivore%20is%20a%20complete%2C%20open,people%20who%20love%20to%20read) + +. + +### 5. Voice Notes and Real-Time Audio Clipping + +**Opportunity:** Allow users to capture ideas or content *in the moment* using their voice. This could range from voice memos saved into Omnivore (with transcription) to clipping ambient audio (like a lecture or live podcast) for later reference. Essentially, make Omnivore a tool for hands-free content capture, not just consumption. + +**Use Cases:** + +- A user is driving or cooking and hears something interesting (on the radio, in a conversation, or they just have a thought). They want to save it to Omnivore without stopping to type. +- The user is attending a lecture or listening to a live podcast stream and wants to capture a quote or point made by the speaker. +- The user likes to journal or take notes by speaking out loud, then later organize those notes. + +**Feature Ideas:** + +- **Quick Voice Note in Mobile App:** Add a microphone button in Omnivore’s mobile app (and maybe web app via microphone input) to record a voice note. Upon stopping, the audio is saved to the library (perhaps as a “Voice Note” content type) and automatically transcribed to text. The transcription can be stored as the note’s content for easy reading/searching, attached to the audio file. Technologies like Whisper (offline) or cloud STT APIs could be used for this. The result: the user gets a text note of what they said, which they can tag or highlight like any article. This is akin to having a personal dictation device integrated into the knowledge library. +- **Smart Clipping “Retrobuffer”:** For real-time clipping (like capturing the last 30 seconds of what you heard), Omnivore could implement a rolling recording buffer (when activated) on the phone or an Alexa-type device. For example, a user could say, “Omnivore, save that” and the last half-minute of audio (which was temporarily recorded in RAM) gets saved and transcribed. This is technically complex and might be limited by device capabilities (e.g., on iOS background audio recording is restricted). An easier approach: if the user knows they’re in a scenario where they might want to capture audio (say listening to a lecture with their phone on), they hit “Record” at the start. Omnivore records the whole session (or until they stop). During or after recording, the user can tap a button to highlight the **current moment**; the app would mark the timestamp and later present that snippet’s transcript as a highlight. This requires some UI for the user to indicate highlights while recording (Snipd solves this by letting users triple-tap their Bluetooth headphones to mark a highlight while listening). Omnivore could use a similar approach: e.g. click the phone’s volume button or a smartwatch button as a marker. Afterward, the user has the full audio with certain segments highlighted and transcribed. + + [snipd.com](https://www.snipd.com/#:~:text=) + +- **Integration with Voice Assistants:** Simplify voice notes by leveraging voice assistant phrases. For instance, a user with Google Assistant or Siri could use a custom command: “Hey Siri, add a note to Omnivore: [idea]”. Using Siri Shortcuts on iOS, Omnivore could accept text or audio and create a new item. On Alexa or Google Home, an Omnivore skill could allow: “Alexa, ask Omnivore to record a note” – and Alexa records 30 seconds and sends it to Omnivore’s cloud for transcription. These would be advanced integrations (discussed more in Companion Tools), but they directly serve the hands-free capture goal. +- **Voice-to-Highlight in Playback:** If Omnivore implements the podcast player or article TTS playback, we can allow voice commands during playback. E.g., while Omnivore is reading an article out loud, the user could say “Hey Omnivore, highlight that” to capture the last sentence read. Or “Next” to skip to the next article. This would mimic how some podcast players allow voice control. While not trivial to implement cross-platform, this would truly enable hands-free reading sessions. + +**Why it’s Valuable:** This moves Omnivore from a passive repository to an *active capture tool*. Many knowledge-management systems (Notion, Evernote, Obsidian via plugins) are incorporating audio input and transcription because spoken notes can be faster or possible when typing isn’t (e.g. walking or driving). Integrating this into Omnivore means users don’t need a separate app for voice memos – their spoken ideas and snippets live alongside articles and can be searched and tagged. It also aligns with Omnivore’s philosophy of frictionless capture: just as the browser extension made it one-click to save an article, voice capture makes it *one sentence* to save an idea. Additionally, by capturing *contextual audio*, users can grab content from sources that might not be on the web – like an in-person lecture or a clubhouse conversation – extending Omnivore’s reach beyond digital text. + +From a technical standpoint, smartphones and smart speakers are now powerful enough to handle this. The rise of virtual assistants means users are increasingly comfortable talking to their devices. We have examples of this trend: for instance, **Pocket’s Alexa skill** lets you **listen to saved articles by voice command** (“Tell Pocket to get my articles” and Alexa will read them) + +[theverge.com](https://www.theverge.com/2018/10/11/17961564/pocket-redesign-listening-amazon-polly#:~:text=The%20new%20listening%20feature%2C%20which,record%20featured%20articles%2C%20Weiner%20says) + +. Omnivore can flip that script by also + +*accepting* + +content via voice command. This two-way voice integration (capture and consumption) would position Omnivore as a modern, AI-savvy reading tool. + +### 6. AI-Powered Smart Highlights and Semantic Tagging + +**Opportunity:** Leverage AI to help users distill and organize content automatically. Omnivore can go beyond manual highlighting by providing **smart highlights, summaries, and auto-tagging** powered by natural language processing. This adds a layer of intelligence on top of all content types (articles, PDFs, transcripts, etc.) to help users get to the key ideas faster. + +**Feature Ideas:** + +- **Auto-Highlights:** When an article or document is saved, Omnivore could run an AI model to identify the most important sentences or paragraphs. These could be automatically highlighted or presented as “Key Points”. For example, an LLM or summary algorithm might select 3-5 sentences that capture an article’s gist. The user could review these and promote them to official highlights with one click. This is similar to what some services call *AI summary highlights* or what tools like Kindle’s Popular Highlights do (crowd-sourced). Here it’s AI-sourced. The user remains in control – they can accept or ignore the suggestions – but it saves time if you just need the main points. +- **Article Summaries & TL;DR:** Provide one-click generation of a summary or TL;DR for any item. This could be shown at the top of the item (perhaps collapsible) or in a sidebar. It might be a few bullet points or a short paragraph generated by an LLM. Notably, Omnivore has already experimented with this in the form of the **Daily Digest**, which used LLM summarization to create daily highlights of your saved articles. In fact, Omnivore’s digest was described as using AI to *“sort and rank your recent items, and make summaries of them”*. Extending that, a user could request a summary of a single long article or a PDF and get an AI-generated synopsis. This is especially useful for very long reads or academic papers where you want to know if it’s relevant before diving in fully. + + [blog.omnivore.app](https://blog.omnivore.app/p/updates-notion-youtube#:~:text=The%20Omnivore%20Digest%20is%20a,skip%20through%20items%20using%20chapters) + + [blog.omnivore.app](https://blog.omnivore.app/p/updates-notion-youtube#:~:text=The%20Omnivore%20Digest%20is%20a,skip%20through%20items%20using%20chapters) + +- **Semantic Tagging:** Use AI to auto-tag or label content based on its topics or entities. For instance, an article about climate change might get tags like “environment”, “climate change”, “policy”. A transcript of a history podcast might get tagged with “World War II” if mentioned, etc. These tags can either complement user-created labels or suggest new ones. Omnivore could present “Suggested tags” when viewing an item, which the user can confirm. Semantic tagging can be powered by models that detect key topics or by services like OpenAI entities recognition. This makes organizing and filtering easier – e.g., quickly pull up all items related to “machine learning” or all that mention “Einstein”. +- **Related Content & Backlinks:** Going further, AI could help identify relationships between saved items. For example, if you have two articles on similar topics, the system could link them or recommend “You might want to read X next” because it’s semantically related. If using Omnivore as a knowledge base, it could even generate a simple knowledge graph of concepts. While this is advanced, it aligns with Logseq/Obsidian integration – those tools use backlinking to connect notes. Omnivore could auto-generate a “reference” section for each article listing other Omnivore items with similar content or overlapping themes (like how Wikipedia has “See also”). This effectively turns your saved library into an interlinked web of knowledge, with AI doing the heavy lifting to connect the dots. +- **Customization:** Users might be allowed to configure the AI assistance level. Some may want full auto-summaries on everything; others might prefer it only on demand. Ensuring transparency (maybe highlight AI-generated text in a certain color or label it) will maintain trust in the content. + +**Why it’s Valuable:** Smart highlights and tagging address the information overload problem. Many people save far more than they can read – Omnivore’s own founder noted users *“save things they’re fascinated by, to become better people,”* but often struggle to get through it all + +[theverge.com](https://www.theverge.com/2018/10/11/17961564/pocket-redesign-listening-amazon-polly#:~:text=%E2%80%9CIt%E2%80%99s%20a%20dedicated%2C%20quiet%20place,seriously%20to%20complete%20that%20loop%2C%E2%80%9D) + +[theverge.com](https://www.theverge.com/2018/10/11/17961564/pocket-redesign-listening-amazon-polly#:~:text=The%20new%20listening%20feature%2C%20which,record%20featured%20articles%2C%20Weiner%20says) + +. AI can help surface the most important content and free users from having to read every word of every item. This meets users where the industry is going: even Omnivore’s new parent project (ElevenLabs’ ElevenReader) touts AI summaries and narration of content + +[itsfoss.community](https://itsfoss.community/t/will-omnivore-continue-to-function-as-before/12733#:~:text=Is%20anyone%20here%20using%20the,specifically%20a%20FOSS%20issue%2C%20but) + +[itsfoss.community](https://itsfoss.community/t/will-omnivore-continue-to-function-as-before/12733#:~:text=Bring%20any%20book%2C%20article%2C%20PDF%2C,AI%20narration%20in%20one%20app) + +. Competing apps like Matter and Readwise’s Reader have introduced AI summarization (“ask the app to summarize this article” features), and research tools like Scholarcy or NotebookLM are exploring AI-generated summaries and Q&A on documents. If Omnivore implements these, it stays cutting-edge for power users. + +Additionally, automatic tagging and related-item linking would dramatically improve *discovery* in a personal library. Over time, a user’s Omnivore collection becomes large; semantic tagging means they can slice and dice the library by topic without manual effort. It also lays groundwork for **semantic search** (discussed as a premium feature) by adding metadata. + +Importantly, these AI features can be implemented in a privacy-conscious way (possibly running locally or on a self-hosted server with open models, or with opt-in cloud services). The Omnivore team has already piloted such features – the Daily Digest that **“turns your saved articles into a daily podcast with AI narration and summaries”** was highlighted by A16Z + +[blog.omnivore.app](https://blog.omnivore.app/p/updates-notion-youtube#:~:text=Andreessen%20Horowitz%20mentioned%20our%20demo,their%20articles%20related%20to%20AI) + +. Bringing that power directly to individual pieces of content (not just a digest) would be a compelling enhancement. + +## Companion Tools for Frictionless Capture and Consumption + +To complement Omnivore’s core web and mobile apps, a suite of companion tools and integrations can make capturing content effortless – even hands-free – and ensure access everywhere. These tools focus on minimizing friction when saving new content and maximizing flexibility when consuming it. + +### 1. Mobile App Enhancements (Capture & Offline Use) + +**Current Mobile Offerings:** Omnivore already provides native iOS and Android apps + +[docs.omnivore.app](https://docs.omnivore.app/#:~:text=,support%20via%20our%20Logseq%20Plugin) + +. These allow reading saved content and use the share sheet to save links from mobile browsers or other apps. There’s also a progressive web app option. The apps support offline reading (caching content) to some extent + +[docs.omnivore.app](https://docs.omnivore.app/#:~:text=,support%20via%20our%20Obsidian%20Plugin) + +and even basic text-to-speech on iOS + +[docs.omnivore.app](https://docs.omnivore.app/#:~:text=,support%20via%20our%20Obsidian%20Plugin) + +. + +**Enhancement Ideas:** + +- **Voice and Camera Input:** Incorporate the **voice note** feature discussed earlier directly into the mobile app. With one tap, a user can record audio (e.g., a thought or a quote from a nearby speaker) and have it saved/transcribed. Similarly, use the device camera to capture content: for instance, scanning a page from a physical book or a document and using OCR to save it as text in Omnivore. This would let users quickly grab a excerpt from the real world (a page, a poster, etc.) into their digital library. An example use-case: you’re reading a paper book and want to save a paragraph – you snap a photo in the Omnivore app, it OCRs the text and adds it as a quote with perhaps the book title as context. +- **Improved Offline Reading:** Ensure that when the app is offline (no internet), the user can still access all previously synced content, including images in articles or PDFs. Possibly add an **“Offline Mode”** toggle where the app will download all newly saved items (or those in certain folders) for offline use. This might include caching audio/video for offline if rights allow (e.g., downloading a podcast episode audio when on Wi-Fi so you can play its transcript and audio offline). For large content, perhaps allow user to specify what to keep offline (all, last 100 items, etc.). Many users on Reddit noted that offline support was a lacking point in read-it-later apps, so making it robust would stand out. + + [github.com](https://github.com/omnivore-app/omnivore/issues/4462#:~:text=What%20are%20some%20open%20source,yet%29%20though) + +- **Background Sync & Notifications:** The mobile app could periodically sync in background so that new items saved via browser or other devices are ready when the user opens the app. In addition, consider push notifications for certain events: for example, a notification like “Your Daily Digest is ready” for the AI summary each morning, or “New article from [RSS feed]” if the user wants alerts. (Currently, Omnivore added push rules for new subscriptions on iOS). This keeps users engaged and reminds them to check their content. + + [blog.omnivore.app](https://blog.omnivore.app/p/updates-notion-youtube#:~:text=The%20iOS%20app%20received%20a,few%20improvements%20and%20changes) + +- **In-App Quick Capture:** Beyond the share sheet, have a **“quick add”** button in the app where the user can paste a URL or type in some text to save. This avoids needing to switch apps to share. It could even have an integrated mini-browser: e.g., if you copy a URL and open Omnivore, it could detect the clipboard link and prompt “Save this page to Omnivore?”. +- **Integration with Shortcuts and Intents:** On iOS, provide Siri Shortcut actions (e.g., an action to “Save URL to Omnivore” which users can include in their custom workflows, or “Speak article via Omnivore”). On Android, use Intents and maybe a Google Assistant Action as discussed. These integrations allow power users to automate: imagine a Siri voice command or an automation that every time you take a screenshot or every time you say a certain phrase, it triggers adding content to Omnivore. +- **Wearable and Car Integration:** For truly hands-free scenarios, integrate Omnivore with wearables. For example, an **Apple Watch app** or complication to quickly save a voice note or check your reading queue. With an LTE-enabled watch, you could even leave your phone and still capture that idea you got while jogging by just talking to your watch. Similarly, support **CarPlay/Android Auto** by presenting a simplified interface (like a playlist of article titles or the Daily Digest) that can be read out. Snipd has done something similar by letting you use CarPlay controls to capture highlights while listening. For Omnivore, the CarPlay interface could be used to listen to text-to-speech of articles or your daily AI-generated summary, using steering wheel buttons or Siri for control – turning commute time into productive “reading” time. + + [snipd.com](https://www.snipd.com/#:~:text=Learn%20While%20Driving) + + +**Why it’s Valuable:** Mobile is where much of the content capture happens – we constantly encounter interesting links on phones. By supercharging the mobile apps with more capture methods (voice, camera) and deep integration into the mobile OS, Omnivore becomes ubiquitous and easy. No matter where the user is or what they’re doing, there’s a quick way to save or consume content: if they can’t type, they can speak; if reading on the phone is uncomfortable, they can listen to it via the app’s TTS or via a connected speaker. These improvements also cater to accessibility – voice notes help those who have difficulty typing, and TTS/CarPlay helps those who prefer auditory learning or have low vision. A **real-world scenario**: a user hears about a book in a podcast while walking – they use their phone to voice-record “Check out *Sapiens* audiobook” into Omnivore. Later at home, they find that note, search for the book, maybe even use Omnivore to get the audiobook or summary. This closes the loop in a way that currently would require juggling a separate notes app or memory. By making capture *as frictionless as a voice memo*, Omnivore keeps users engaged and reliant on it for all knowledge intake. + +### 2. Browser Extension & Web Clipper Improvements + +**Current Status:** Omnivore’s browser extensions (Chrome, Firefox, Safari, Edge) let users save the current page with one click + +[lifehacker.com](https://lifehacker.com/tech/read-later-app-omnivore-lets-you-save-articles-and-newsletters-for-free#:~:text=Omnivore%20supports%20all%20major%20browsers,those%20I%20save%20for%20enjoyment) + +. The extension can clean the page (remove ads) and send it to Omnivore. It also supports adding tags and notes at save time, which is very useful for organization + +[lifehacker.com](https://lifehacker.com/tech/read-later-app-omnivore-lets-you-save-articles-and-newsletters-for-free#:~:text=articles%20with%20a%20single%20click,those%20I%20save%20for%20enjoyment) + +. This is already a strong offering. + +**Enhancement Ideas:** + +- **Multi-Content Detection:** Upgrade the extension to detect special content on pages. For example, if the user is on a YouTube video page, the extension button could show an option “Save video with transcript to Omnivore” (using YouTube integration). One click and it grabs the video ID and triggers Omnivore’s back-end to fetch the transcript (as described earlier). Similarly, on a page that has an audio element (like a podcast web player or SoundCloud), the extension could detect the audio source and offer “Save audio to Omnivore”. This might involve some clever parsing of the HTML or known domain support (for instance, detect Spotify podcast URLs, or Recognize a Pocket Casts share URL as in the GitHub suggestion). Essentially, the extension becomes context-aware and can handle not just generic “save this link” but specific “save this media”. This reduces steps (e.g., copying a podcast link and manually adding to Omnivore later – instead it’s one click in the moment). + + [github.com](https://github.com/omnivore-app/omnivore/issues/3736#:~:text=quinncomendant%20%20%20commented%20,71) + +- **Partial Content Clipping:** Allow users to highlight a portion of a webpage and save **only the selection** to Omnivore, perhaps as a quote or excerpt. Sometimes you don’t need the whole article – just a specific recipe, code snippet, or paragraph. The extension could have a right-click context menu like “Save selection to Omnivore” that captures the highlighted text, the source URL, and maybe the title of the page. In Omnivore, this could appear as a special item (perhaps a “clipping”) with a reference back to the original page. This is analogous to Evernote’s web clipper which allows full page, selection, or simplified article clipping. It would appeal to researchers who gather bits of info from many pages. +- **Image and Screenshot Support:** Extend clipping to images or screenshots. If a user right-clicks an image, an option “Save image to Omnivore” could send it (with the page URL) to the library. This might be useful for infographics or charts that accompany an article. Alternatively, allow the extension to take a screenshot of the visible page or entire page and save that as an image/pdf in Omnivore. While Omnivore is text-centric, sometimes an image is the content (think: an infographic tweeted, or a diagram on a blog). Having it in the library with proper citation could be useful. If OCR is integrated, Omnivore could even OCR the image later to make any text in it searchable. +- **Better Feedback and Controls:** Improve the extension UI to show a preview of what will be saved (e.g., the title and an excerpt after cleaning) – so the user knows it’s captured correctly. Also, allow the user to choose the format: maybe sometimes you want the original HTML archived (for offline or if the article might vanish), or just plain text. A small toggle could allow “Save as PDF” or “Save as Markdown” if Omnivore supports multiple formats. For advanced use, a *“Read now in Omnivore”* button could not only save but immediately open the cleaned article in a new tab or in an Omnivore web app pop-up, for those times you want to use Omnivore’s reader view on the fly without committing to “later”. +- **Web Highlighter Mode:** The extension could offer an on-page highlighter that syncs with Omnivore. For instance, you’re reading a long article on the original site and you highlight text using the extension’s tool – it could behind the scenes save those highlights to Omnivore (without saving the whole article, or maybe saving it in the background). This way, if you prefer reading on the publisher’s site (perhaps to support them with a pageview), you can still capture notes to Omnivore. Those highlights would show up next time if you do open the article in Omnivore. This requires injection of a script to track highlights, similar to what tools like Hypothes.is or Liner do, but with the twist of syncing to your personal library. +- **Integrations with Other Web Tools:** The extension could also integrate with context menus or other services. For example, *Save to Omnivore* could appear in Twitter’s UI via an extension script, letting you save a particular tweet or thread (converted to text via something like ThreadReader) into Omnivore. Or integrate with Gmail/webmail: a button to “Send this email to Omnivore” (though Omnivore already has email-in, the extension might streamline it for webmail usage). These are smaller conveniences that make Omnivore catch-all for various content streams. + +**Why it’s Valuable:** The browser is where a lot of “content discovery” happens. By making the extension smarter and more powerful, Omnivore ensures **no compelling content slips through the cracks**. If it’s as easy to save a podcast or video as it is an article (just a single click), users will be more likely to capture those formats for later – feeding into the multi-content goal. Partial clipping and on-page highlighting are features that power users (researchers, students) love, since they often gather bits and pieces rather than full pages. Competing products like Notion Web Clipper and Evernote emphasize such capabilities; adding them would make Omnivore more attractive as an all-purpose research tool. + +Moreover, a great web clipper reduces the effort to build your knowledge library. The less time a user spends fiddling (like copy-pasting info or dealing with unsupported content types), the more they can focus on consuming and thinking about the content. By investing in the extension, Omnivore continues its mission of *“focused and distraction free”* reading + +[docs.omnivore.app](https://docs.omnivore.app/#:~:text=We%20built%20Omnivore%20because%20we,it%20to%20be%20more%20fun) + +right from the point of capture. + +### 3. Voice Assistant & Smart Speaker Integrations + +**Vision:** Make Omnivore accessible via voice on devices like Amazon Alexa, Google Assistant (Nest/Home devices), and Siri. This allows users to add content or retrieve content from Omnivore without a screen – for truly hands-free operation (driving, cooking, etc.). + +**Features:** + +- **Voice Commands to Add Content:** Similar to how one might add a to-do item via Alexa (“Alexa, ask Todoist to add buy milk”), users could add reading items. E.g., “**Alexa, tell Omnivore to save the latest New York Times headline**” – this would require the skill to fetch a URL (maybe not feasible generically) or better: “**Alexa, ask Omnivore to save a note**” and then the user dictates a quick note (which gets transcribed and saved, as discussed). Another use-case: if the user is listening to a podcast on an Alexa device, they could say “Alexa, ask Omnivore what I can do” and maybe Omnivore skill can reply with “I can save a short recording or read your saved articles.” There are limitations (speaking a long URL isn’t practical), so this is most useful for notes and maybe saving by title (which requires some search mechanism). Alternatively: “Alexa, ask Omnivore to add this article” could attempt to grab the content currently being heard or something – but that crosses into Amazon’s domain. For Google Assistant, it might be possible on Android to catch the current app’s link via a voice interaction (though not sure if that’s open). At minimum, voice capture of notes as described is doable. +- **Listening to Saved Articles (Personal Podcast):** The other side is more straightforward and very useful: enabling voice assistants to **read out content** from Omnivore. Pocket’s Alexa integration already does this for recent articles, essentially turning your backlog into a playlist you control by voice. Omnivore can do similarly, but even better with AI voices. For example, “**Hey Google, ask Omnivore to read my next article**” – the assistant could fetch the next item in your Omnivore queue (the service would render it to text if not already) and use the assistant’s TTS to read it. Commands like “skip, pause, next article” would be available just like an audio book or music. Alexa could say, “Reading: *‘How to improve urban farming’* from your Omnivore. [reads…]”. Because Omnivore is open-source, implementing an official Alexa Skill or Google Action is feasible (Amazon provides skills SDKs). The integration would use Omnivore’s API to fetch the user’s content. + + [theverge.com](https://www.theverge.com/2018/10/11/17961564/pocket-redesign-listening-amazon-polly#:~:text=The%20new%20listening%20feature%2C%20which,record%20featured%20articles%2C%20Weiner%20says) + +- **Daily Briefings:** Many smart speakers have a concept of a “daily briefing” or routine. Omnivore could integrate here by providing a summary of your saved items or reading out your Daily Digest each morning. For example, at 7am Alexa could automatically say “Here’s your Omnivore Digest: You have 5 new items saved. Top highlight: ...” and maybe read the AI summaries of a couple articles. This ties into the AI digest feature – delivering it via voice adds convenience. + + [blog.omnivore.app](https://blog.omnivore.app/p/updates-notion-youtube#:~:text=The%20Omnivore%20Digest%20is%20a,skip%20through%20items%20using%20chapters) + +- **Voice Search in Library:** A user might ask, “Hey Siri, what do I have in Omnivore about climate change?” If integrated with Siri shortcuts, Siri could potentially search Omnivore’s database (on-device if synced or via API) and answer with, “You have 12 items about climate change, the most recent is ‘Climate Change 2025 Report’ saved 3 days ago.” This is advanced and may require deeper integration, but it’s an interesting direction. Alexa/Google might not easily do arbitrary library Q&A without custom handling, but one could imagine the Omnivore assistant skill being able to list titles: “You have 3 unread PDFs and 2 videos waiting.” +- **Automated Reading Mode:** Combining with the earlier personal podcast idea – Omnivore could generate a daily audio file of your chosen articles (using natural TTS, maybe via ElevenLabs voices or Amazon Polly). The voice assistant integration could simply play that audio. In fact, Omnivore’s own blog mentioned turning your queue into a daily podcast using text-to-speech – the assistant could be the way users listen to that. + + [blog.omnivore.app](https://blog.omnivore.app/p/updates-notion-youtube#:~:text=Andreessen%20Horowitz%20mentioned%20our%20demo,their%20articles%20related%20to%20AI) + + +**Why it’s Valuable:** Voice integrations extend Omnivore’s reach into times and places where screen use isn’t possible or convenient. They make consuming content as easy as saying "play". Pocket’s CEO called their TTS feature a way to turn your queue into *“a personal podcast that you curate”* + +[theverge.com](https://www.theverge.com/2018/10/11/17961564/pocket-redesign-listening-amazon-polly#:~:text=Pocket%2C%20which%20lets%20you%20save,today%20on%20iOS%20and%20Android) + +– this addresses the problem of too much saved content and not enough time to read; you can now + +*listen* + +while commuting or doing chores. By offering this, Omnivore provides the same convenience, but potentially with even better AI voices (since ElevenLabs specializes in realistic speech). + +On the capture side, being able to *add* to your library by voice is a differentiator. Imagine hearing about a website or book on the radio: you can quickly say “Alexa, add [name] to Omnivore” – even if it just creates a note for you to follow up, it’s saved. This prevents the common scenario of forgetting things you intended to read. It also increases user engagement; Omnivore becomes part of daily routines (morning briefings, etc.), not just an app you open intentionally. + +From a strategic view, voice assistants are an interface growing in adoption. Ensuring Omnivore is accessible in that ecosystem future-proofs its usability. Few open-source or privacy-focused apps venture into this (due to needing cloud connectivity), but as an open project, Omnivore can do it transparently (the user links their account to the skill with an API key and can self-host if needed). The end result is an **eyes-free, hands-free Omnivore**, aligning with the needs of busy users. + +### 4. Desktop & Workflow Integrations + +While the web app and browser extension cover desktop use cases, there’s room for dedicated desktop integrations to streamline workflows: + +- **Desktop App / Menubar Tool:** Develop a lightweight desktop companion (for Windows/Mac/Linux) that lives in the system tray or menu bar. This app could allow quick search of your Omnivore library and quick saving without opening a browser. For example, a user copying a URL or snippet could click the Omnivore tray icon, see a quick-add dialog (with the copied text pre-filled, if any) and save it. It could also show the reading list, so users can open an item in their default browser or a minimal reading window. In fact, there’s already a community-made Omnivore extension for Raycast (a Mac launcher app) that lets you **search saved articles and quickly add a URL via a command**. This indicates demand for rapid desktop access. An official Omnivore mini-app could integrate with global hotkeys (e.g., Ctrl+Shift+O could bring up “Omnivore quick add” or search anywhere). This is similar to how Evernote’s legacy app had a quick note global shortcut, or how Notion has a quick capture. + + [raycast.com](https://www.raycast.com/karolusd/omnivore#:~:text=1) + + [raycast.com](https://www.raycast.com/karolusd/omnivore#:~:text=2) + +- **Clipboard Monitor:** The desktop app could optionally monitor the clipboard for URLs. If you copy a URL, it could prompt “Add this link to Omnivore?” – saving you the step of opening anything. Some users might find that too intrusive, so it could be opt-in, but for heavy researchers it’s a boon. +- **Email Integration:** While Omnivore provides an email address to forward newsletters and articles, a desktop tool could integrate with email clients (via plugins or simple mail rules) to automate that. For instance, a plugin for Outlook/Thunderbird to right-click an email and “Send to Omnivore” (which essentially forwards it to the user’s Omnivore email behind the scenes). This is a niche, but helpful for users who get a lot of content via email beyond newsletters (like PDF attachments or press releases). +- **IFTTT and Zapier Workflows:** Creating Omnivore integrations with automation platforms like IFTTT or Zapier (if an API is available) would let users set up custom triggers. For example: “When I bookmark a page in my browser, add it to Omnivore” or “When I highlight a quote in Readwise, save the source article in Omnivore” (for those using multiple tools). While not a user-facing “app”, providing actions/triggers in these services extends Omnivore’s reach to tons of apps. This is relatively low-effort if the API exists, and can be a selling point to power users. +- **Integration with Note-taking and PKM Apps:** Omnivore already has plugins for Logseq and Obsidian. Continue to support and enhance these, as they act like companion apps: e.g., in Obsidian, you can fetch your Omnivore highlights into your vault, effectively integrating Omnivore into desktop note-taking. Perhaps add a plugin for Notion (the blog mentioned a Notion integration where Omnivore writes to a Notion page) and one for Evernote or OneNote if demand exists. These let people incorporate saved readings into their larger knowledge systems. For instance, a scholar might have Obsidian for their thesis notes and Omnivore for reading papers – with the plugin they can pull in quotes from Omnivore directly into the thesis notes with backlinks to source. This deepens the user’s reliance on Omnivore for gathering sources. + + [docs.omnivore.app](https://docs.omnivore.app/#:~:text=,support%20via%20our%20Obsidian%20Plugin) + + [blog.omnivore.app](https://blog.omnivore.app/p/updates-notion-youtube#:~:text=To%20set%20up%20the%20Notion,to%20video) + +- **Command-Line Interface (CLI):** For tech-savvy users, a simple CLI tool (`omnivore-cli`) could allow adding URLs or querying the library from a terminal. This is especially useful for scripting and for those who live in the terminal (e.g., you come across a link in a terminal environment, you can add it without switching context). Given Omnivore’s open-source nature, a community member might even create this. + +**Why it’s Valuable:** These desktop and workflow integrations aim to fit Omnivore into the user’s *existing* workflows, rather than requiring the user to always go to Omnivore explicitly. By having a menubar quick-add or a global search hotkey, Omnivore becomes a *ubiquitous layer* on the desktop – always there to capture or retrieve information. This convenience can boost adoption among power users. + +The Raycast extension already being installed by many + +[raycast.com](https://www.raycast.com/karolusd/omnivore#:~:text=1%2C183%20Installs) + +shows that users want faster ways to access Omnivore on desktop. By providing official tools, it can be even more polished and secure. Also, these tools can operate even if the browser isn’t open, which is helpful if you’re trying to avoid distraction by not having a million tabs but still want to quickly save something. + +Integration with automation services (IFTTT/Zapier) and PKM apps ensures Omnivore plays nicely in the larger ecosystem of productivity tools. A user is more likely to stick with Omnivore if it doesn’t silo their data – and indeed, Omnivore’s philosophy has been openness (with APIs and webhooks + +[docs.omnivore.app](https://docs.omnivore.app/#:~:text=Org%20Mode) + +[docs.omnivore.app](https://docs.omnivore.app/#:~:text=API) + +). These integrations echo that by giving users control to connect Omnivore to whatever systems they use. + +In summary, companion tools – mobile, browser, voice, desktop – all serve to **reduce friction**. Whether it’s one-click, one-phrase, or one-keypress, capturing knowledge into Omnivore or retrieving it should be as easy as possible. The less friction, the more comprehensive the user’s library becomes, and the more valuable the tool is in their daily life. + +## Premium Features to Unlock Advanced Capabilities + +While keeping the core reading and saving experience free and open-source, Omnivore can offer a **Premium tier** with powerful features for enthusiasts and professionals. These features provide significant added value and often incur higher costs (computational or development-wise), justifying a subscription. *Importantly, any premium features should be implemented in a way that doesn’t “lock in” user data or compromise the open nature – they are enhancements, not necessities.* The Omnivore team itself anticipated this approach, noting they were experimenting with paid add-ons like **AI integration, collaborative tools, translation, and premium text-to-speech voices** + +[docs.omnivore.app](https://docs.omnivore.app/about/pricing.html#:~:text=Future%20Pricing%20Plans%20) + +. Building on that, here are proposed premium features: + +- **AI-Powered Smart Highlights & Summaries:** Premium users get advanced AI assistance while reading. This includes automatic highlight suggestions (key sentences flagged by AI), one-click article summarization, and even the ability to ask questions about the article (a conversational AI that can answer, say, “What’s the main argument here?”). These features rely on large language models which may incur API costs, so putting them in a paid tier covers those expenses. It saves users time and helps them extract insights without effort. *Example:* After saving a 30-minute article, a premium user can see a generated 3-bullet summary and 5 recommended highlights at the top, and can click “Accept All” to turn them into actual highlights. This “reading with AI” experience could be a flagship premium feature – akin to what Readwise’s **Ghostreader** or Matter’s GPT assistant offers. It effectively turns Omnivore into a reading coach or research assistant. (Core users could still use manual highlighting and might even use their own API key in self-hosted setups, but the official service could bundle it for subscribers.) +- **High-Quality Text-to-Speech (Listen to Articles):** While basic TTS (like iOS’s built-in voice) might be free, premium could offer *dramatically better* narration for all content. This ties closely to Omnivore’s partnership with ElevenLabs – which provides ultra-realistic voices. Premium users could have unlimited access to have any saved article or PDF read aloud in a natural human-like voice. As noted in the docs, “premium text to speech voices” were already in beta for ultra-realistic narration. This feature effectively turns every article into a podcast-like experience on demand. The value-add of premium here is covering the licensing or compute cost of those AI voices. It’s a compelling sell: *“Listen to your articles in lifelike voices”*. Additionally, premium might allow generating audio playlists or downloading the audio for offline listening – features casual users might not need, but power users love. This pairs well with the voice integrations (Alexa, CarPlay) – subscribers get the best listening experience across devices. Considering Pocket made a big deal of their Amazon Polly integration, Omnivore’s premium voices could leapfrog that with even better quality. + + [docs.omnivore.app](https://docs.omnivore.app/about/pricing.html#:~:text=We%20have%20a%20few%20product,beta) + + [theverge.com](https://www.theverge.com/2018/10/11/17961564/pocket-redesign-listening-amazon-polly#:~:text=Pocket%2C%20which%20lets%20you%20save,today%20on%20iOS%20and%20Android) + +- **OCR and Audio Transcription Services:** Premium tier can include heavy-duty processing like OCR for images/PDFs and transcription for audio that goes beyond the basics. For example, free users may rely on community OCR (maybe manual or limited pages per month), whereas premium users get unlimited OCR on imported PDFs or images (scanned book pages, etc.). Similarly, transcribing lengthy audio (long podcasts, entire hour-long lectures, or user’s own uploads) could be a premium perk due to the significant compute required. A premium user might have the ability to upload an audio file (say a recorded interview) to Omnivore and receive a full transcript in their library, thanks to cloud processing. This is very attractive to researchers and journalists. It basically positions Omnivore Premium as a Swiss-army tool for capturing any kind of content and turning it into text. Since services like Snipd or Otter.ai charge for large amounts of transcription, it’s reasonable to include it in a paid plan. The key is that the output (text, highlights) remains the user’s data. Omnivore could integrate with open-source models for self-hosters but use more accurate or faster cloud APIs for subscribers (e.g., Google Speech or AWS Transcribe for quick results, or Whisper on GPU servers). The GitHub suggestion to use Whisper + LLM for podcasts noted it might be viable if “paid APIs might be affordable enough to offer to premium subscribers” – exactly capturing why this fits a premium model. + + [github.com](https://github.com/omnivore-app/omnivore/issues/3736#:~:text=It%20would%20be%20great%20if,offer%20to%20premium%20Omnivore%20subscribers) + +- **Priority Sync & Offline Access:** Although Omnivore offers offline reading, a premium tier could enhance this with **priority syncing, backup, and storage benefits**. Priority sync means your content and annotations update across devices instantly with higher bandwidth or push mechanisms (the free version might sync on intervals to reduce server load). For example, if you highlight on your phone, a premium user sees it on their laptop almost immediately. Additionally, guarantee offline availability of **all** content for premium: the service could proactively download all articles (and perhaps even audio for TTS) to the device. If Omnivore runs a cloud service, premium could also mean **cloud backup of original pages** (saved HTML/PDF snapshots of webpages). So if a webpage is taken down, a premium user still has the full content archived, whereas a free user might only have the parsed text. This is similar to how Pinboard (bookmarking service) charged for an archival option. Premium could also offer bigger storage quotas (if there are limits on PDF size or number of items for free accounts to conserve costs). Another aspect: **speed** – premium users might be served by faster servers or have their requests (like fetching a new article’s text) prioritized, making the experience snappier under load. These differences would mostly be subtle, but power users notice them. The key message is reliability: *“Your library, always available, always fast”* for premium. +- **Email-to-Library and Custom Aliases:** Omnivore’s free version provides a generic email address to send content (especially newsletters) to your library. A premium tier could expand this: for instance, giving users a **custom email alias** (e.g., alice@omnivore.mail) that’s easier to remember or multiple aliases for different purposes. Multiple aliases could let a user have one for newsletters, one for sending random web content, or even one they share with friends to suggest articles – each alias could drop items into a specific folder/tag. Additionally, premium could allow sending **attachments** or other formats via email (like you could email a PDF or an EPUB file into Omnivore and it would process it). On free accounts, maybe only text emails or newsletter sources are allowed, while premium gets the full power of an email-to-library gateway (bigger size limits, etc.). Another feature: **email newsletters archiving** – premium might automatically pull all issues of certain newsletters (past archives) not just future ones, so you have the complete set in Omnivore. Since heavy email processing can increase load, it makes sense to tie extras to premium. This appeals to newsletter enthusiasts and people who might otherwise pay for services like Readwise’s Mail-to-Reader or Feedbin’s newsletter support. +- **Semantic Search & Filtering:** Premium users could gain access to more advanced search technology in Omnivore. For example, implement **full-text search** across all content (maybe free version limits search to titles and highlights to save index costs). Premium could enable searching within PDFs and transcripts. Even more powerfully, introduce **semantic search** – the ability to search by meaning, not just exact keyword. This could use vector embeddings of all saved content. A premium user could type a query like “climate effects on agriculture” and find not just literal matches but any content that discusses related concepts, even if using different terms. This kind of search often requires maintaining an embedding index, which is resource-intensive, hence a good premium differentiator. Similarly, advanced filters or analytics could be premium: e.g., filter your library by reading time, by sentiment (AI can classify articles as positive/negative tone), or generate a word cloud or summary of all your highlights. These are special features that knowledge power-users would value for research. Premium could also offer **saved search alerts** – e.g., you save a search query and whenever new content in your library matches it (or perhaps even on the web via RSS), you get notified. This crosses into RSS reader territory, but because Omnivore does have feed support, a premium user might enjoy an AI-curated feed: “notify me if any of my feeds has an article similar to X topic”. This blurs with AI, but the idea is premium gets the “smarts” in finding and organizing content. +- **Deep Linking with Knowledge Apps (Logseq/Obsidian, etc.):** While the basic integration via plugins might be free, premium could provide a more seamless or enhanced experience. For example, a premium feature could be **automatic sync of highlights to Obsidian in real-time** with rich metadata (backlinks, context, page references). Perhaps a premium user can have Omnivore maintain an up-to-date Obsidian vault of their library: each article as a note with all highlights and an automatically generated summary, plus backlinks to other notes if common topics are found. This essentially offloads a lot of manual knowledge management. Similarly, for Logseq, premium might allow two-way links: not only can Logseq pull from Omnivore, but actions in Logseq (like editing a highlight note or adding a comment) could sync back to Omnivore’s note on that item. Setting up and maintaining these robust integrations may involve using third-party APIs or more server storage, thus sensible for a paid tier. Another idea is **Notion integration** as premium: writing highlights into a Notion database or page (the basic Notion integration was introduced in beta, possibly free – but a more advanced one or higher usage could be premium). Essentially, any integration that could otherwise require self-managed servers or is targeted at professionals could be monetized. Premium could also ensure **priority support** for these workflows – e.g., if something in the API breaks due to an update, paying users’ issues get addressed faster. + + [blog.omnivore.app](https://blog.omnivore.app/p/updates-notion-youtube#:~:text=To%20set%20up%20the%20Notion,to%20video) + +- **Collaboration and Shared Libraries:** (Not explicitly in the user’s list, but mentioned in Omnivore’s ideas and worth noting as a premium path.) Premium tier could unlock collaborative features like shared folders or team libraries. For instance, two premium users (or a small team) could have a shared space where they both can add and highlight articles for a project. Real-time sync of highlights and the ability to comment on each other’s notes could be enabled. This is similar to how Pocket Premium for Teams or other knowledge tools charge organizations. It stays ethical (not exploiting data) and aligns with open-source (perhaps the code for it is open but the hosted service charges per seat). Given Omnivore’s initial focus was individual, this could be a premium extension for power users in academic or professional teams. + + [docs.omnivore.app](https://docs.omnivore.app/about/pricing.html#:~:text=Future%20Pricing%20Plans%20) + + +All these features provide *significant extra value* that hardcore users would pay for, while keeping the core – saving content, reading it, basic highlighting – free for everyone. Crucially, none of these premium features trap the user’s data in a proprietary system; if they cancel, they still have their articles and basic highlights (perhaps they lose the AI summaries or the fancy voices, which is acceptable). This is in line with open-source ethos and avoids user hostility. + +To illustrate viability: **the Omnivore team has already identified AI and better TTS as paid options, and even launched the “ultra realistic voices” beta for premium subscribers** + +[**docs.omnivore.app**](https://docs.omnivore.app/about/pricing.html#:~:text=We%20have%20a%20few%20product,beta) + +**.** + +That shows user interest in such features. Early adopters likely *expect* + +some of these to become paid. By packaging them into a subscription, Omnivore can generate revenue to sustain development, which benefits all users. + +For example, a researcher might gladly pay for premium to get unlimited transcription of interviews and semantic search through all their sources – that’s easily worth a monthly fee. A journalist might pay for the convenience of high-quality narration and instant summaries when sifting through dozens of articles. The key is communicating that these features cost money to provide (GPU time, API calls, etc.), so the subscription directly funds those capabilities, all while the core product remains open-source and community-driven. + +## Ethical Monetization Strategies Aligned with Open-Source Values + +Monetizing an open-source project like Omnivore requires a careful balance: generating sustainable revenue **without betraying user trust or the community spirit**. Below are strategies that prioritize ethical considerations and compatibility with open-source principles: + +- **Freemium Subscription Model (Open-Core):** Adopt a **transparent freemium** approach: the core Omnivore features remain free and open-source for everyone, while a subscription unlocks the premium features outlined above. This model is ethical if done right, because it doesn’t take away existing functionality or trap data – it only adds new capabilities for those who choose to pay. It’s important to communicate that subscription revenue is used to **maintain servers and fund improvements** (perhaps with regular transparency reports). Pricing should be fair and perhaps tiered (e.g., a student discount or a higher “supporter” tier for those who want to contribute more). By keeping the code open, even premium features can be audited – maybe self-hosters can enable them with their own resources, but they’d pay the official service for convenience and support. This open-core model is used by many successful open-source projects (for instance, Bitwarden open-source password manager offers a paid plan with extra features, without limiting free functionality). For Omnivore, ensure that if a premium subscription lapses, the user retains access to their data (they just lose access to premium services like AI or sync priority). This prevents any feeling of hostage-taking. It’s also worth explicitly **not** monetizing via ads or selling user data – make a pledge that user reading data is never sold or used for ad targeting, differentiating Omnivore from ad-driven platforms. A freemium model aligns incentives: Omnivore will focus on developing features users find worth paying for, which often are the ones that genuinely add value (like saving time or improving experience). The open-source community tends to accept this model when done above-board, as it enables the project’s longevity. +- **Donations and Sponsorships (Community Funding):** Continue and expand donation-based funding channels such as **GitHub Sponsors, Open Collective, Patreon, or direct donations**. Omnivore already uses Open Collective for server costs and acknowledges contributors. This method is very aligned with open-source ethos: those who love the project can chip in voluntarily. To encourage this, Omnivore could offer non-feature perks to donors – for example, a badge on their profile, a shout-out on a contributors page, or access to a community forum section. Sometimes, just goodwill and recognition are enough motivation. Also, institutional sponsorships can be pursued: perhaps a university or company that relies on Omnivore might sponsor development (this could even fund specific features). It would be wise to highlight that even small monthly donations help keep the service running for all. On Discord or newsletters, gentle reminders or campaigns (like a yearly fundraiser) could rally support. Importantly, **donation should remain optional** and not affect the user’s feature set – aside from maybe early access to new beta features as a thank-you. This keeps it ethical and inclusive. The project could aim to be partially community-funded to reduce reliance on purely commercial decisions. By diversifying funding (some from subscriptions, some from donations), Omnivore can remain independent and community-driven. Successful open-source apps (e.g., VLC, OBS) often have donation drives; Omnivore can similarly leverage its passionate user base. It might also integrate with GitHub’s sponsorship program so that within the GitHub repo, users see the “Sponsor” button and know how to contribute. Since Omnivore’s code will remain on GitHub, this is a natural fit. + + [docs.omnivore.app](https://docs.omnivore.app/about/pricing.html#:~:text=Many%20people%20have%20asked%20us,like%20copy%20editing%20and%20translations) + +- **Community-Funded Plugins or Features:** Leverage the open-source community to create an ecosystem of plugins/extensions (similar to Obsidian’s community plugins). Omnivore could allow community-developed extensions for niche features or integrations. While most of these would be free/open, there’s an opportunity for **bounty funding or crowdfunding specific enhancements**. For instance, if a group of users really wants a Zotero integration or a new feature, they could pool funds (via Kickstarter or BountySource) and either hire a developer or incentivize the core team to implement it. This way, features get funded by those who need them, and then released to everyone (perhaps in premium if it’s high-cost, or free if maintainable). The ethics here are solid: it’s a voluntary patronage for development. Omnivore could set up a roadmap page where users can **vote on features** and even pledge money toward them. Open Collective supports earmarking funds for specific purposes. Another angle is a **marketplace for plugins** – e.g., a developer makes a proprietary plugin that adds some value and sells it. However, that’s less ideal in an open environment and can complicate things. A better model is likely **patronage**: people pay to accelerate development of open features. This is how some open-source features in larger projects get done via grants or sponsor contracts. For example, perhaps a digital library foundation might grant Omnivore funds to improve PDF handling for all. Omnivore could actively seek such grants in the open knowledge space. By keeping the community involved in funding decisions, Omnivore ensures development is aligned with user needs and that funding is seen as positive contribution rather than extraction. +- **Affiliate and Partnership Programs:** Explore affiliate revenue in a way that is user-friendly and transparent. One idea is to integrate affiliate links for books or products referenced in content. For example, if a user saves a book review or highlight, Omnivore could unobtrusively provide a link like “Buy on Bookshop/Amazon” – using an affiliate code so that if the user does purchase, a small commission goes to Omnivore. This doesn’t cost the user anything and can be positioned as a convenience (and maybe even promote indie bookstores for ethics). Another affiliate angle: partnerships with related tools. If Omnivore integrates with, say, Evernote or Notion, perhaps those companies have referral programs. For instance, Notion might give a bonus if someone signs up via Omnivore’s integration prompt. As long as recommendations are honest and not spammy, this can be a minor revenue stream. **Newsletters and paid content**: If Omnivore supports newsletters, maybe partner with Substack – if a user subscribes to a paid newsletter through Omnivore’s recommendation, Omnivore could get a referral cut. Again, this should be clearly communicated (e.g., “Omnivore may earn a commission”). Many users are fine with affiliate links when disclosed, especially if it helps an open-source project. The key is not to let affiliates drive the product direction – they should be passive income, not a primary strategy. But over time, if Omnivore has a large user base, even small commissions (like people buying books or software that Omnivore surfaces) could add up. Another partnership example: text-to-speech voices – maybe offer a discount code for ElevenLabs or AWS Polly through Omnivore, getting a referral fee. **In-app content recommendations** could also be considered (like a “Recommended Reads” section powered by a partner such as a news service), but this must be done carefully to not compromise the neutrality of a user’s space. If done, it should be clearly separated and maybe only enabled if user opts in for “suggested content”. This can generate affiliate revenue (some platforms pay per click or per sign-up if you recommend their content). + +Overall, any monetization must preserve **user agency and privacy**. For instance, no ads injected into your articles, no selling reading history to advertisers – those would violate trust and drive users away (and contradict the open nature). Instead, focus on *value-add services that users choose to pay for* or *voluntary support*. + +Omnivore can also publish a manifesto or policy about data usage and monetization to be transparent (e.g., a commitment that the user’s highlights and data belong to them and will never be locked behind a paywall or sold). This kind of stance actually can *attract* paying users, because they see they are supporting a project that respects them. + +Finally, marketing the premium tier and monetization should emphasize alignment with open-source values. For example: *“Omnivore Premium helps fund the free, open-source Omnivore for everyone. By subscribing, you’re not just unlocking features for yourself – you’re directly supporting the maintenance of the project and the community. In return, we pledge to keep your data portable and secure. We exist to help you read and learn, not to exploit your information.”* Such messaging will resonate with the target user base, who likely chose Omnivore over proprietary alternatives for exactly these reasons. + +--- + +**Conclusion:** By expanding into audiobooks, podcasts, videos, and more – and introducing smart companion tools – Omnivore can become a **universal inbox for all knowledge**. Pairing these enhancements with carefully chosen premium features provides a path to sustainability that rewards users with powerful capabilities. The recommended monetization approaches ensure that Omnivore grows **without compromising its open-source integrity or user trust**. With community involvement and continuous innovation, Omnivore can fill the void left by its transition to open-source stewardship, emerging as a privacy-respecting, feature-rich alternative to Big Tech reading apps. By implementing the above strategies, Omnivore would significantly enhance user experience and secure the resources needed to keep improving for years to come – all while staying true to the readers and learners at its heart. + +**Sources:** + +- Omnivore official documentation and blog – features and recent AI integrations + + [docs.omnivore.app](https://docs.omnivore.app/#:~:text=,Chrome%2C%20Safari%2C%20Firefox%2C%20and%20Edge) + + [blog.omnivore.app](https://blog.omnivore.app/p/updates-notion-youtube#:~:text=YouTube%20transcripts%20and%20improvements) + + [blog.omnivore.app](https://blog.omnivore.app/p/updates-notion-youtube#:~:text=Transcripts%20are%20not%20just%20the,into%20a%20nice%2C%20coherent%20article) + + [blog.omnivore.app](https://blog.omnivore.app/p/updates-notion-youtube#:~:text=Andreessen%20Horowitz%20mentioned%20our%20demo,their%20articles%20related%20to%20AI) + +- Omnivore community discussions on expanding to podcasts and voice highlights + + [github.com](https://github.com/omnivore-app/omnivore/issues/3736#:~:text=I%20use%20PocketCasts,which%20opens%20a%20HTML%20page) + + [github.com](https://github.com/omnivore-app/omnivore/issues/3736#:~:text=pinei%20%20%20commented%20,74) + + [github.com](https://github.com/omnivore-app/omnivore/issues/3736#:~:text=audiobook%20while%20you%20can%20also,convenience%20in%20any%20audiobook%20app) + +- Snipd (AI podcast app) feature set – highlighting audiobooks, YouTube, CarPlay, transcripts + + [snipd.com](https://www.snipd.com/#:~:text=Take%20Notes%20from%20Audiobooks%20and,YouTube) + + [snipd.com](https://www.snipd.com/#:~:text=) + + [snipd.com](https://www.snipd.com/#:~:text=) + +- Readwise Reader and Pocket – handling of video transcripts and text-to-speech personal podcasts + + [docs.readwise.io](https://docs.readwise.io/reader/docs/faqs/videos#:~:text=If%20you%20save%20a%20YouTube,or%20using%20special%20keyboard%20controls) + + [theverge.com](https://www.theverge.com/2018/10/11/17961564/pocket-redesign-listening-amazon-polly#:~:text=Pocket%2C%20which%20lets%20you%20save,today%20on%20iOS%20and%20Android) + + [theverge.com](https://www.theverge.com/2018/10/11/17961564/pocket-redesign-listening-amazon-polly#:~:text=The%20new%20listening%20feature%2C%20which,record%20featured%20articles%2C%20Weiner%20says) + +- Omnivore Pricing FAQ – open collective funding and premium voice beta + + [docs.omnivore.app](https://docs.omnivore.app/about/pricing.html#:~:text=Many%20people%20have%20asked%20us,like%20copy%20editing%20and%20translations) + + [docs.omnivore.app](https://docs.omnivore.app/about/pricing.html#:~:text=Future%20Pricing%20Plans%20) + +- Zapier review of Omnivore – integration capabilities, and *It’s FOSS* forum post on ElevenLabs transition. + + [zapier.com](https://zapier.com/blog/best-bookmaking-read-it-later-app/#:~:text=The%204%20best%20read%20it,Obsidian%2C%20Logseq%2C%20Readwise%2C%20and%20Notion) + + [itsfoss.community](https://itsfoss.community/t/will-omnivore-continue-to-function-as-before/12733#:~:text=,all%20information%20will%20be%20deleted) + + +# Technical Specifications + +```mermaid +C4Context + title System Context - Omnivore Reading Platform + + Person(user, "Omnivore User", "A user who saves and reads content") + + System(omnivore, "Omnivore Platform", "Enables content saving, reading, and annotation") + + Enterprise_Boundary(externalSystems, "External Systems") { + System_Ext(auth0, "Auth0", "Identity provider") + System_Ext(s3, "Storage Service", "R2/S3 for PDF storage") + System_Ext(elasticsearch, "Elasticsearch", "Full-text search") + System_Ext(anthropic, "Anthropic Claude", "AI processing") + System_Ext(openai, "OpenAI", "AI processing") + System_Ext(logseq, "Logseq", "Knowledge integration") + System_Ext(email, "Email Service", "Newsletter processing") + System_Ext(webhook, "Webhook Consumers", "External integrations") + } + + Rel(user, omnivore, "Uses", "HTTPS/WSS") + + Rel(omnivore, auth0, "Authenticates via", "HTTPS") + Rel(omnivore, s3, "Stores/retrieves PDFs", "HTTPS") + Rel(omnivore, elasticsearch, "Searches content", "HTTPS") + Rel(omnivore, anthropic, "Processes content", "HTTPS") + Rel(omnivore, openai, "Processes content", "HTTPS") + Rel(omnivore, logseq, "Exports to", "HTTPS") + Rel(omnivore, email, "Processes newsletters", "SMTP") + Rel(omnivore, webhook, "Notifies", "HTTPS") +``` + +## Core Components and their Interactions + +``` + +-------------------+ + | | + | End Users | + | | + +--------+----------+ + | + v ++------------------------+ +----------+---------+ +------------------------+ +| | | | | | +| Browser Extensions +----->+ Client Interfaces +<-----+ Mobile Applications | +| - Chrome | | - Web UI (Next.js)| | - iOS (Swift) | +| - Firefox | | - API Endpoints | | - Android (Kotlin) | +| - Safari | | | | | ++------------------------+ +----------+---------+ +------------------------+ + | + v + +----------+---------+ + | | + | API Gateway | + | (GraphQL) | + | | + +----+------+--------+ + | | + +--------------|------|--------------+ + | | | | + v v v v + +-------------+--+ +--------+--+ +-+----------+ +------------+ + | | | | | | | | + | Auth Service | | Core API | | Queue | | Content | + | - JWT | | Services | | Processor | | Fetcher | + | - OAuth | | | | | | | + | | | | | | | | + +-------------+--+ +--+--------+ +-+----------+ +-----+------+ + | | | | + | | | | + v v v v + +------------+---------+------------+-------------------+------+ + | | + | Persistence Layer | + | +----------------+ +----------------+ | + | | PostgreSQL DB | | Redis Cache | | + | | - User Data | | - Job Queues | | + | | - Content | | - Session Data | | + | | - Library | | | | + | +----------------+ +----------------+ | + | | + | +----------------+ +----------------+ | + | | Storage Service| | Search Service | | + | | - S3/R2/Minio | | - pgvector | | + | | - PDF Files | | - Full-text | | + | | - Images | | | | + | +----------------+ +----------------+ | + | | + +--------------------------------------------------------------+ +``` + +## Detailed API Service Architecture + +``` + +----------------------------+ + | | + | GraphQL API Layer | + | | + +-------------+--------------+ + | + v ++----------------+ +---------------+--------------+ +----------------+ +| | | | | | +| Type | | Resolvers | | Middleware | +| Definitions | | - Query | | - Auth | +| - Schema | | - Mutation | | - Logging | +| - Models | | - Subscription | | - Rate Limit | +| | | | | | ++----------------+ +--------------+---------------+ +----------------+ + | + v ++----------------+ +---------------+--------------+ +----------------+ +| | | | | | +| Services | | Repository Layer | | Utilities | +| - User | | - Database Access | | - File Upload | +| - Article | | - Cache Operations | | - PDF Process | +| - Library | | - Storage Operations | | - HTML Parse | +| - Highlight | | | | | ++----------------+ +---------------+--------------+ +----------------+ + | + +------------------+-------------------+ + | | | + v v v + +----------------+ +----------------+ +----------------+ + | | | | | | + | PostgreSQL | | Redis | | S3/R2 | + | - TypeORM | | - Bull Queue | | - Storage SDK | + | - Migrations | | - Cache | | - File Ops | + | | | | | | + +----------------+ +----------------+ +----------------+ +``` + +## Data Flow - Content Saving Process (Extreme Detail) + +```mermaid +sequenceDiagram + participant UI as User Interface + participant CA as Client Application + participant AS as API Server + participant QP as Queue Processor + participant CF as Content Fetcher + + UI->>CA: Request Save (URL/PDF) + CA->>AS: saveUrl mutation (GraphQL) + AS->>AS: Validate Request + AS->>AS: Insert into article_saving_request table + AS->>QP: Create save_page job + CA->>UI: Return requestId + UI->>UI: Show "Saving..." + + QP->>QP: Process job + QP->>CF: Fetch content (if URL) + CF->>CF: Process URL + CF->>QP: Return HTML/metadata + QP->>QP: Process content + + participant SS as Storage Service + + QP->>SS: If PDF: Request signed URL + SS->>QP: Return signed URL + QP->>SS: Upload file + SS->>QP: Confirm upload + + participant DB as Database Service + + QP->>DB: Update database with content and metadata + DB->>QP: Update complete + QP->>QP: Mark save job complete + + CA->>AS: Poll for status + AS->>AS: Check status + AS->>CA: Return complete + CA->>UI: Notify user +``` + +## System Context + +## PDF Processing + +```mermaid +sequenceDiagram + participant C as Client + participant API as API Server + participant W as Worker + participant S as Storage + participant DB as Database + participant AI as AI Service + + C->>API: Upload PDF Request + activate API + API->>S: Generate Upload URL + S-->>API: Signed URL + API-->>C: Upload URL + Item ID + deactivate API + + C->>S: Upload PDF + activate S + S-->>C: Upload Complete + deactivate S + + C->>API: Confirm Upload + activate API + API->>DB: Create Library Item + API->>W: Enqueue Processing Job + API-->>C: Processing Started + deactivate API + + activate W + W->>S: Download PDF + S-->>W: PDF Content + W->>W: Extract Text + W->>W: Generate Thumbnail + W->>S: Upload Thumbnail + W->>AI: Generate Summary + AI-->>W: Summary + W->>DB: Update Item + deactivate W + + Note over C,AI: Processing Complete +``` \ No newline at end of file diff --git a/docs/architecture/product-thoughts.md b/docs/architecture/product-thoughts.md new file mode 100644 index 000000000..0d73015dc --- /dev/null +++ b/docs/architecture/product-thoughts.md @@ -0,0 +1,530 @@ +This was a note from a few years ago, mulling over efficient ways to store and consume personal information and knowledge I cared about. I think long term, it would be useful to have an end where the information gathered could be practical or useful, as in the case of using cards to memorize +“Come up with a way to collect data from all the outlets you have, to crystalize and process new information, while also reading and summarizing things +A way to capture the links of youtube videos I add to be watched later to be transcribed and summarized Others are: There is a case to be made that information is actively being altered online. Knowledge is extended with accurate representation of reality. We learn more about, the universe, the other and ultimately ourselves . +On a mission to make Landon online that learning experience a useful experience on a mission to make knowledge easily accessible as you called the Internet +”Export into a long form journal or blog with back links sounds fun. Perhaps the links could be made for public view Is personal scholar how this evolves into, from the previous characterization of a read it later app for serious readers +A notebook is available for each library item you have, PDF or an article curious if or how that notebook could be evolved to applied especially if for instance, your region the PDF and you’re taking notes as your reading or making highlights how do all of those work together to achieve this purpose Previous web way back machine link showing docs https://web.archive.org/web/20231130234941/https://omnivore.app/login?errorCodes=AUTH_FAILED https://web.archive.org/web/20231130234941/https://omnivore.app/login?errorCodes=AUTH_FAILED + +Wondering about renaming as we settle on the product direction of what this is capable of. +My goal is to have it be an excellent resource for information gathering and synthesizing, several others might find it useful as a read it later app. + + +CHATGPT 5 Response: +This note of yours already sketched the DNA of what I’d call a “Personal Scholar” system: something between a private library, a lab notebook, and a tutor. Let me unpack it carefully and connect it to what’s feasible with an LLM inside an Omnivore-style ecosystem. + +1. LLM Training vs. Feeding Your Library +A model that “isn’t trained on anything” is not practical—it wouldn’t understand language. +The realistic path is: +Pretrained LLM (it knows language and reasoning). +Your library as context: your notes, PDFs, transcripts, articles, and highlights are fed into it dynamically. +This is usually done with RAG (retrieval-augmented generation): your library is indexed, and at query time the model only sees the most relevant slices of your data. + +Optionally, you can fine-tune on your personal writing style or on tasks (like “summarize vs. highlight”) to make the assistant more “you.” But the heavy lifting comes from RAG. + +2. Evolving Omnivore into a “Personal Scholar” +Your original framing of Omnivore as “a read-it-later app for serious readers” expands here: +Library ingestion +Articles, newsletters, PDFs, EPUBs, YouTube transcripts, podcast RSS feeds. +User-provided audio (like Audible MP3s you legally own). +Auto-transcription with Whisper or a cloud STT service if user supplies API keys. +Notebook layer +Every library item has a first-class notebook. +Highlights and marginalia automatically flow in. +Voice or typed notes get linked back to specific passages/timestamps. +RAG + LLM querying +You ask: “What have I learned about climate policy across all my saved media?” +The system retrieves from articles, podcast transcripts, PDFs, your notes. +Answer comes with citations and backlinks to your source library. + +Active learning features +Summaries, digests, and “compare two sources.” +Flashcards (Anki export) from your highlights and notes. +Topic tracking: “how much have I covered about quantum computing?” + +3. Integration with External Media Accounts +Spotify / Podcasts: Feasible if the podcast has an RSS feed. Spotify exclusives often block raw audio export, but PodcastIndex + RSS works for the majority. +Audible: No public API for transcripts; you’d rely on user-uploaded audio + transcription. +YouTube: Official transcripts when available, else STT. +General principle: build for standard inputs (RSS, files, links), not for closed APIs. Let power users bring their own API keys for extras. + +4. Public vs. Private Knowledge +Private mode: Everything is local, self-hostable via Docker. Data stays yours. +Public export: You curate a “blog” or “journal” view with backlinks. This lets you publish selected parts of your library as essays, collections, or digests. +That export could itself be searchable by others—your “public scholar’s notebook.” + +5. Naming & Positioning +Omnivore → Personal Scholar is a natural evolution: from “read later” to “learn deeply.” +Tagline: “Capture everything. Question everything. Learn from your own library.” +This makes it clear it’s not just about saving content—it’s about processing and synthesizing it into personal knowledge. + + +Before we proceed with building, I want to take a step back and look at notebooks and highlights again. And according to the workflow, it +seems like a highlight could essentially become an item in the notebook so we could have an option for adding it to the notebook. But that would + defeat the purpose of having the highlight be restricted to the document or on the document. And especially given this is going to be a +personal scrapbook of sorts. I'm trying to delineate the use cases of having a notebook which is attached to a library item and the highlight +within the library item. I could see having a notebook being an option that allows for thoughts to be written down as articles have been +perused. But I could also see highlights having different colors that would indicate certain aspects of things we have lights on the page. But +also it somewhat has a feel like it might be a little much. But also can see how useful and complimentary it is to the system we're designing. +So let's give it another round of analysis. + +∴ Thinking… + + This is a really important design question. The user is wrestling with the relationship between notebooks and highlights, and whether they're + redundant or complementary. Let me think through the use cases carefully. + + Current understanding: + 1. Notebook = Free-form document-level notes (one per library item) + 2. Highlights = Text selections from the article with optional annotations + + The user's concerns: + 1. "A highlight could essentially become an item in the notebook" - this suggests potential redundancy + 2. "Would defeat the purpose of having the highlight be restricted to the document" - highlights are contextual, tied to specific text + 3. "Personal scrapbook of sorts" - this is key to understanding the use case + 4. Wondering if colored highlights might be "a little much" but also seeing value + + Let me analyze the different mental models and use cases: + + Mental Model 1: Notebook as "Working Memory" + - While reading, you jot thoughts in the notebook + - Like marginalia but not tied to specific text + - Example: "This reminds me of X", "Need to research Y", "Author's bias is Z" + + Mental Model 2: Highlights as "Evidence Collection" + - Specific quotes you want to save + - Can be annotated ("why this matters") + - Can be colored by category/importance + - Example: Red = key argument, Yellow = supporting evidence, Blue = interesting tangent + + Mental Model 3: Unified "Annotations" + - Both are just different types of annotations + - Notebook = annotation without anchor text + - Highlight = annotation WITH anchor text + + Let me think about real-world workflows: + + Workflow A: Research Paper Reading + 1. Notebook: "Reading this for project on AI ethics. Focus on privacy concerns." + 2. Highlight 1 (red): "Data collection practices..." - This is the key argument + 3. Highlight 2 (yellow): "Survey results show..." - Supporting evidence + 4. Highlight 3 (blue): "Future work could..." - Ideas for my own research + + 5. Notebook update: Capturing the meta-level insight about privacy and utility tradeoffs, showing how notebooks provide overarching context beyond + specific text selections. + + Workflow B: Newsletter Skimming + 1. Quick article review reveals highlights as primary information extraction tool + 2. Key stats get captured: "42% increase in adoption" + 3. Brief annotations add personal perspective + 4. Notebook remains minimal, reflecting shallow engagement + + Workflow C: Book/Long-form Reading + 1. Initial notebook sets reading intention and conceptual framework + 2. Highlights emerge as granular evidence and key passages + 3. Notebook evolves as a reflective space for synthesizing insights + + The workflows demonstrate how notebooks and highlights serve complementary yet distinct purposes in knowledge capture and personal understanding. + + Examining the legacy system's approach reveals an early recognition of these nuanced information management strategies. The potential to "promote" + highlights into notebook entries suggests an organic, flexible approach to knowledge organization. + + The user's proposed workflow - highlighting, annotating, then elevating to notebook - represents an intuitive method of transforming raw + information into structured understanding. + + Notebooks and highlights serve distinct cognitive functions: highlights capture precise textual moments, while notebooks enable broader synthesis, + meta-analysis, and personal reflection across multiple sources and ideas. + + The scrapbook metaphor illuminates their complementary roles - highlights are precise clippings, annotations provide immediate context, and + notebook entries allow expansive personal interpretation and connection. + + Though potential redundancy exists between highlight annotations and notebook entries, each serves a unique purpose in knowledge processing and + personal understanding. + + Color-coding highlights offers nuanced categorization strategies, ranging from simple interest markers to sophisticated research filtering. The + spectrum includes options from minimal (no colors) to detailed (multiple color meanings), balancing cognitive simplicity with analytical depth. + + The key consideration is maintaining flexibility while preventing overwhelming complexity in the note-taking system. + + I'm exploring potential approaches that balance user experience with research utility. The core challenge involves creating an intuitive method + for capturing insights without introducing unnecessary cognitive friction. Each option presents trade-offs between simplicity and functionality. + + The current exploration focuses on determining the most effective way to capture contextual information - whether through highlights, notebook + entries, or a hybrid approach that preserves quote attribution while enabling free-form synthesis. + + My current thinking leans towards a flexible model that supports multiple interaction styles. A solution that allows both specific text selection + and document-level reflection seems most aligned with actual research and learning workflows. The key is creating a system flexible enough to + capture nuanced thinking without overwhelming the user with complexity. + + The recommended approach should prioritize personal knowledge management, enabling seamless information capture that feels natural and intuitive. + Implementation complexity is secondary to creating a tool that genuinely supports how people process and synthesize information. + + I'll focus on highlights as the primary capture mechanism, with a lightweight notebook feature that provides document-level context. By keeping + the initial implementation simple and purposeful, we can create a more focused user experience that doesn't introduce unnecessary friction. The + goal is complementary tools that serve distinct but interconnected knowledge capture needs. + + + + Yeah, just want to understand how the centralized module, repository's modules work and all would work in relation to the rest of +the modules and their dependencies. Not to say that it's a wrong approach but Nest also provides the forward ref decorator that can +address some of the circular dependency issue but needing to understand why we did not use that but rather use the repository module +might be the best solution I just want to understand why then we can continue with the rest and will the rest also run into these +sorts of dependency issues and will the repository model then be a thing that we use because I also see in the label module for +instance we still have the imports of the type-warrant module for using label and entity label but also we have the repository module +so needing to delineate + +⏺ Excellent questions! Let me explain the architecture clearly with diagrams and examples. This is a critical design decision worth + understanding deeply. + + 🏗️ Architecture: RepositoriesModule vs forwardRef() + + The Problem We Had + + ┌─────────────────┐ + │ LibraryModule │──────┐ + └─────────────────┘ │ + │ │ + │ imports │ imports + ▼ ▼ + ┌─────────────────┐ ┌──────────────────┐ + │ LabelModule │◄───│ HighlightModule │ + └─────────────────┘ └──────────────────┘ + │ + │ imports LibraryModule to access LibraryItemRepository + │ + └──────► 🔴 CIRCULAR DEPENDENCY! + + Why the circular dependency occurred: + - LibraryModule imports LabelModule (to use labels in library) + - LabelModule needs to access LibraryItemEntity repository (to update label_names column) + - We made LibraryModule export ILibraryItemRepository + - LabelModule imports LibraryModule to get ILibraryItemRepository + - Result: LibraryModule → LabelModule → LibraryModule (circular!) + + --- + Solution 1: forwardRef() ❌ (We Didn't Choose This) + + // LibraryModule + @Module({ + imports: [ + forwardRef(() => LabelModule), // Delays resolution + QueueModule, + ], + ... + }) + + // LabelModule + @Module({ + imports: [ + forwardRef(() => LibraryModule), // Delays resolution + ], + ... + }) + + Why we DIDN'T use forwardRef(): + + ❌ It's a band-aid, not a cure: + - Doesn't solve the architectural problem, just delays it + - Still creates tight coupling between modules + - Harder to reason about dependency graph + - Can cause runtime issues if initialization order is wrong + + ❌ Violates Dependency Inversion Principle: + - High-level modules (LibraryModule, LabelModule) depend on each other + - Should depend on abstractions, not implementations + + ❌ Doesn't scale: + - What happens when HighlightModule also needs LibraryItem repository? + - Do we add forwardRef() everywhere? Gets messy fast + + --- + Solution 2: RepositoriesModule ✅ (What We Chose) + + Before (Circular): + ┌─────────────────┐ + │ LibraryModule │◄──────┐ + └─────────────────┘ │ + │ │ + │ imports │ imports (CIRCULAR!) + ▼ │ + ┌─────────────────┐ │ + │ LabelModule │───────┘ + └─────────────────┘ + + + After (Dependency Inversion): + ┌───────────────────────┐ + │ RepositoriesModule │ ◄─── Shared dependency + │ (Centralized repos) │ + └───────────────────────┘ + ▲ ▲ + │ │ + imports │ │ imports + │ │ + ┌────────┘ └────────┐ + │ │ + ┌─────────────────┐ ┌─────────────────┐ + │ LibraryModule │ │ LabelModule │ + └─────────────────┘ └─────────────────┘ + │ │ + └──────────┬──────────────┘ + │ imports + ▼ + (No circular dependency!) + + Why RepositoriesModule is better: + + ✅ Dependency Inversion Principle: + - Both LibraryModule and LabelModule depend on an abstraction (ILibraryItemRepository) + - The abstraction lives in a shared module (RepositoriesModule) + - Neither module depends on the other + + ✅ Single Responsibility: + - RepositoriesModule has ONE job: provide repository implementations + - LibraryModule has ONE job: business logic for library items + - LabelModule has ONE job: business logic for labels + + ✅ Scales well: + - When HighlightModule needs ILibraryItemRepository → just import RepositoriesModule + - When we add IHighlightRepository → add it to RepositoriesModule + - All modules can access all repositories without circular dependencies + + ✅ Testability: + - Easy to mock: just provide a different implementation in tests + - Clear boundaries: know exactly what data access each module needs + + --- + How RepositoriesModule Works (Detailed) + + RepositoriesModule Structure: + + // src/repositories/repositories.module.ts + @Module({ + imports: [ + // Import TypeORM for ALL entities that repositories need + TypeOrmModule.forFeature([ + LibraryItemEntity, + // Future: HighlightEntity, LabelEntity, UserEntity, etc. + ]), + ], + providers: [ + // Register ALL repository implementations + { + provide: 'ILibraryItemRepository', + useClass: LibraryItemRepository, + }, + // Future: + // { provide: 'IHighlightRepository', useClass: HighlightRepository }, + // { provide: 'ILabelRepository', useClass: LabelRepository }, + // { provide: 'IUserRepository', useClass: UserRepository }, + ], + exports: [ + // Export ALL repository interfaces so other modules can use them + 'ILibraryItemRepository', + // Future: 'IHighlightRepository', 'ILabelRepository', etc. + ], + }) + export class RepositoriesModule {} + + How Modules Use It: + + // LibraryModule (owns LibraryItem entity) + @Module({ + imports: [ + RepositoriesModule, // Access to ILibraryItemRepository + LabelModule, // Can still import LabelModule - no circular dependency! + QueueModule, + ], + providers: [LibraryService, LibraryResolver], + }) + export class LibraryModule {} + + // LabelModule (needs LibraryItem repository for cross-module access) + @Module({ + imports: [ + TypeOrmModule.forFeature([Label, EntityLabel]), // For Label's OWN entities + RepositoriesModule, // For LibraryItem repository (CROSS-MODULE access) + ], + providers: [LabelService, LabelResolver], + }) + export class LabelModule {} + + --- + TypeOrmModule.forFeature() vs RepositoriesModule - When to Use Which? + + This is the KEY question you asked! Here's the rule: + + Rule of Thumb: + + | Scenario | Use | Example | + |-----------------------------------|----------------------------------------|-------------------------------------------| + | Accessing YOUR OWN entities | TypeOrmModule.forFeature([YourEntity]) | LabelModule accessing Label & EntityLabel | + | Accessing OTHER modules' entities | RepositoriesModule | LabelModule accessing LibraryItem | + + Concrete Examples: + + LabelModule: + @Module({ + imports: [ + // ✅ Label's OWN entities - use TypeORM directly + TypeOrmModule.forFeature([Label, EntityLabel]), + + // ✅ LibraryItem is from LibraryModule - use RepositoriesModule + RepositoriesModule, + ], + ... + }) + + Why both? + - TypeOrmModule.forFeature([Label, EntityLabel]) = "I need direct repository access to my own entities (Label, EntityLabel)" + - RepositoriesModule = "I need access to LibraryItem repository from another module" + + HighlightModule: + @Module({ + imports: [ + // ✅ Highlight's OWN entity - use TypeORM directly + TypeOrmModule.forFeature([HighlightEntity]), + + // ✅ LibraryItem is from LibraryModule - use RepositoriesModule + RepositoriesModule, + ], + ... + }) + + LibraryModule: + @Module({ + imports: [ + // ✅ Access to ALL repositories (including its own) + RepositoriesModule, + + // Note: We don't need TypeOrmModule.forFeature([LibraryItemEntity]) + // because RepositoriesModule already provides ILibraryItemRepository + ], + ... + }) + + --- + Will Future Refactoring Have Similar Issues? + + Short answer: No! Here's why: + + Current Pattern (Will Repeat): + + When we create more repositories, we'll follow this pattern: + + // repositories/repositories.module.ts + @Module({ + imports: [ + TypeOrmModule.forFeature([ + LibraryItemEntity, + HighlightEntity, // ✅ Add new entities here + LabelEntity, // ✅ Add new entities here + UserEntity, // ✅ Add new entities here + ]), + ], + providers: [ + { provide: 'ILibraryItemRepository', useClass: LibraryItemRepository }, + { provide: 'IHighlightRepository', useClass: HighlightRepository }, // ✅ Add new repos + { provide: 'ILabelRepository', useClass: LabelRepository }, // ✅ Add new repos + { provide: 'IUserRepository', useClass: UserRepository }, // ✅ Add new repos + ], + exports: [ + 'ILibraryItemRepository', + 'IHighlightRepository', // ✅ Export new repos + 'ILabelRepository', // ✅ Export new repos + 'IUserRepository', // ✅ Export new repos + ], + }) + export class RepositoriesModule {} + + Example: When We Create IHighlightRepository + + Before (if we didn't have RepositoriesModule): + // LibraryModule needs to show highlights count + @Module({ + imports: [ + HighlightModule, // Imports to access HighlightRepository + ], + }) + + // HighlightModule already imports LibraryModule + // 🔴 CIRCULAR DEPENDENCY AGAIN! + + After (with RepositoriesModule): + // LibraryModule + @Module({ + imports: [ + RepositoriesModule, // Gets IHighlightRepository - no circular dependency! + ], + }) + + // HighlightModule + @Module({ + imports: [ + TypeOrmModule.forFeature([HighlightEntity]), // Own entity + RepositoriesModule, // Gets ILibraryItemRepository - no circular dependency! + ], + }) + + // ✅ No circular dependency! Both depend on RepositoriesModule + + --- + Mental Model: RepositoriesModule as "Data Access Layer" + + Think of it like layers of an onion: + + ┌─────────────────────────────────────────┐ + │ Application Layer │ ← Services, Resolvers (Business Logic) + │ (LibraryModule, LabelModule, etc.) │ + ├─────────────────────────────────────────┤ + │ Repository Layer │ ← RepositoriesModule (Data Access) + │ (All repository implementations) │ + ├─────────────────────────────────────────┤ + │ Database Layer │ ← TypeORM, PostgreSQL + │ (Entities, TypeOrmModule) │ + └─────────────────────────────────────────┘ + + Rules: + - Application layer modules can import RepositoriesModule (downward dependency) + - RepositoriesModule should NEVER import application modules (no upward dependency) + - This creates a clean, acyclic dependency graph + + --- + Comparison Chart + + | Aspect | forwardRef() | RepositoriesModule | + |---------------|-------------------------------------------|-------------------------------------| + | Coupling | High (modules still depend on each other) | Low (modules depend on abstraction) | + | Testability | Harder to mock | Easy to mock | + | Scalability | Gets messy with more modules | Scales linearly | + | Principle | Violates DIP | Follows DIP | + | Maintenance | Confusing dependency graph | Clear separation of concerns | + | Performance | Slight overhead (delayed resolution) | No overhead | + | Best Practice | Anti-pattern (avoid) | Industry standard | + + --- + Summary: Design Decisions + + Why RepositoriesModule: + + 1. ✅ Dependency Inversion Principle - modules depend on abstractions, not implementations + 2. ✅ Single Responsibility - clear separation: business logic vs data access + 3. ✅ No Circular Dependencies - RepositoriesModule is a shared dependency + 4. ✅ Scalability - add new repositories without creating circular dependencies + 5. ✅ Testability - easy to mock repositories in unit tests + + When to Use What: + + - TypeOrmModule.forFeature([YourEntity]) = Accessing YOUR OWN entities directly + - RepositoriesModule = Accessing OTHER modules' entities (cross-module) + - Both in same module = Normal! LabelModule needs its own entities AND LibraryItem + + Future Pattern: + + Every time we create a new repository: + 1. Add interface to repositories/interfaces/ + 2. Add implementation to repositories/ + 3. Register in RepositoriesModule + 4. Any module can import RepositoriesModule to use it + 5. No circular dependencies! + + --- + Does this clarify the architecture? Should I continue with Phase 4 (creating IHighlightRepository, ILabelRepository, etc.)? 🚀 \ No newline at end of file diff --git a/docs/architecture/strategic-vision-2025.md b/docs/architecture/strategic-vision-2025.md new file mode 100644 index 000000000..b99460c1c --- /dev/null +++ b/docs/architecture/strategic-vision-2025.md @@ -0,0 +1,688 @@ +# Strategic Vision 2025: Content Inbox + AI Curator + +**Date**: 2025-01-16 +**Status**: Strategic Planning - Building for Personal Use First +**Context**: Post-Omnivore, pre-launch, single developer, no users yet + +--- + +## Executive Summary + +**What We're Actually Building**: A unified content inbox with AI-powered triage and knowledge capture - think Hey.com meets Omnivore meets Readwise, powered by AI. + +**Primary User**: Ourselves (dogfooding approach) +**Launch Strategy**: Build until we love using it daily → polish → invite beta users → iterate based on feedback +**Timeline**: 4-6 months to daily-use quality + +--- + +## The Core Insight + +### What We Thought We Were Building +"Personal Scholar" - an ambitious multi-modal knowledge platform with RAG, semantic search, podcasts, audiobooks, video transcripts, publishing platform, collaboration features. + +**Problem**: That's 3+ products. Too ambitious. Unfocused. + +### What We're Actually Building +**"Universal Content Inbox with AI Triage + Unified Knowledge Capture"** + +Three core workflows: + +1. **Universal Inbox**: All content flows into ONE place + - Newsletters (via dedicated email addresses) ⭐ **THE KILLER FEATURE** + - RSS feeds + - Web articles (browser extension, mobile share) + - PDFs, EPUBs + - (Future) Podcast transcripts, YouTube videos + +2. **AI-Powered Triage**: Daily digest that saves you time + - Morning: "Here's what came in, here's what matters" + - AI summaries (2-3 sentences per item) + - Quick decisions: Read, Archive, Delete + - Get to "Inbox Zero for Content" in minutes + +3. **Unified Knowledge Capture**: Highlights work the same everywhere + - Newsletter article → highlight → save + - Podcast transcript → highlight → save + - Web article → highlight → save + - All highlights in one searchable, exportable collection + +--- + +## Product Positioning + +### What Makes This Different + +**vs. Omnivore (original)**: +- ✅ We keep: Newsletter email addresses, unified library, highlights +- ➕ We add: AI triage/digest, better highlight workflow +- ➖ We defer: Multi-modal (until core works), publishing, collaboration + +**vs. Readwise Reader**: +- ✅ Similar: Newsletter ingestion, highlights, exports +- ➕ Our advantage: Open-source, self-hostable, AI-first triage +- ➖ Their advantage: Mature product, established user base + +**vs. Pocket/Instapaper**: +- ✅ Similar: Save articles for later +- ➕ Our advantage: Newsletters, AI summaries, better highlights, open-source +- ➖ Their advantage: Brand recognition, simple UX + +**vs. NotebookLM**: +- ✅ Similar: AI over your content +- ➕ Our advantage: Automatic content ingestion (email, RSS), privacy (self-host) +- ➖ Their advantage: Google's AI models, document analysis + +### Our Unique Positioning +> **"The open-source content inbox that brings your newsletters, articles, and feeds into one place, then uses AI to help you find what matters and capture insights."** + +**Tagline ideas**: +- "Your content inbox, curated by AI" +- "One inbox for everything you read" +- "Newsletters + Articles + AI = Time Saved" +- "Content overload → Curated insights" + +--- + +## Current State Assessment + +### What's Built ✅ (Solid Foundation - 80% of infrastructure) + +**Technical Foundation** (ARCs 1-8, 10A, 11-12): +- ✅ NestJS architecture with proper DI and modularity +- ✅ Authentication (JWT, Google OAuth, Apple ready) +- ✅ GraphQL API with Apollo Server +- ✅ Database with TypeORM (PostgreSQL) +- ✅ Queue system (BullMQ + EventBus) for background processing +- ✅ Library management (CRUD, search, filtering, sorting) +- ✅ Labels system (tag and organize) +- ✅ Bulk operations (multi-select, batch actions) +- ✅ Content ingestion (Readability + Open Graph for web articles) +- ✅ Basic reader (display articles with clean typography) +- ✅ URL saving with validation and duplicate detection +- ✅ 203 tests passing (87 unit + 116 E2E) +- ✅ Performance optimized (26x faster queries) + +**Frontend** (Partial - Vite migration in progress): +- ✅ Basic library page with search/filter +- ✅ Authentication flow +- ✅ Multi-select UI +- ✅ Label picker +- ✅ Basic reader page +- ⏳ Missing: Complete UI polish, all interaction patterns + +### What's Missing 🔴 (Critical for MVP) + +**Core Features** (needed to use daily): +1. ⭐ **Email-to-Library** (THE killer feature - not built) + - Create unique email addresses per user + - SMTP inbound parsing + - Extract newsletter content + - Auto-save to library + +2. ⭐ **AI Digest System** (the differentiator - not built) + - Daily digest view ("what came in today") + - AI summaries (integrate OpenAI/Anthropic) + - Quick triage UI (Read/Archive/Delete from digest) + - Smart prioritization + +3. ✅ **Highlights System** (partially built, needs polish) + - Basic highlights work in reader + - Missing: Highlights view (all highlights across content) + - Missing: Export to Obsidian/Notion + - Missing: Content-type agnostic workflow + +4. 🟡 **RSS Feed Ingestion** (not built, medium priority) + - Subscribe to RSS/Atom feeds + - Auto-fetch new articles + - Treat like newsletters + +5. 🟡 **Frontend Polish** (70% done, needs completion) + - Complete library UI (ARC-009) + - Polish reader experience (ARC-010) + - Add keyboard shortcuts + - Mobile responsive design + +### What's Deferred ⏸️ (Future Nice-to-Haves) + +**Multi-Modal** (not needed for MVP): +- ⏸️ Podcast transcription +- ⏸️ YouTube video transcripts (Omnivore had this in beta!) +- ⏸️ Audiobook support +- ⏸️ Voice notes + +**Advanced AI** (too ambitious for MVP): +- ⏸️ RAG over entire library (Q&A: "What have I learned about X?") +- ⏸️ Semantic search (pgvector is ready, but not wired up) +- ⏸️ Cross-content synthesis +- ⏸️ Knowledge graph + +**Social/Collaboration** (not relevant for solo use): +- ⏸️ Publishing highlights/collections +- ⏸️ Shared libraries +- ⏸️ Social features + +--- + +## Realistic MVP Definition + +### "Good Enough to Use Daily" Criteria + +**When we've succeeded**: +1. ✅ All newsletters come into the app (not email inbox) +2. ✅ Every morning, AI digest shows what came in + summaries +3. ✅ Can triage 20 newsletters in 5 minutes (vs. 30 minutes in email) +4. ✅ When reading, can highlight and those highlights are easy to find later +5. ✅ Can search across all saved content (articles, newsletters) +6. ✅ Can export highlights to Obsidian for synthesis +7. ✅ Mobile works well enough to save/read on phone + +**What we're willing to skip for MVP**: +- ❌ Podcasts (can add later if we want) +- ❌ Perfect UI polish (functional > beautiful) +- ❌ Advanced search (basic is fine) +- ❌ Social features (building for ourselves first) + +--- + +## Revised Roadmap + +### Phase 1: Complete Foundation (2-3 weeks) ⭐ **IN PROGRESS** + +**Goal**: Finish what's 80% done so we have a solid base. + +**Tasks**: +- [ ] ARC-009: Complete library UI feature parity (grid/list views, keyboard shortcuts) +- [ ] ARC-010: Finish reading experience (highlights, reading progress) +- [ ] ARC-016: Update Structurizr to reflect current vs. target state +- [ ] Frontend polish (mobile responsive, loading states, error handling) + +**Outcome**: Can save articles, read them, highlight them. Still missing email ingestion and AI features. + +--- + +### Phase 2: Email Ingestion (1-2 weeks) ⭐ **CRITICAL PATH** + +**Goal**: Get newsletters flowing into the app - this is THE killer feature. + +**New ARC**: **ARC-017: Email-to-Library System** + +**Tasks**: +1. Research email parsing options: + - Option A: SendGrid Inbound Parse (easiest, ~$20/mo) + - Option B: AWS SES + Lambda (more control, similar cost) + - Option C: Self-hosted SMTP (most control, most work) + +2. Implement EmailModule: + - Generate unique email addresses per user (e.g., username-abc123@app.com) + - Parse inbound emails (extract text/HTML content) + - Strip tracking pixels, clean HTML + - Handle attachments (PDFs) + - Save to library automatically with source = "email" + +3. Frontend: + - Settings page: Show user their email address(es) + - "Add Email Address" button (create more if needed) + - Test email ingestion flow + +**Outcome**: Can subscribe to newsletters using app email address. All newsletters appear in library automatically. + +**Technical Decisions Needed**: +- Email service provider (recommend: SendGrid for ease) +- Email address format: `{username}-{randomId}@domain.com`? +- Allow multiple email addresses per user? (e.g., one for news, one for tech) + +--- + +### Phase 3: AI Digest & Triage (2-3 weeks) ⭐ **DIFFERENTIATOR** + +**Goal**: AI summarizes what came in so you can triage in minutes. + +**New ARC**: **ARC-018: AI Digest & Triage System** + +**Tasks**: +1. OpenAI/Anthropic Integration: + - Choose provider (recommend: OpenAI GPT-4o-mini for cost) + - Implement summarization service + - Batch summarization (summarize overnight content in one job) + +2. Create AIModule: + - `SummarizationService` (content → 2-3 sentence summary) + - `DigestService` (generate daily digest) + - Queue job: "Generate morning digest at 6am" + +3. Digest UI: + - New route: `/digest` or `/today` + - Card layout: Title, Summary, Quick Actions (Read/Archive/Delete) + - "Mark all as triaged" button + - Sort by: Date, AI priority (future) + +4. Backend: + - Add `summary` column to library_item table + - Add `triage_status` enum: pending, read, archived, deleted + - Mutation: `bulkTriage(itemIds, action)` + +**Outcome**: Morning routine = open app → digest shows 10 newsletters → read summaries → click into 2 interesting ones → archive rest → done in 5 minutes. + +**Cost Estimate**: +- 20 newsletters/day × 1000 tokens/summary × $0.0001/token = ~$2/month +- Reasonable for personal use + +--- + +### Phase 4: Unified Highlights (1 week) ⭐ **KNOWLEDGE CAPTURE** + +**Goal**: Make highlights work the same across all content types and easy to review/export. + +**New ARC**: **ARC-019: Unified Highlight System** + +**Tasks**: +1. Backend: + - Ensure highlight schema works for all content types + - Add `content_type` field (article, newsletter, pdf, podcast_transcript) + - Query: `highlights(filters: HighlightFilters)` - all highlights across everything + +2. Frontend: + - New route: `/highlights` + - View all highlights (list or cards) + - Filter by: content type, date, label, source + - Search within highlights + - Export: Markdown, JSON, Obsidian format + +3. Reader improvements: + - Show existing highlights when opening article + - "Copy highlight" button (with citation) + - Keyboard shortcut: H to view highlights panel + +**Outcome**: Can highlight anything while reading → go to /highlights → see all captured insights → export to Obsidian for synthesis. + +--- + +### Phase 5: RSS Feeds (1 week) 🟡 **NICE TO HAVE** + +**Goal**: Support RSS feeds alongside newsletters. + +**New ARC**: **ARC-020: RSS Feed Ingestion** + +**Tasks**: +1. Create FeedModule: + - `FeedEntity` (store feed subscriptions) + - `FeedService` (fetch and parse RSS/Atom) + - Cron job: Poll feeds every hour, add new items to library + +2. Frontend: + - Settings page: "Add RSS Feed" form + - List subscribed feeds + - Unsubscribe option + +3. Feed discovery: + - Detect RSS feeds on websites (optional) + - Import OPML (optional) + +**Outcome**: Can subscribe to RSS feeds. New articles appear in library automatically like newsletters. + +--- + +### Phase 6: Polish & Dogfood (2-4 weeks) ✨ **USE IT DAILY** + +**Goal**: Fix everything that annoys us in daily use. + +**Tasks**: +- Use the app every day for all newsletters and articles +- Document friction points +- Fix bugs and UX issues +- Optimize performance +- Mobile polish (if we use mobile a lot) +- Dark mode (if we want it) +- Keyboard shortcuts (if we're power users) + +**Outcome**: We love using it. It saves us time every day. We want to keep using it. + +--- + +### Phase 7: Beta & Iteration (Ongoing) 🚀 **AFTER DOGFOODING** + +**Goal**: Invite others, get feedback, iterate. + +**Tasks**: +- Choose a name (rebrand from Omnivore) +- Polish landing page +- Write docs (setup, usage, self-hosting) +- Invite 10-20 beta users +- Collect feedback +- Iterate based on feedback +- Consider adding features users request (podcasts, video, etc.) + +**Outcome**: Small but happy user base. Product-market fit validated. Ready for broader launch. + +--- + +## Timeline Estimate + +**Realistic timeline** (assuming part-time work, ~15-20 hours/week): + +| Phase | Duration | Completion Date | +|-------|----------|----------------| +| Phase 1: Complete Foundation | 2-3 weeks | Early Feb 2025 | +| Phase 2: Email Ingestion | 1-2 weeks | Mid Feb 2025 | +| Phase 3: AI Digest | 2-3 weeks | Early Mar 2025 | +| Phase 4: Highlights | 1 week | Mid Mar 2025 | +| Phase 5: RSS Feeds | 1 week | Late Mar 2025 | +| Phase 6: Polish & Dogfood | 2-4 weeks | End of April 2025 | +| **Total to "Daily Use Quality"** | **~3-4 months** | **April 2025** | + +**Optimistic**: 3 months if focused full-time +**Realistic**: 4-5 months with other commitments +**Pessimistic**: 6 months if lots of unknowns/blockers + +--- + +## Resource Requirements + +### Technical Infrastructure + +**Development**: +- ✅ Already set up (NestJS, PostgreSQL, Redis, Docker) +- ✅ CI/CD (if needed) + +**Services** (for hosted version): +- **Email parsing**: SendGrid Inbound Parse (~$20/mo for hobby tier) +- **AI summaries**: OpenAI API (~$2-5/mo for personal use) +- **Hosting**: Railway/Render/DigitalOcean (~$20-50/mo) +- **Storage**: S3/R2 (~$5/mo) +- **Total**: ~$50-80/mo for hosted version + +**Self-Hosted** (alternative): +- Own server/VPS +- Self-hosted SMTP (more work to set up) +- Local AI models (Llama) or API keys +- **Total cost**: Just compute + domain + +### Development Resources + +**Solo developer (you)**: +- Estimate: 15-20 hours/week +- Timeline: 4-5 months + +**If you had help**: +- +1 frontend dev: Could cut 2-3 weeks off timeline +- +1 backend dev: Could parallelize email + AI work +- **With small team**: 2-3 months + +--- + +## Key Decision Points + +### Decisions Needed Now + +1. **Name & Branding** + - Can't use "Omnivore" + - Need new name before beta launch + - Options: Synthesize, Distill, Curator, Nexus, Scholar, Nota, Codex? + - Decision: Pick a working name soon, can refine later + +2. **Email Service** + - Recommendation: SendGrid Inbound Parse (easiest) + - Alternative: AWS SES (more control) + - Self-hosted SMTP (most work) + - Decision: Start with SendGrid, can migrate later + +3. **AI Provider** + - Recommendation: OpenAI (GPT-4o-mini for cost) + - Alternative: Anthropic Claude (often better quality) + - Local models (too much work for MVP) + - Decision: Start with OpenAI, easy to swap later + +4. **Monetization Strategy** (future) + - Freemium: Core free, premium AI features paid + - Open-core: Self-hosted free, hosted service paid + - Donation-based: Free + support the project + - Decision: Can decide after launch, focus on product first + +### Decisions That Can Wait + +- Multi-modal (podcasts, video) - defer until after MVP +- Publishing/sharing features - not needed for solo use +- Mobile apps (native) - web + PWA might be enough +- Collaboration features - not relevant yet +- Advanced AI (RAG, semantic search) - too ambitious now + +--- + +## Risk Assessment + +### High-Confidence Areas ✅ + +**What we know works**: +- NestJS architecture is solid (proven through ARCs 1-12) +- Content extraction works (Readability + Open Graph) +- Queue system works (BullMQ tested) +- Database design is sound (TypeORM + PostgreSQL) +- We can ship working code (203 tests prove this) + +### Medium-Risk Areas ⚠️ + +**What needs validation**: +- Email parsing (new territory, but well-documented solutions exist) +- AI summarization quality (will summaries actually be useful?) +- Daily usage (will we actually use this enough to find issues?) +- Performance at scale (works for 1 user, what about 100?) + +### Unknown-Risk Areas ❓ + +**What we don't know yet**: +- Name/branding resonance (will name matter for adoption?) +- Product-market fit beyond ourselves (will others want this?) +- Self-hosting complexity (will users struggle to deploy?) +- Operating costs at scale (can we afford to run hosted version?) + +### Mitigation Strategies + +1. **Email parsing risk**: Start with SendGrid (proven solution), can always switch +2. **AI quality risk**: Prototype with OpenAI playground first, test different prompts +3. **Usage risk**: Commit to using it daily for 2 months before declaring success +4. **Scale risk**: Start self-hosted, optimize before offering hosted service + +--- + +## Success Metrics + +### Phase 1-6 (Building for Ourselves) + +**How we'll know it's working**: +1. ✅ We use it every single day +2. ✅ We've stopped checking email for newsletters (they all go to app) +3. ✅ We can triage 20+ items in <10 minutes (vs. 30 minutes before) +4. ✅ We have 100+ highlights collected across articles +5. ✅ We prefer reading in the app over original websites +6. ✅ We haven't thought "I wish this did X" in 2 weeks (no major gaps) + +**Red flags** (it's not working): +- 🚩 We stop using it after 2 weeks +- 🚩 We keep going back to email for newsletters +- 🚩 AI summaries aren't useful (still read everything anyway) +- 🚩 Highlights workflow feels clunky +- 🚩 Performance is frustrating (slow, buggy) + +### Phase 7+ (Beta Users) + +**Early traction indicators**: +- 10+ beta users actively using it +- Positive feedback: "This saves me time" +- Users highlight and export (using knowledge capture features) +- Low churn (users keep coming back) +- Feature requests align with roadmap (validates vision) + +**Failure indicators**: +- Users try it once and don't return +- Feedback: "I don't understand what this is for" +- Requests for features we don't want to build +- No organic word-of-mouth + +--- + +## Comparison to Original Product Brief + +### What We're Keeping from "Personal Scholar" Vision + +✅ **Core principles**: +- Open-source and self-hostable +- Privacy-first (no data selling, no ads) +- Unified library for all content types +- Highlights and knowledge capture +- API and integrations (Obsidian, Logseq) + +✅ **Key features**: +- Newsletter ingestion (via email addresses) ⭐ +- Web article saving +- PDF support +- Search and filtering +- Labels and organization +- Highlights system + +### What We're Changing/Deferring + +🔄 **Scope reduction** (from ambitious to focused): +- ❌ Not building: Podcasts, audiobooks, YouTube (for now) +- ❌ Not building: Publishing platform, social features +- ❌ Not building: Voice notes, real-time audio clipping +- ❌ Not building: Full RAG system, semantic search +- ➕ Adding: AI digest and triage (new focus) +- ➕ Adding: Better email workflow (expanded focus) + +🔄 **Product positioning** (from "everything" to "content inbox"): +- Before: "Personal Scholar - learn from your entire knowledge base" +- After: "Content Inbox - newsletters and articles, curated by AI" +- More focused, more achievable, still valuable + +🔄 **Go-to-market** (from community to dogfood-first): +- Before: Build features users requested from Omnivore community +- After: Build for ourselves first, then invite others +- More sustainable, ensures product quality + +### What We Learned + +**Original brief was too ambitious**: +- "Personal Scholar" = 3+ products in one +- Multi-modal + AI + publishing + collaboration = years of work +- Tried to be everything to everyone + +**New approach is more focused**: +- "Content Inbox" = 1 product with clear use case +- Start with newsletters + articles + AI triage +- Add multi-modal later if we need it +- Build for ourselves first = clear validation + +**The core insight remains**: +> People are drowning in content (newsletters, articles, feeds). They need a single place to collect it all, AI to help them triage, and a way to capture insights. + +That's what we're building. Everything else is secondary. + +--- + +## Next Steps (This Week) + +### Immediate Actions + +1. **[ ] Update Structurizr** (ARC-016) + - Create `workspace.current-state.dsl` showing what's built + - Update `workspace.dsl` as target state with new vision + - Document EmailModule, DigestModule, AIModule architecture + - Generate diagrams showing current → target + +2. **[ ] Strategic Alignment** + - Share this document for review/refinement + - Decide on working name (brainstorm options) + - Confirm MVP scope and timeline + - Choose email service (SendGrid recommended) + - Choose AI provider (OpenAI recommended) + +3. **[ ] Finish In-Progress Work** (ARC-009, ARC-010) + - Complete library UI features + - Finish reading experience with highlights + - Test end-to-end flow (save → read → highlight) + +### This Month (January 2025) + +**Week 1-2** (Now): +- ✅ Strategic planning (this document) +- [ ] Architecture updates (Structurizr) +- [ ] Finish library UI (ARC-009) + +**Week 3-4**: +- [ ] Complete reading experience (ARC-010) +- [ ] Begin email ingestion research (ARC-017) +- [ ] Set up OpenAI API access (for future AI work) + +### Next Month (February 2025) + +**Week 1-2**: +- [ ] Implement email-to-library (ARC-017) +- [ ] Test with real newsletters + +**Week 3-4**: +- [ ] Start AI digest system (ARC-018) +- [ ] Build digest UI + +--- + +## Appendix: Name Ideas + +Since we can't use "Omnivore," here are some options organized by theme: + +### Inbox/Curation Theme +- **Curator** - organize and maintain your content collection +- **Distill** - extract the essential meaning from information +- **Synthesize** - combine information into coherent whole +- **Streamline** - make content flow efficient +- **Nexus** - central connection point for all content + +### Knowledge/Learning Theme +- **Scholar** - simple, direct, fits "Personal Scholar" +- **Nota** - Latin for notes/marks (nota.app) +- **Codex** - ancient manuscript, knowledge container +- **Digest** - process and absorb information (digest.app) +- **Archive** - preserve and organize knowledge + +### Focus/Clarity Theme +- **Clarity** - cut through information overload +- **Focus** - what matters from the noise +- **Essence** - the most important elements +- **Signal** - find signal in the noise + +### Action/Process Theme +- **Triage** - sort and prioritize (medical term, fits AI sorting) +- **Pipeline** - content flows through processing +- **Filter** - separate valuable from noise +- **Sift** - carefully examine and select what's valuable + +**My top picks**: +1. **Synthesize** - captures curation + knowledge synthesis +2. **Distill** - captures AI triage + extracting essence +3. **Nexus** - captures central hub concept +4. **Scholar** - connects to original "Personal Scholar" vision +5. **Curator** - self-explanatory, professional + +--- + +## Conclusion + +**We're building the content inbox we wish existed**: + +- One place for newsletters, articles, feeds +- AI that triages so we don't have to read everything +- Highlights that actually become a useful knowledge base +- Open-source so we own our data + +**We're at 80% of the technical foundation. We need 4-5 more months to get to "daily use quality."** + +The path forward is clear: +1. Finish what's in progress (1 month) +2. Add email ingestion (2 weeks) +3. Add AI digest (3 weeks) +4. Polish highlights (1 week) +5. Dogfood until we love it (1 month) + +**Then we launch.** + +Let's build something we actually want to use every day. diff --git a/docs/architecture/unified-migration-backlog-complete.md b/docs/architecture/unified-migration-backlog-complete.md new file mode 100644 index 000000000..00bdec589 --- /dev/null +++ b/docs/architecture/unified-migration-backlog-complete.md @@ -0,0 +1,1330 @@ +## ARC-001 NestJS Package Setup ✅ **COMPLETED** + +- **Problem/Objective**: Create the foundational NestJS package and Docker infrastructure to run alongside existing Express API without disruption. +- **Approach**: Bootstrap NestJS application with proper workspace integration and Docker configuration. Tasks: + - [x] Create `packages/api-nest` directory structure + - [x] Initialize NestJS project with `nest new api-nest --skip-git` + - [x] Configure TypeScript with strict settings extending workspace root + - [x] Set up package.json with proper scripts and dependencies + - [x] Create Docker service in docker-compose.yml for new API on port 4001 + - [x] Configure environment loading and basic logging +- **Acceptance Criteria**: ✅ **ALL COMPLETED** + - [x] NestJS application boots without errors on port 4001 + - [x] Docker Compose runs both Express API (4000) and NestJS API (4001) + - [x] TypeScript compilation works with workspace configuration + - [x] Basic logging and environment loading functional +- **Dependencies**: None. +- **Effort Estimate**: 2 days. +- **Status**: ✅ Completed + +## ARC-002 Health Checks & Observability ✅ **COMPLETED** + +- **Problem/Objective**: Establish basic health monitoring and structured logging before migrating business logic. +- **Approach**: Set up comprehensive health checking and observability infrastructure. Tasks: + - [x] Install `@nestjs/terminus` for health checks + - [x] Create `/api/health` endpoint for basic status + - [x] Create `/api/health/deep` endpoint with database and Redis connectivity checks + - [x] Set up request logging middleware matching Express format + - [x] Configure structured logging with consistent error handling +- **Acceptance Criteria**: ✅ **ALL COMPLETED** + - [x] `/api/health` returns 200 with basic status + - [x] `/api/health/deep` checks database and Redis connectivity + - [x] Request/response logging matches Express format + - [x] Error handling returns consistent JSON responses +- **Dependencies**: ARC-001. +- **Effort Estimate**: 1 day. +- **Status**: ✅ Completed + +## ARC-003 Authentication Module ✅ **COMPLETED** + +- **Problem/Objective**: Migrate authentication to NestJS with improved validation while maintaining JWT compatibility with Express. +- **Approach**: Build comprehensive authentication system in NestJS with enhanced security. Tasks: + - [x] Create `AuthModule` with JWT strategy and passport integration + - [x] Implement authentication guards and decorators for route protection + - [x] Create `/api/v2/auth/*` endpoints for login, register, and OAuth flows + - [x] Set up OAuth providers structure (Google, Apple) + - [x] Add rate limiting and security middleware + - [x] Implement comprehensive E2E testing with user personas + - [x] Add Swagger/OpenAPI documentation +- **Acceptance Criteria**: ✅ **ALL COMPLETED** + - [x] `/api/v2/auth/login` works alongside Express `/api/auth/login` + - [x] JWT tokens are compatible between Express and NestJS APIs + - [x] Role-based access control (RBAC) implemented + - [x] Comprehensive test coverage achieved (>90%) +- **Dependencies**: ARC-001, ARC-002. +- **Effort Estimate**: 3 days. +- **Status**: ✅ Completed (3 days actual) + +## ARC-003B Database & Entity Integration ✅ **COMPLETED** + +- **Problem/Objective**: Integrate NestJS with existing PostgreSQL schema without breaking Express API or requiring complex migrations. +- **Approach**: Create TypeORM entities mapping to existing tables using hybrid migration strategy. Tasks: + - [x] Fix User entity to map exactly to existing schema (migrations 0001-0188) + - [x] Create UserProfile entity mapping to `user_profile` table (migration 0019) + - [x] Create UserPersonalization entity mapping to `user_personalization` table (migrations 0008+) + - [x] Create first new migration (0189) for role column using existing Postgrator system + - [x] Update DatabaseModule to include all entities + - [x] Update UserModule with full entity support + - [x] Document repeatable process for future entity migrations +- **Acceptance Criteria**: ✅ **ALL COMPLETED** + - [x] Entities map exactly to existing database schema + - [x] New role column added via traditional migration system + - [x] Both APIs can access same database tables + - [x] Repeatable process documented for future entities +- **Dependencies**: ARC-003. +- **Effort Estimate**: 2 days. +- **Status**: ✅ Completed (1 day actual) + +## ARC-004 GraphQL Module Setup ✅ **COMPLETED** + +- **Problem/Objective**: Set up GraphQL in NestJS to work alongside Express GraphQL without breaking existing clients. +- **Approach**: Establish parallel GraphQL endpoint in NestJS to gradually migrate resolvers from Express. Tasks: + - [x] Install `@nestjs/graphql` and `@nestjs/apollo` packages + - [x] Configure GraphQL module with Apollo Driver on `/api/graphql` path (aligned with Vite + legacy clients) + - [x] Create base GraphQL schema with essential types (User, AuthPayload) + - [x] Implement authentication context middleware to extract JWT tokens + - [x] Create initial resolvers (viewer + session) returning authenticated context + - [x] Add schema introspection and playground for development + - [x] Add Jest e2e coverage for `/api/graphql` viewer/session flows + - [x] Create LibraryModule with LibraryItemEntity mapping to existing `library_item` table + - [x] Implement `libraryItems` query with cursor-based pagination + - [x] Implement `libraryItem(id)` query for single item lookup +- **Acceptance Criteria**: ✅ **ALL COMPLETED** + - [x] GraphQL endpoint accessible at `/api/graphql` + - [x] Authentication context properly extracts user from JWT tokens + - [x] Viewer query returns current user data matching Express format + - [x] Schema introspection works without errors + - [x] Both Express and NestJS GraphQL endpoints function simultaneously + - [x] LibraryItemEntity correctly maps to existing database schema + - [x] Library queries return paginated results with proper type safety +- **Dependencies**: ARC-003B. +- **Effort Estimate**: 2 days. +- **Status**: ✅ Completed + +## ARC-004B Frontend Performance Optimization (Vite Migration) ✅ **FOUNDATION COMPLETE** + +- **Problem/Objective**: Migrate from Next.js to Vite for dramatically improved development experience and build performance. +- **Approach**: Complete frontend migration to Vite + React Router for 50-100x performance gains. Tasks: + - [x] Create Vite configuration with React, TypeScript, and SWC + - [x] Set up React Router for client-side routing with auth guards + - [x] Create packages/web-vite with initial structure + - [x] Configure GraphQL client targeting `/api/graphql` + - [x] Implement authentication store with JWT token management + - [x] Create basic LibraryPage component fetching from NestJS GraphQL + - [x] Integrate `libraryItems` query with pagination + - [x] Create all page stubs (Login, Register, Settings, Reader, Admin) + - [x] Implement protected routes and navigation + - [ ] ~~Implement advanced library features~~ → **Moved to ARC-009** + - [ ] ~~Configure Vite plugins for optimization~~ → **Infrastructure (can be done anytime)** + - [ ] ~~Update build pipeline and Docker~~ → **Infrastructure (can be done anytime)** + - [ ] ~~Update testing configuration~~ → **Infrastructure (can be done anytime)** +- **Acceptance Criteria**: ✅ **FOUNDATION COMPLETE** + - [x] Basic library page loads and displays items + - [x] Authentication flow works with login/logout + - [x] GraphQL queries successfully fetch from NestJS backend + - [x] All routes configured with proper protection + - [x] Dev experience significantly improved (HMR working) + - [ ] ~~Feature parity with legacy library UI~~ → **See ARC-009** + - [ ] ~~Production build optimization~~ → **Infrastructure backlog** +- **Dependencies**: ARC-003, ARC-004. +- **Effort Estimate**: Foundation: 1 week ✅ Complete | Remaining UI features: See ARC-009 +- **Status**: ✅ Foundation Complete - Ready for backend-driven feature development +- **Note**: Remaining UI features naturally roll into ARC-009 after backend APIs are ready (ARC-005 through ARC-008) + +## ARC-005 Library Core Mutations ✅ **COMPLETED** + +- **Problem/Objective**: Implement essential library item mutations to enable basic user actions without content processing. +- **Approach**: Add GraphQL mutations for core library management operations that don't require queue/content processing. This unblocks frontend action buttons and establishes mutation patterns. Tasks: + + **Backend (NestJS):** + - [x] Add mutations to LibraryResolver: + - [x] `archiveLibraryItem(id: String!, archived: Boolean!): LibraryItem!` + - [x] `deleteLibraryItem(id: String!): DeleteResult!` + - [x] `updateReadingProgress(id: String!, progress: ReadingProgressInput!): LibraryItem!` + - [x] `moveLibraryItemToFolder(id: String!, folder: String!): LibraryItem!` + - [x] Implement service methods in LibraryService: + - [x] `archive(userId, itemId, archived)` - update state column + - [x] `delete(userId, itemId)` - soft delete or hard delete based on current folder + - [x] `updateProgress(userId, itemId, progressInput)` - update reading progress fields + - [x] `moveToFolder(userId, itemId, folder)` - update folder column + - [x] Add input types to GraphQL schema: + - [x] `ReadingProgressInput` (topPercent, bottomPercent, anchorIndex) + - [x] `DeleteResult` (success, message) + - [x] Add validation and error handling for all mutations + - [x] Create E2E tests for each mutation covering success and error cases (18 tests, all passing) + + **Frontend (web-vite):** + - [x] Create mutation hooks in packages/web-vite/src/lib/graphql-client.ts: + - [x] `useArchiveItem()` hook + - [x] `useDeleteItem()` hook + - [x] `useUpdateReadingProgress()` hook + - [x] `useMoveToFolder()` hook + - [x] Wire mutations to LibraryPage action buttons + - [x] Add optimistic updates for better UX + - [x] Add success/error toast notifications + - [x] Handle loading states during mutation execution + +- **Acceptance Criteria**: ✅ **ALL COMPLETED** + - [x] Archive button archives/unarchives items successfully + - [x] Delete button removes items from library with confirmation + - [x] Reading progress updates persist correctly + - [x] Move to folder changes item location + - [x] All mutations work with proper authentication + - [x] Error handling displays user-friendly messages + - [x] Optimistic UI updates provide instant feedback + - [x] E2E tests achieve >90% coverage (18/18 passing) + - [x] Mutations maintain data consistency with database +- **Dependencies**: ARC-004, ARC-004B. +- **Effort Estimate**: 3-5 days. +- **Actual Time**: ~1 day +- **Status**: ✅ Completed + +## ARC-006 Advanced Search & Filtering ✅ **COMPLETED** + +- **Problem/Objective**: Implement comprehensive search and filtering capabilities to match legacy system functionality. +- **Approach**: Add full-text search, advanced filters, and sorting to library queries. Tasks: + + **Backend (NestJS):** + - [x] Enhance `libraryItems` query parameters: + - [x] Add `searchQuery: String` for full-text search + - [x] Add `folder: String` filter (inbox, archive, trash, all) + - [x] Add `state: LibraryItemState` filter + - [x] Add `sortBy: String` (savedAt, updatedAt, publishedAt, title, author) + - [x] Add `sortOrder: String` (ASC, DESC) + - [x] Implement full-text search in LibraryService: + - [x] Basic ILIKE search across title/description/author + - [ ] **DEFERRED**: PostgreSQL `ts_vector` full-text search (performance optimization) + - [ ] **DEFERRED**: Support multi-word queries with proper ranking + - [ ] **DEFERRED**: Handle special search operators (in:, is:, label:, has:) + - [x] Add query builder logic for complex filters + - [ ] **TODO**: Optimize database queries with proper indexes + - [x] Add query validation and sanitization + - [x] Create E2E tests for search scenarios (12 new tests, 30/30 passing) + + **Frontend (web-vite):** + - [x] Enhance search box with debounced input (300ms) + - [ ] **DEFERRED**: Add visual query builder UI (optional) + - [ ] **DEFERRED**: Implement search suggestions/typeahead + - [x] Add folder filter tabs (Inbox, Archive, All, Trash) + - [x] Add sort controls (saved date, updated date, published date, title, author) + - [x] Show search result count + - [ ] **DEFERRED**: Add search history/saved searches + - [x] Handle debounced search input + - [x] Add loading indicators during search + +- **Acceptance Criteria**: + - [x] Full-text search returns relevant results (basic ILIKE matching) + - [x] Folder filters correctly scope results + - [x] State filters work correctly (archived, deleted, etc.) + - [x] Sort controls change result ordering + - [ ] **DEFERRED**: Search query syntax matches legacy system (in:inbox, label:tech, etc.) + - [ ] **TODO**: Search performance acceptable (<500ms for typical queries) - needs indexes + - [x] Empty search states display helpful messages + - [x] Search works correctly with pagination + - [ ] **DEFERRED**: Legacy search queries migrate seamlessly +- **Dependencies**: ARC-005. +- **Effort Estimate**: 2-3 days. +- **Actual Time**: ~4 hours +- **Status**: ✅ Completed (with performance optimizations deferred to ARC-006B) + +## ARC-006B Performance & UX Optimizations ✅ **COMPLETED** + +- **Problem/Objective**: Optimize search performance, logging, and UX based on initial implementation feedback. +- **Approach**: Add database indexes, simplify logging, improve debounce behavior, add query monitoring. Tasks: + + **Performance:** + - [x] Add PostgreSQL indexes for search fields (title, author, description, folder, state, savedAt) + - [x] Add pg_trgm extension for fast ILIKE queries + - [x] Add GIN indexes for array columns (labels) + - [x] Created migration 0190 with 8 strategic indexes + - [x] Benchmark query performance and set targets (<200ms for search) + + **Logging:** + - [x] Simplify structured logging format for better readability + - [x] Create dev-friendly format (one-line with key info) + - [x] Keep structured format for production + - [x] Add color coding for log levels + + **Query Monitoring:** + - [x] Create TypeORM query logger to track slow queries + - [x] Add execution time threshold (warn if >500ms) + - [x] Log query execution times in development + - [x] Create QueryTimer utility for manual timing + + **UX Improvements:** + - [x] Fix search debounce to not trigger loading on empty query + - [x] Add "searching..." indicator separate from full page load + - [x] Separate loading vs searching states + - [x] Smart debounce: 300ms for search, 0ms for folder changes + - [x] Show result count prominently + +- **Acceptance Criteria**: ✅ **ALL COMPLETED** + - [x] Search queries execute in <200ms with indexes (tested: ~150ms) + - [x] Logs are readable in terminal without JSON parsing + - [x] Slow queries (>500ms) are logged with details + - [x] Deleting search text doesn't cause jarring reload + - [x] Users can type rapidly without performance issues +- **Dependencies**: ARC-006. +- **Effort Estimate**: 1-2 days. +- **Actual Time**: ~1 day +- **Status**: ✅ Completed +- **Files Created**: + - `packages/db/migrations/0190.do.add_library_item_search_indexes.sql` + - `packages/db/migrations/0190.undo.add_library_item_search_indexes.sql` + - `packages/db/migrations/0190.README.md` + - `packages/api-nest/src/database/query-logger.ts` + - `packages/api-nest/PERFORMANCE_OPTIMIZATIONS.md` +- **Performance Impact**: + - Folder filter: 26x faster (~800ms → ~30ms) + - Text search: 8x faster (~1200ms → ~150ms) + - Sort operations: 30x faster (~600ms → ~20ms) + +## ARC-007 Bulk Operations & Multi-select ✅ **COMPLETED** +- **Problem/Objective**: Enable power users to perform actions on multiple library items simultaneously. +- **Approach**: Implement bulk mutations that operate on multiple items efficiently. Tasks: + + **Backend (NestJS):** + - [x] Add bulk mutations to LibraryResolver: + - [x] `bulkArchiveItems(itemIds: [String!]!, archived: Boolean!): BulkActionResult!` + - [x] `bulkDeleteItems(itemIds: [String!]!): BulkActionResult!` + - [x] `bulkMoveToFolder(itemIds: [String!]!, folder: String!): BulkActionResult!` + - [x] `bulkMarkAsRead(itemIds: [String!]!): BulkActionResult!` + - [x] Implement bulk operations in LibraryService: + - [x] Support explicit item ID lists + - [x] Use database transactions for atomicity + - [x] Implement batch processing (100 items per batch) + - [x] Handle partial failures gracefully + - [x] Add GraphQL types: + - [x] `BulkActionResult` (success, successCount, failureCount, errors, message) + - [x] Add bulk operation limits (1000 items max) and validation + - [x] Create E2E tests for bulk scenarios (14 tests, all passing) + + **Frontend (web-vite):** + - [x] Implement multi-select mode UI: + - [x] Add checkbox to each library card + - [x] Add "Select All" / "Deselect All" controls + - [x] Show multi-select action bar when items selected + - [x] Add visual indicators for selected items + - [x] Multi-Select toggle button + - [x] Create bulk action buttons: + - [x] Archive/Unarchive selected + - [x] Delete selected + - [x] Move to folder (inbox, archive) + - [x] Mark as read + - [x] Add bulk action confirmation modals + - [x] Handle partial failures gracefully + - [x] Show success/failure counts via toast notifications + - [ ] **DEFERRED**: Keyboard shortcuts for multi-select (Shift+Click, Cmd+A) + - [ ] **DEFERRED**: Query-based selection (all items matching search) + +- **Acceptance Criteria**: ✅ **CORE COMPLETE** + - [x] Users can select multiple items via checkboxes + - [x] Bulk actions execute successfully on selected items + - [x] Bulk operations maintain data consistency (transactions) + - [x] Partial failures are reported clearly + - [x] Multi-select UI functional and intuitive + - [x] Bulk operations have reasonable performance (batched processing) + - [x] Optimistic UI updates provide instant feedback + - [ ] **DEFERRED**: Keyboard shortcuts (future enhancement) + - [ ] **DEFERRED**: Query-based bulk actions (future enhancement) +- **Dependencies**: ARC-005, ARC-006. +- **Effort Estimate**: 2 days. +- **Actual Time**: ~2 hours +- **Status**: ✅ Completed +- **Test Coverage**: 44/44 tests passing (30 existing + 14 new bulk operation tests) + +## ARC-007B Architecture Refinements (Technical Debt) ✅ **COMPLETED** + +- **Problem/Objective**: Address identified architectural concerns and technical debt before adding more complex features. +- **Approach**: Refactor existing code to follow NestJS best practices and improve maintainability. Tasks: + + **Constants & Type Safety:** ✅ **COMPLETED** (see TD-004) + - [x] Create constants file for folder names (`FOLDER_INBOX`, `FOLDER_ARCHIVE`, `FOLDER_TRASH`) + - [x] Replace all folder magic strings with constants throughout codebase + - [x] Add TypeScript const assertions for immutability + - [ ] **DEFERRED**: Create constants for library item states (extract from enum) + - [ ] **DEFERRED**: Create constants for config keys (all `EnvVariables` references) + + **Repository Pattern:** ✅ **COMPLETED** (see TD-003) + - [x] Create `LibraryItemRepository` class extending TypeORM Repository + - [x] Move all DataSource operations from `LibraryService` to repository + - [x] Move bulk operations (transaction logic) into repository methods + - [x] Create `UserRepository` class for user-specific database operations + - [x] Update services to use repositories exclusively (remove DataSource injections) + - [x] Update tests to mock repositories instead of DataSource + + **Service Layer Cleanup:** ✅ **COMPLETED** + - [x] Review `LibraryService` - ensure business logic only, no direct DB queries + - [x] Review `AuthService` - move seedLibraryItems to dedicated seeding service + - [x] Ensure consistent error handling patterns across services + - [x] Add JSDoc comments to public service methods + + **Testing:** ✅ **COMPLETED** + - [x] Verify all unit tests still pass after refactoring + - [x] Verify all E2E tests still pass after refactoring (151/151 passing) + - [x] Add integration tests for repository methods + +- **Acceptance Criteria**: + - [x] Zero magic strings for folders in services/resolvers (all constants) + - [x] Services use repositories exclusively (no DataSource injections) + - [x] Repository pattern consistently applied across all entities + - [x] All tests passing (151/151 E2E tests) + - [x] Service layer properly separated (business logic vs data access) + - [x] JSDoc comments on all public service methods +- **Dependencies**: ARC-007. +- **Effort Estimate**: 1-2 days. +- **Actual Time**: ~3 days (Constants + Repository + Service Layer + Testing fixes) +- **Priority**: Medium (can be done after ARC-008 or ARC-009) +- **Status**: ✅ **COMPLETED** +- **Service Layer Improvements**: + - **LibraryService**: All business logic with no direct DB queries, comprehensive JSDoc + - **AuthService**: Removed DataSource injection, seeding moved to DefaultUserResourcesService + - **DefaultUserResourcesService**: Enhanced with seedExampleLibraryItems method + - **Error Handling**: Consistent use of NestJS exceptions (NotFoundException, BadRequestException, UnauthorizedException) + - **JSDoc**: All public methods documented with parameters, return types, and thrown exceptions + +## ARC-008 Labels System ✅ **COMPLETED** + +- **Problem/Objective**: Implement label management to enable users to organize and filter their library items. +- **Approach**: Create comprehensive label system with CRUD operations and item associations. Tasks: + + **Backend (NestJS):** ✅ **COMPLETE** + - [x] Create Label and EntityLabel entities mapping to existing database schema + - [x] Label entity: id, name, color, description, position, internal, timestamps, userId + - [x] EntityLabel junction table for many-to-many with library items + - [x] Create LabelModule with service and resolver + - [x] Add GraphQL queries: + - [x] `labels: [Label!]!` - list all user's labels ordered by position + - [x] `label(id: String!): Label` - get single label + - [x] Add GraphQL mutations with validation: + - [x] `createLabel(input: CreateLabelInput!): Label!` - with duplicate name check + - [x] `updateLabel(id: String!, input: UpdateLabelInput!): Label!` - with internal label protection + - [x] `deleteLabel(id: String!): DeleteResult!` - with internal label protection + - [x] `setLibraryItemLabels(itemId: String!, labelIds: [String!]!): [Label!]!` - replace item labels + - [x] Update LibraryItemEntity with EntityLabel relation + - [x] Add field resolver for labels in LibraryResolver + - [x] Add comprehensive input validation: + - [x] Label name: 1-100 chars, unique per user + - [x] Color: Hex format (#FF5733) + - [x] Description: 0-500 chars + - [x] Register entities in DatabaseModule + - [x] Schema generation complete with all types and mutations + - [x] Fix label filtering by syncing label_names column when labels are assigned + - [x] Database migration 0191 for labels.updated_at default value + - [ ] **DEFERRED**: E2E tests (testing infrastructure needs updates) + + **Frontend (web-vite):** ✅ **COMPLETE** + - [x] Create Labels management page: + - [x] List all labels with colors + - [x] Create new label form + - [x] Edit label inline + - [x] Delete label with confirmation + - [x] Add label selection UI to library items: + - [x] Label picker dropdown component + - [x] Multi-select label checkboxes + - [x] Visual label chips on cards + - [x] Add label filtering to search: + - [x] Filter by label dropdown + - [x] Show active label filters count + - [x] Clear individual label filters + - [x] Create label management hooks: + - [x] `useLabels()` - fetch all labels + - [x] `useCreateLabel()` - create new label + - [x] `useUpdateLabel()` - update existing label + - [x] `useDeleteLabel()` - delete label + - [x] `useSetLibraryItemLabels()` - assign labels to item + +- **Acceptance Criteria**: ✅ **ALL COMPLETED** + - [x] Users can create, update, and delete labels + - [x] Labels can be assigned to library items + - [x] Multiple labels per item supported + - [x] Label filtering works in search + - [x] Label colors display correctly in UI + - [x] Label deletion handles item associations gracefully (cascade delete) + - [x] Label assignment syncs both entity_labels and label_names columns + - [x] Label names are unique per user + - [x] Label UI provides intuitive dropdown picker +- **Dependencies**: ARC-005, ARC-006. +- **Effort Estimate**: 2-3 days. +- **Actual Time**: ~1 day +- **Status**: ✅ Completed +- **Key Fixes Applied**: + - Fixed LabelPicker to convert label names to UUIDs before API call + - Added schema specification to LibraryItemEntity (`schema: 'omnivore'`) + - Fixed all column name mappings (snake_case vs camelCase) + - Created migration 0191 for `labels.updated_at` default value + - Updated `setLibraryItemLabels` to sync `label_names` column for filtering + - Injected LibraryItemEntity repository into LabelService for column updates + +- **UI/UX Enhancements (Jan 2025)** ✅: + - Redesigned Labels page with Linear-inspired clean table layout + - Replaced grid cards with minimalist table view (Name, Description, Created columns) + - Implemented small 8px color dots instead of large colored blocks + - Added search functionality with icon and filtering + - Created modal overlay for create/edit forms (600px width for better UX) + - Implemented three-dot menu with Feather icons (Edit, Delete actions) + - Fixed dropdown menu clipping issues: + - Changed table wrapper to `overflow: visible` + - Increased dropdown z-index to 1000 + - Added auto-positioning logic (opens upward when near bottom of viewport) + - Optimized color picker UX (simplified to single full-width input) + - Implemented full-width table layout (`max-width: 100vw`) + - Fixed search box rendering issue on navigation: + - Used rem units instead of CSS variables for critical positioning + - Added vertical centering with `transform: translateY(-50%)` + - Increased z-index to prevent icon/placeholder overlap + - All improvements maintain design token system and theming capability + +## ARC-010A Minimal Reader ✅ **COMPLETED** + +- **Problem/Objective**: Enable users to read saved articles with basic display functionality before implementing advanced features. +- **Approach**: Create simple, clean reader page that displays extracted content without highlights/annotations. This unblocks content extraction testing and delivers core reading value quickly. Tasks: + + **Backend (NestJS):** + - [x] Add `content` field to LibraryItem GraphQL type (HTML content) + - [x] ~~Add `textContent` field~~ - Not needed (readable_content serves this purpose) + - [x] Ensure `libraryItem(id)` query returns content fields + - [x] Add basic content sanitization (DOMPurify on frontend) + + **Frontend (web-vite):** + - [x] Create `/reader/:id` route with ReaderPage component + - [x] Implement reader layout: + - [x] Article header (title, author, date, original URL) + - [x] Content display area with clean typography + - [x] Back to library button + - [ ] ~~Share/actions menu~~ - Deferred to ARC-010 + - [x] Add loading state while fetching content + - [x] Add error state for missing/failed content + - [x] Handle CONTENT_NOT_FETCHED state gracefully (show message) + - [x] Responsive design (mobile + desktop) + - [x] Basic reading styles (font size, line height, max-width) + - [x] Update LibraryPage to link to reader (click title/Read button) + +- **Acceptance Criteria**: ✅ **ALL CORE CRITERIA MET** + - [x] Users can click an item and navigate to reader page + - [x] Content displays with clean, readable typography + - [x] Works on mobile and desktop devices + - [x] Gracefully handles items without content yet + - [x] Back navigation returns to library + - [x] Reader route is protected (requires auth) +- **Dependencies**: None (works with current backend, enhanced by ARC-013) +- **Effort Estimate**: 1-2 days +- **Actual Time**: ~2 hours +- **Status**: ✅ Completed (2025-10-05) +- **Note**: This is a minimal viable reader. Advanced features (highlights, notes, progress) come in ARC-010. +- **Completion Analysis**: See `/docs/architecture/ARC-010A-COMPLETION-ANALYSIS.md` + +## ARC-010 Notebooks & Colored Highlights (Backend) ✅ **COMPLETED** + +- **Problem/Objective**: Implement notebooks and colored highlights system with clean data model. +- **Approach**: Migrate notebooks from highlight table to library_item, implement 4-color highlight system. Tasks: + + **Backend (NestJS):** + - [x] Database Migration 0192 (consolidate notebooks): + - [x] Add `note` and `note_updated_at` columns to library_item table + - [x] Migrate existing notebooks from highlight table (type='NOTE') + - [x] Remove legacy notebook-type highlights + - [x] Zero data loss, fully transactional migration + - [x] Update LibraryItemEntity with notebook fields + - [x] Create HighlightEntity mapping to existing `highlight` table + - [x] Add color support to HighlightEntity (yellow, red, green, blue) + - [x] Create HighlightModule with service and resolver + - [x] Add GraphQL queries: + - [x] `highlights(itemId: String!): [Highlight!]!` - get all highlights for item + - [x] `highlight(id: String!): Highlight` - get single highlight + - [x] Add GraphQL mutations: + - [x] `updateNotebook(itemId: String!, note: String!): LibraryItem!` + - [x] `createHighlight(input: CreateHighlightInput!): Highlight!` + - [x] `updateHighlight(id: String!, input: UpdateHighlightInput!): Highlight!` + - [x] `deleteHighlight(id: String!): DeleteResult!` + - [x] Add input types with validation (color enum, required fields) + - [x] Create E2E tests: + - [x] 9 notebook tests (create, update, concurrent edits) + - [x] 30+ highlight tests (CRUD, colors, filtering, pagination) + +- **Acceptance Criteria**: ✅ **ALL COMPLETED** + - [x] Notebooks stored in library_item.note (not highlight table) + - [x] Highlights use normalized highlight table with color support + - [x] Migration 0192 successfully consolidates data model + - [x] 4-color system works (yellow, red, green, blue) + - [x] GraphQL mutations handle all CRUD operations + - [x] All 39 E2E tests passing + - [x] Data model validated and documented +- **Dependencies**: ARC-010A (minimal reader as foundation), ARC-005. +- **Effort Estimate**: 3-4 days. +- **Actual Time**: ~2 days +- **Status**: ✅ Completed (2025-01-17) +- **Key Decisions**: + - Moved notebooks to library_item.note for cleaner separation + - Implemented 4-color highlight system matching Omnivore design + - Note column stores TEXT (raw markdown, frontend renders) + - One-to-many relationship: library_item → highlight +- **Files**: + - Migration: `packages/db/migrations/0192.do.consolidate-notebooks.sql` + - Entity: `packages/api-nest/src/entities/highlight.entity.ts` + - Module: `packages/api-nest/src/modules/highlight/*` + - Tests: `packages/api-nest/test/notebook.e2e-spec.ts`, `test/highlight.e2e-spec.ts` + +## ARC-011 Add Link & Content Ingestion ✅ **COMPLETED** + +- **Problem/Objective**: Implement the core "save to library" functionality with URL parsing and content extraction. +- **Approach**: Build the link saving pipeline. Content extraction deferred to ARC-012 (queue) and ARC-013 (readability). Tasks: + + **Backend (NestJS):** + - [x] Add GraphQL mutation: + - [x] `saveUrl(input: SaveUrlInput!): LibraryItem!` + - [x] Create SaveUrlInput type with url and folder fields + - [x] Add validation (URL format using @IsUrl, duplicate detection) + - [x] Generate unique slugs from URLs with timestamp + - [x] Set initial state to CONTENT_NOT_FETCHED (extraction deferred to ARC-012) + - [x] Create E2E tests for save URL flow (17 tests, all passing) + - [ ] ~~Handle different content types (article, PDF, etc.)~~ → **Deferred to ARC-013** + - [ ] ~~Add rate limiting for URL saving~~ → **Can be added anytime** + - [ ] ~~Implement basic content extraction~~ → **Deferred to ARC-012 (queue) and ARC-013 (readability)** + + **Frontend (web-vite):** + - [x] Add useSaveUrl hook to graphql-client + - [x] Implement "Add Link" modal with URL input + - [x] Add folder selection to save modal (inbox/archive) + - [x] Add content type tabs (Link, PDF, RSS) with "coming soon" for PDF/RSS + - [x] Show save progress indicator (loading spinner) + - [x] Handle save errors gracefully (validation + error messages) + - [x] Add URL validation in UI (client-side validation) + - [x] Show newly saved item in library immediately (refetch after save) + - [x] Integrate modal with "+ Add Article" buttons + - [ ] ~~Add browser extension integration points~~ → **Future enhancement** + - [ ] ~~Folder selection persists preference~~ → **Future UX enhancement** + +- **Acceptance Criteria**: ✅ **ALL CORE CRITERIA MET** + - [x] Users can save URLs to their library + - [x] Duplicate URLs detected and handled (ConflictException) + - [x] Save errors provide helpful messages (validation errors shown in UI) + - [x] Saved items appear in library immediately (refetch on success) + - [x] Folder selection works (inbox/archive dropdown) + - [x] All 17 E2E tests passing (including validation and error cases) + - [ ] ~~Basic content extraction works for common sites~~ → **Deferred to ARC-012/ARC-013** + - [ ] ~~Rate limiting prevents abuse~~ → **Can be added anytime** + - [ ] ~~Browser extension can save URLs~~ → **Future enhancement** +- **Dependencies**: ARC-005. +- **Effort Estimate**: 2-3 days. +- **Status**: ✅ **Completed** (actual: 1 day for MVP focusing on URL saving, content extraction deferred) + +## ARC-012 Queue Integration & Background Processing ⭐ **80% COMPLETE** +- **Problem/Objective**: Integrate BullMQ queues for robust background processing of content extraction and other async tasks in single-service architecture. +- **Architectural Decisions** (see `/docs/architecture/ARC-012-QUEUE-ARCHITECTURE-DESIGN.md` and `ARC-012-EVENT-AND-REDIS-ANALYSIS.md`): + - **Event Pattern**: Node.js EventEmitter (not full EventManager) for simplicity ✅ + - **Redis Architecture**: Sentinel (master-slave with HA) for BullMQ compatibility ✅ + - **Worker Strategy**: In-process workers (not separate microservice) ✅ + - **Scaling**: Horizontal pod autoscaling with shared Redis ✅ + - **Configuration**: Constants file (no magic strings) ✅ + +- **Approach**: Establish queue infrastructure with event-driven processing. Implementation in 5 phases: + + ### **Phase 1: Infrastructure Setup** ✅ **COMPLETE** + - [x] Install dependencies: `@nestjs/bullmq`, `bullmq`, `ioredis` + - [x] Create `queue.constants.ts` with all queue names, job types, priorities + - [x] Create `QueueModule` with Redis Sentinel configuration + - [x] Set up shared Redis connection (cache + queue) + - [x] Create health check endpoints for queue/Redis (QueueHealthIndicator) + - [x] Add graceful shutdown handling (OnModuleDestroy) + - [x] Fix Redis maxRetriesPerRequest (null for BullMQ blocking operations) + - [x] Fix Jest ESM configuration for bullmq/msgpackr + - [x] **Testing**: Unit tests for QueueModule, health checks (13/13 passing) + - [ ] Add Prometheus metrics integration → **DEFERRED to Phase 5** + + ### **Phase 2: Event System** ✅ **COMPLETE** + - [x] Create `EventBusService` extending EventEmitter + - [x] Define event types in `events.constants.ts` + - [x] Create event data interfaces (type-safe) + - [x] Wire event handlers to queue operations + - [x] Add event emission logging + - [x] **Testing**: Unit tests for EventBusService (13/13 passing) + + ### **Phase 3: Content Processing Queue** ✅ **INFRASTRUCTURE COMPLETE** ⏳ **CONTENT STUB** + - [x] Create `ContentProcessorService` with `@Processor()` decorator + - [x] Implement `@Process('fetch-content')` job handler with **STUB** content fetching + - [x] Add job priority configuration (HIGH, NORMAL, LOW) + - [x] Implement retry logic with exponential backoff (3 attempts) + - [x] Add job deduplication by libraryItemId as jobId + - [x] Add progress tracking (updateProgress at 10%, 20%, 70%, 90%, 100%) + - [x] **Testing**: Unit tests for processor (15/15 passing) + - [ ] **TODO**: Implement real content fetching (readability extraction) → **ARC-013** + - [ ] Configure rate limiting per user → **DEFERRED** (can add later) + + ### **Phase 4: Library Integration** ✅ **COMPLETE** + - [x] Update `saveUrl` mutation to emit ContentSaveRequested event + - [x] Update library item state: PROCESSING → SUCCEEDED/FAILED + - [x] Inject EventBusService into LibraryService + - [x] Add source tracking to SaveUrlInput + - [x] **Testing**: E2E test for full saveUrl → queue → process flow (17/17 passing) + - [ ] Add job status polling endpoint for frontend → **NOT NEEDED** (can query item state) + - [ ] Implement job cancellation endpoint → **DEFERRED** (future enhancement) + - [ ] Add user notification on processing completion/failure → **Event system ready**, UI integration deferred + + ### **Phase 5: Monitoring & Optimization** ⏸️ **DEFERRED** + - [ ] Add BullMQ Board UI endpoint (`/admin/queues`) + - [ ] Implement queue depth metrics (Prometheus) + - [ ] Add job latency histograms + - [ ] Create AlertManager rules for queue backlog + - [ ] Add worker concurrency auto-adjustment + - [ ] Performance profiling and optimization + - [ ] **Testing**: Load test with 100+ concurrent jobs + + ### **Configuration Management (No Magic Strings)** + ```typescript + // queue.constants.ts + export const QUEUE_NAMES = { + CONTENT_PROCESSING: 'content-processing', + NOTIFICATIONS: 'notifications', + POST_PROCESSING: 'post-processing', + } as const + + export const JOB_TYPES = { + FETCH_CONTENT: 'fetch-content', + SEND_NOTIFICATION: 'send-notification', + } as const + + export const JOB_PRIORITY = { + CRITICAL: 1, + HIGH: 5, + NORMAL: 10, + LOW: 20, + } as const + ``` + +- **Testing Requirements**: + - [ ] **Unit Tests**: + - [ ] QueueModule configuration and dependency injection + - [ ] EventBusService event emission and handling + - [ ] ContentProcessorService job processing logic + - [ ] Redis connection management and failover + - [ ] Job priority and deduplication logic + - [ ] **Integration Tests**: + - [ ] Queue → Worker communication + - [ ] Event → Queue → Processing flow + - [ ] Redis Sentinel failover scenarios + - [ ] Graceful shutdown with in-flight jobs + - [ ] **E2E Tests** (see `packages/api-nest/test/queue.e2e-spec.ts`): + - [ ] Complete saveUrl → queue → process → update flow + - [ ] Job retry on failure (3 attempts) + - [ ] Job cancellation by user + - [ ] Rate limiting enforcement + - [ ] Concurrent job processing (50+ jobs) + - [ ] Queue backlog handling + - [ ] **Load Tests**: + - [ ] 100 jobs/minute sustained load + - [ ] Burst traffic (500 jobs in 1 minute) + - [ ] Multiple replica scaling (2x, 3x, 5x) + +- **Acceptance Criteria**: + - [x] API response time <200ms (unchanged from current) ✅ + - [x] Jobs queued and processed reliably (no data loss) ✅ + - [x] Failed jobs retry with exponential backoff (3 attempts) ✅ + - [x] Graceful shutdown completes in-flight jobs (<30s) ✅ + - [x] All tests passing (unit, integration, E2E) - **87 unit + 116 E2E passing** ✅ + - [ ] Queue monitoring UI shows accurate metrics → **Phase 5** + - [ ] Horizontal scaling works (2x replicas = ~2x throughput) → **Future testing** + - [ ] Redis Sentinel failover recovers in <10 seconds → **Future testing** + - [ ] Prometheus metrics exported and alerting configured → **Phase 5** + - [ ] Job throughput: 50+ jobs/hour on single instance → **Needs real content fetching** + - [ ] Real content extraction working → **ARC-013** + +- **Dependencies**: ARC-011 (completed). +- **Effort Estimate**: 3 days (originally estimated). +- **Actual Time**: ~2 days for infrastructure (Phases 1-4), Phase 5 deferred +- **Status**: ✅ **80% Complete** - Infrastructure ready, content fetching stub needs ARC-013 +- **Architecture Docs**: + - Detailed design: `/docs/architecture/ARC-012-QUEUE-ARCHITECTURE-DESIGN.md` + - Event & Redis analysis: `/docs/architecture/ARC-012-EVENT-AND-REDIS-ANALYSIS.md` +- **Key Achievements**: + - ✅ BullMQ integrated with proper Redis configuration + - ✅ Event-driven architecture with EventBusService + - ✅ Worker pattern established with ContentProcessorService + - ✅ Full test coverage (42 queue tests, 17 E2E tests) + - ✅ Clean logger mocking using NestJS .setLogger() pattern + - ✅ Jest ESM configuration fixed for bullmq dependencies + +## TD-003 Repository Pattern Implementation ✅ **COMPLETED** + +- **Problem/Objective**: Services currently inject DataSource directly and use query builders inline. This violates NestJS best practices and makes code harder to test and maintain. **Without repositories, we cannot properly mock at the right boundary for unit tests.** +- **Goal**: Implement repository pattern for all entities to separate data access layer from business logic. This is the **foundation** for TD-006 (testing infrastructure). +- **Why Critical**: + - Enables proper unit testing (mock repositories, not DataSource) + - Separates concerns (services = business logic, repositories = data access) + - Blocks TD-006 (can't convert to unit tests without repositories) + - Industry best practice for testable architecture +- **Status**: ✅ **COMPLETED** +- **Actual Time**: 2 days +- **Results**: + - ✅ All major entities have repository implementations + - ✅ Services refactored to use repository interfaces + - ✅ Centralized RepositoriesModule prevents circular dependencies + - ✅ Test results: 124/151 E2E tests passing (82% pass rate, 6/8 suites passing) + - ✅ Ready for TD-006 (Testcontainers + Factory pattern) + +**Approach**: Create custom repository classes implementing repository interfaces. + +### **Phase 1: Core Repository Infrastructure** (Day 1-2) + +**1. Create Repository Interfaces** (Define contracts) +```typescript +// src/repositories/interfaces/library-item-repository.interface.ts +export interface ILibraryItemRepository { + findByUserId(userId: string, options?: FindOptions): Promise; + findById(id: string, userId: string): Promise; + findWithFilters(filters: LibraryItemFilters): Promise>; + save(item: LibraryItem): Promise; + bulkUpdate(items: Partial[], userId: string): Promise; + delete(id: string, userId: string): Promise; +} +``` + +**2. Create Repository Implementations** +```typescript +// src/repositories/library-item.repository.ts +@Injectable() +export class LibraryItemRepository implements ILibraryItemRepository { + constructor( + @InjectRepository(LibraryItemEntity) + private readonly repo: Repository + ) {} + + async findByUserId(userId: string, options?: FindOptions): Promise { + return await this.repo.find({ + where: { userId }, + ...options + }); + } + + // ... implement all interface methods +} +``` + +**Tasks**: ✅ **ALL COMPLETED** +- [x] Create `src/repositories/interfaces/` directory +- [x] Create interface for each entity: + - [x] `ILibraryItemRepository` (highest priority - most used) - 9 methods + - [x] `IHighlightRepository` - 5 methods + - [x] `ILabelRepository` - 7 methods + - [x] `IEntityLabelRepository` - 4 methods (many-to-many relationship management) + - [ ] `IUserRepository` - Deferred (not needed for current features) +- [x] Create `src/repositories/` directory for implementations +- [x] Implement `LibraryItemRepository`: + - [x] Move all query builder logic from `LibraryService` + - [x] Add custom methods: `findById()`, `findByUrl()`, `listForUser()`, `bulkArchive()`, `bulkDelete()`, `bulkMoveToFolder()`, `bulkMarkAsRead()` + - [x] Include transaction handling for bulk operations (batches of 100) + - [x] Add proper error handling and logging +- [x] Implement `HighlightRepository`: + - [x] Move query logic from `HighlightService` + - [x] Add methods: `findById()`, `findByLibraryItem()`, `create()`, `save()`, `remove()` + - [x] Sorted by position (reading order) +- [x] Implement `LabelRepository`: + - [x] Move label query logic from `LabelService` + - [x] Add methods: `findAll()`, `findById()`, `findByName()`, `findByIds()`, `create()`, `save()`, `remove()` +- [x] Implement `EntityLabelRepository`: + - [x] Add methods: `findByLibraryItemId()`, `deleteByLibraryItemId()`, `create()`, `save()` + +### **Phase 2: Service Refactoring** (Day 3-4) + +**3. Update Services to Use Repositories** +```typescript +// Before (bad - direct DataSource) +@Injectable() +export class LibraryService { + constructor(private dataSource: DataSource) {} + + async getItems(userId: string) { + return await this.dataSource + .getRepository(LibraryItemEntity) + .createQueryBuilder('item') + .where('item.userId = :userId', { userId }) + .getMany(); + } +} + +// After (good - repository injection) +@Injectable() +export class LibraryService { + constructor(private readonly libraryRepo: ILibraryItemRepository) {} + + async getItems(userId: string) { + return await this.libraryRepo.findByUserId(userId); + } +} +``` + +**Tasks**: ✅ **ALL COMPLETED** +- [x] Update `LibraryService`: + - [x] Inject `ILibraryItemRepository` instead of DataSource + - [x] Replace all `dataSource.query()` calls with repository methods + - [x] Keep only business logic (validation, authorization, orchestration) + - [x] Remove all SQL/query builders +- [x] Update `HighlightService`: + - [x] Inject `IHighlightRepository` and `ILibraryItemRepository` + - [x] Delegate all DB operations to repositories +- [x] Update `LabelService`: + - [x] Inject `ILabelRepository`, `IEntityLabelRepository`, and `ILibraryItemRepository` + - [x] Delegate all DB operations to repositories +- [ ] Update `UserService`: Deferred (not needed for current features) +- [ ] Update `AuthService`: Deferred (working well with current implementation) + +### **Phase 3: Module Registration** (Day 4) + +**4. Register Repositories in Modules** +```typescript +// src/library/library.module.ts +@Module({ + imports: [TypeOrmModule.forFeature([LibraryItemEntity])], + providers: [ + { + provide: 'ILibraryItemRepository', + useClass: LibraryItemRepository, + }, + LibraryService, + LibraryResolver, + ], + exports: ['ILibraryItemRepository'], +}) +export class LibraryModule {} +``` + +**Tasks**: ✅ **ALL COMPLETED** +- [x] Create centralized `RepositoriesModule` to prevent circular dependencies +- [x] Register all repository providers with string tokens (`'ILibraryItemRepository'`, etc.) +- [x] Export TypeOrmModule to support test data setup +- [x] Update `LibraryModule` to import RepositoriesModule +- [x] Update `HighlightModule` to import RepositoriesModule (removed direct TypeOrmModule.forFeature) +- [x] Update `LabelModule` to import RepositoriesModule (removed direct TypeOrmModule.forFeature) +- [x] Ensure proper dependency injection throughout (no circular dependencies) + +### **Phase 4: Testing Updates** (Day 5) + +**5. Create Repository Contract Tests** +```typescript +// test/repositories/library-item.repository.spec.ts +describe('LibraryItemRepository Contract', () => { + it('enforces unique constraints on originalUrl', async () => { + await repo.save({ originalUrl: 'https://example.com', userId: 'user1' }); + await expect( + repo.save({ originalUrl: 'https://example.com', userId: 'user1' }) + ).rejects.toThrow('duplicate key'); + }); + + it('returns null for non-existent items', async () => { + const result = await repo.findById('non-existent', 'user1'); + expect(result).toBeNull(); + }); +}); +``` + +**6. Update Unit Tests to Mock Repositories** +```typescript +// Before +const mockDataSource = { + getRepository: jest.fn().mockReturnValue({ + find: jest.fn(), + }), +}; + +// After +const mockRepo: jest.Mocked = { + findByUserId: jest.fn().mockResolvedValue([mockItem]), + findById: jest.fn().mockResolvedValue(mockItem), + save: jest.fn().mockResolvedValue(mockItem), + // ... other methods +}; +``` + +**Tasks**: ⏳ **PARTIALLY COMPLETED** +- [ ] Create contract test suite for each repository (Deferred to TD-006): + - [ ] `LibraryItemRepository` contract tests + - [ ] `HighlightRepository` contract tests + - [ ] `LabelRepository` contract tests + - [ ] `EntityLabelRepository` contract tests +- [x] Update existing unit tests: + - [x] Repository pattern doesn't break existing tests (transparent change) + - [x] All 87 unit tests still pass +- [x] Update E2E tests: + - [x] Added HighlightEntity to test database config + - [x] Fixed EntityMetadataNotFoundError in highlight tests + - [x] Verify E2E tests pass: **124/151 passing (82%), 6/8 suites passing** + - [x] Remaining failures are pre-existing issues (not caused by repository pattern) + +### **Phase 5: Documentation** (Day 5) + +**7. Add Documentation**: ⏳ **DEFERRED TO TD-006** +- [x] Add JSDoc comments to all repository interfaces +- [x] Add JSDoc comments to all repository implementations +- [ ] Create `docs/architecture/REPOSITORY-PATTERN.md` (will be part of TD-006): + - [ ] Explain repository pattern + - [ ] Show examples of usage + - [ ] Document testing patterns + - [ ] Add troubleshooting guide +- [ ] Update TESTING.md with repository mocking examples (will be part of TD-006) + +**Acceptance Criteria**: ✅ **MET** +- [x] All major entities have dedicated repository classes (LibraryItem, Highlight, Label, EntityLabel) +- [x] All repositories implement interfaces +- [x] Services use repositories exclusively (no DataSource injections in LibraryService, HighlightService, LabelService) +- [x] Repository pattern consistently applied across codebase +- [x] All unit tests pass (87 tests) +- [x] E2E tests passing (124/151, 82% - pre-existing failures not related to repository pattern) +- [ ] Contract tests created for each repository (deferred to TD-006) +- [x] Code is more maintainable and testable +- [x] Query logic isolated in repositories +- [x] Business logic isolated in services +- [ ] Documentation complete (deferred to TD-006) + +**Benefits After Completion**: +- ✅ Services are testable with simple mocks +- ✅ Can convert E2E tests to unit tests (TD-006) +- ✅ Clear separation of concerns +- ✅ Easier to refactor (change DB logic without touching services) +- ✅ Ready for Testcontainers (TD-006) + +- **Dependencies**: None (standalone refactoring). +- **Effort Estimate**: 5 days (1 week). +- **Status**: ⭐ **READY TO START** (highest priority) +- **Priority**: ⭐⭐⭐ **CRITICAL** (blocks TD-006, enables all testing improvements) + +--- + +## TD-004 Constants & Magic Strings ✅ **COMPLETED** + +- **Problem/Objective**: Magic strings throughout codebase (folder names like "inbox", "archive", config keys, library item states) make code brittle and error-prone. +- **Goal**: Replace all magic strings with typed constants using TypeScript const assertions. +- **Approach**: Create constants files for all magic strings. Tasks: + - [x] Create `src/constants/folders.constants.ts`: + ```typescript + export const FOLDERS = { + INBOX: 'inbox', + ARCHIVE: 'archive', + TRASH: 'trash', + ALL: 'all', + } as const; + export type FolderName = (typeof FOLDERS)[keyof typeof FOLDERS]; + export const VALID_FOLDERS = [FOLDERS.INBOX, FOLDERS.ARCHIVE, FOLDERS.TRASH] as const; + export const ALL_FOLDERS = [...VALID_FOLDERS, FOLDERS.ALL] as const; + ``` + - [x] Replace all folder magic strings in codebase: + - [x] Services (LibraryService - replaced all folder strings) + - [x] DTOs (library-inputs.type.ts - updated validation decorators) + - [x] Repositories (library-item.repository.ts - replaced folder logic) + - [x] Seeds (library-items.seed.ts) + - [x] Auth (default-user-resources.service.ts) + - [x] Test files (library.e2e-spec.ts, highlight.e2e-spec.ts, notebook.e2e-spec.ts, save-url.e2e-spec.ts) + - [x] Add TypeScript type guards for validation (isValidFolder, isPhysicalFolder) + - [x] Update validation decorators to use constants (@IsIn([...ALL_FOLDERS])) + - [ ] **DEFERRED**: Library state constants (will implement when needed for state management) + - [ ] **DEFERRED**: Config keys constants (will implement with configuration refactoring) +- **Acceptance Criteria**: + - [x] Zero folder magic strings in services and resolvers + - [x] All constants use TypeScript const assertions + - [x] Type-safe folder names throughout codebase + - [x] All tests passing (151/151 E2E tests passing, 100% pass rate) + - [x] Code is more maintainable and less error-prone + - [x] Autocomplete works for all folder constant values +- **Dependencies**: None (standalone refactoring). +- **Effort Estimate**: 1 day. +- **Actual Time**: ~3 hours +- **Status**: ✅ **COMPLETED** +- **Priority**: Medium (code quality improvement) +- **Test Results**: All 151 tests passing across 8 test suites + - Fixed critical bugs discovered during testing: + - HighlightEntity not registered in database.module.ts + - Test configuration synchronize setting + - Dynamic shortId generation to avoid duplicates + - GraphQL schema typos + - Database constraint violations + +--- + +## TD-006 Testing Infrastructure Improvements (Phases 1-2) ✅ **COMPLETED** + +- **Problem/Objective**: Testing infrastructure needs industry-standard approach. Current state: 57% E2E tests vs 43% unit tests (inverted pyramid). Need proper test infrastructure with ephemeral databases, zero maintenance, and factory pattern for test data. +- **Goal**: Implement **Testcontainers + Factory Pattern** for ephemeral databases and explicit test data generation. +- **Status**: ✅ **Phases 1-2 COMPLETED** - Core infrastructure ready +- **Actual Time**: ~1 week (spread across multiple sessions) +- **Results**: + - ✅ Testcontainers setup with PostgreSQL 15-alpine + - ✅ Factory pattern with 5 factories (Base, User, LibraryItem, Highlight, Label) + - ✅ Test count: 120 unit + 151 E2E = 271 total tests (100% pass rate) + - ✅ Test pyramid: 44% unit / 56% E2E (improving toward 85/10/5 target) + - ⏳ Phases 3-5 deferred (test conversion, documentation, deprecation of TD-001) + +**Why Critical**: +- Enables 10-100x faster unit tests with mocked repositories +- Eliminates test database maintenance (ephemeral containers) +- Allows parallel test execution (each suite gets own container) +- Industry best practice for scalable testing +- **Blocks**: All future testing improvements, production readiness + +### **Phase 1: Testcontainers Setup** ✅ **COMPLETE** + +**Tasks Completed**: +- [x] Installed `@testcontainers/postgresql` and `@faker-js/faker` +- [x] Created `test/setup/testcontainers.ts` with setup/teardown functions +- [x] Created `test/setup/testcontainers-setup.ts` (Jest global setup) +- [x] Created `test/setup/testcontainers-teardown.ts` (Jest global teardown) +- [x] Created `test/setup/transaction-rollback.ts` (per-test isolation via transactions) +- [x] Updated `test/jest-e2e.json` with global hooks +- [x] Verified container startup/shutdown (migrations run automatically) +- [x] PostgreSQL 15-alpine container with full schema synchronization +- [x] Transaction-based test isolation (BEGIN/ROLLBACK per test) + +**Key Implementation**: +```typescript +// test/setup/testcontainers.ts +export async function setupTestContainer() { + container = await new PostgreSqlContainer('postgres:15-alpine') + .withDatabase('test_omnivore') + .withUsername('test_user') + .withPassword('test_password') + .withExposedPorts(5432) + .start(); + + dataSource = new DataSource({ + type: 'postgres', + host: container.getHost(), + port: container.getPort(), + entities: [/* all entities */], + synchronize: false, + }); + + await dataSource.initialize(); + await dataSource.query('CREATE SCHEMA IF NOT EXISTS omnivore'); + await dataSource.synchronize(); // Apply schema from entities + return { container, dataSource }; +} +``` + +### **Phase 2: Factory Pattern** ✅ **COMPLETE** + +**Tasks Completed**: +- [x] Created `test/factories/base.factory.ts` with abstract BaseFactory +- [x] Created `test/factories/user.factory.ts` with UserFactory +- [x] Created `test/factories/library-item.factory.ts` with LibraryItemFactory +- [x] Created `test/factories/highlight.factory.ts` with HighlightFactory +- [x] Created `test/factories/label.factory.ts` with LabelFactory +- [x] Added helper methods for common scenarios (admin(), archived(), withUser(), etc.) +- [x] Created `test/factories/index.ts` to export all factories +- [x] Tested factories: build() for in-memory, create() for database persistence +- [x] Created example test demonstrating factory usage + +**Factory Pattern Implementation**: +```typescript +// Base factory with build() and create() methods +export abstract class BaseFactory { + build(overrides?: DeepPartial): Entity { + const defaults = this.generateDefaults(); + return { ...defaults, ...(overrides || {}) } as Entity; + } + + async create(overrides?: DeepPartial): Promise { + const entity = this.build(overrides); + const repository = this.getRepository(); + return await repository.save(entity as any); + } + + async createMany(count: number, overrides?: DeepPartial): Promise { + const entities: Entity[] = []; + for (let i = 0; i < count; i++) { + entities.push(await this.create(overrides)); + } + return entities; + } + + protected abstract generateDefaults(): DeepPartial; + protected abstract getRepository(): Repository; +} + +// Example: UserFactory with helper methods +class UserFactoryClass extends BaseFactory { + protected generateDefaults() { + return { + id: faker.string.uuid(), + email: faker.internet.email().toLowerCase(), + name: faker.person.fullName(), + password: bcrypt.hashSync('password123', 10), + role: UserRole.USER, + status: StatusType.ACTIVE, + }; + } + + async admin(overrides: Partial = {}): Promise { + return this.create({ role: UserRole.ADMIN, ...overrides }); + } +} + +export const UserFactory = new UserFactoryClass(); +``` + +### **Phase 3-5: Deferred** ⏳ + +**Remaining Work** (to be completed when converting E2E to unit tests): +- [ ] Phase 3: Convert 80-90 E2E tests to unit tests with mocked repositories +- [ ] Phase 4: Update TESTING.md documentation with factory patterns +- [ ] Phase 5: Deprecate TD-001 utilities (test database scripts) + +**Current Test Distribution**: +- **Unit Tests**: 120 tests (44% - up from 87, 38% increase) +- **E2E Tests**: 151 tests (56% - 100% pass rate) +- **Total**: 271 tests +- **Target**: 85% unit / 10% integration / 5% E2E (170+ unit / 30-40 E2E) + +**Acceptance Criteria**: +- [x] Testcontainers integrated with PostgreSQL ✅ +- [x] Factory pattern implemented for all entities ✅ +- [x] Test execution with ephemeral containers ✅ +- [x] Transaction rollback for test isolation ✅ +- [x] All tests passing (271 total) ✅ +- [ ] Test conversion to unit tests (deferred to future work) +- [ ] Documentation complete (deferred to future work) +- [ ] TD-001 utilities deprecated (deferred to future work) + +**Benefits Achieved**: +- ✅ Zero test database maintenance (ephemeral containers) +- ✅ Proper test isolation (transaction-based) +- ✅ Explicit test data (factories replace seed files) +- ✅ Ready for parallel test execution +- ✅ Industry-standard testing infrastructure +- ✅ Easy to add new tests (factories handle complexity) + +**Files Created**: +- `packages/api-nest/test/setup/testcontainers.ts` - Container setup/teardown +- `packages/api-nest/test/setup/testcontainers-setup.ts` - Jest global setup +- `packages/api-nest/test/setup/testcontainers-teardown.ts` - Jest global teardown +- `packages/api-nest/test/setup/transaction-rollback.ts` - Per-test isolation +- `packages/api-nest/test/factories/base.factory.ts` - Abstract base factory +- `packages/api-nest/test/factories/user.factory.ts` - User factory with helpers +- `packages/api-nest/test/factories/library-item.factory.ts` - LibraryItem factory +- `packages/api-nest/test/factories/highlight.factory.ts` - Highlight factory +- `packages/api-nest/test/factories/label.factory.ts` - Label factory +- `packages/api-nest/test/factories/index.ts` - Factory exports +- `packages/api-nest/test/factories-example.e2e-spec.ts` - Usage example + +- **Dependencies**: TD-003 (Repository Pattern) ✅ Complete +- **Effort Estimate**: 5 days for all phases (1 week) +- **Actual Time**: ~1 week for Phases 1-2 (Phases 3-5 deferred) +- **Priority**: ⭐⭐⭐ **CRITICAL** (foundation for all testing improvements) +- **Next Steps**: Convert E2E tests to unit tests when refactoring services (incremental, no rush) + + +--- + +## ARC-010: Reading Progress & Highlights (Backend + Frontend) ✅ **COMPLETED** + +- **Merged to Main**: November 21, 2025 (PR #18) +- **Problem/Objective**: Implement robust reading progress tracking and highlight system with modern data model +- **Approach**: Migrate from percentage-based to sentinel-based progress tracking, implement W3C Web Annotation-aligned selectors + +### Deliverables Completed: + +#### 1. Sentinel-Based Reading Progress Tracking ✅ +- **Database Migration 0196**: Added `reading_progress` table with sentinel-based tracking +- **Entity**: Created `ReadingProgressEntity` with content versioning +- **GraphQL API**: Complete CRUD operations for reading progress +- **Content Versioning**: Hash-based tracking for content changes +- **Sentinel System**: Multi-sentinel positioning for accurate progress tracking + +#### 2. Robust Anchored Selectors for Highlights ✅ +- **JSONB Storage**: Multi-strategy selector storage (CSS, XPath, text position, quote) +- **W3C Alignment**: Follows Web Annotation Data Model standards +- **Backward Compatibility**: Maintains support for legacy highlight format +- **Content Version Tracking**: Detects when highlighted content changes +- **Entity Updates**: Enhanced `HighlightEntity` with selectors field + +#### 3. Symbol-Based Injection Tokens ✅ +- **Created**: `injection-tokens.ts` with repository tokens as Symbols +- **Migrated**: All repository injections from strings to Symbols +- **Type Safety**: Prevents token collision, improves IDE support +- **Services Updated**: HighlightService, LabelService, LibraryService, ReadingProgressService + +### Technical Details: + +**Backend (NestJS)**: +- [x] Database migration 0196 for reading progress schema +- [x] `ReadingProgressEntity` with sentinel-based fields +- [x] `ReadingProgressModule` with service and resolver +- [x] GraphQL queries: + - [x] `readingProgress(libraryItemId: String!): ReadingProgress` + - [x] `readingProgressList(itemIds: [String!]!): [ReadingProgress!]!` +- [x] GraphQL mutations: + - [x] `updateReadingProgress(input: UpdateReadingProgressInput!): ReadingProgress!` + - [x] `saveReadingPosition(input: SaveReadingPositionInput!): ReadingProgress!` +- [x] Content hash generation for version tracking +- [x] Sentinel management (create, update, validate) +- [x] Enhanced `HighlightEntity` with selectors JSONB field +- [x] Highlight queries support selector strategies +- [x] Symbol-based injection token refactoring + +**Frontend (web-vite)**: +- [x] Reading progress bar component +- [x] Progress persistence on scroll +- [x] Resume reading from last position +- [x] Highlight creation with robust positioning +- [x] Highlight sidebar improvements +- [x] Notebook modal enhancements +- [x] Integration with reader page scroll tracking + +**Testing**: +- [x] E2E tests for reading progress (create, update, resume) +- [x] E2E tests for highlights with selectors +- [x] Content version change detection tests +- [x] Sentinel boundary validation tests +- [x] All existing tests still passing (174 E2E + 87 unit) + +### Acceptance Criteria: ✅ **ALL MET** +- [x] Reading progress tracks position with multiple sentinels +- [x] Progress persists and resumes correctly +- [x] Highlights use robust anchored selectors +- [x] W3C Web Annotation Data Model alignment +- [x] Content version changes detected +- [x] Backward compatibility maintained +- [x] Symbol-based DI prevents token collisions +- [x] All tests passing (261 total) +- [x] Performance acceptable (<100ms for progress updates) + +### Key Decisions: +- **Sentinel-based vs Percentage**: More accurate, handles dynamic content +- **JSONB for Selectors**: Flexible multi-strategy positioning +- **W3C Alignment**: Future-proof for annotation standards +- **Symbol Tokens**: Type-safe dependency injection + +### Files Modified/Created: +**Backend**: +- Migration: `packages/db/migrations/0196.do.reading-progress-sentinel-tracking.sql` +- Entity: `packages/api-nest/src/reading-progress/entities/reading-progress.entity.ts` +- Module: `packages/api-nest/src/reading-progress/*` +- Updated: `packages/api-nest/src/highlight/entities/highlight.entity.ts` (added selectors) +- Updated: `packages/api-nest/src/library/entities/library-item.entity.ts` (content hash) +- Token Refactor: `packages/api-nest/src/common/injection-tokens.ts` +- Tests: `packages/api-nest/test/reading-progress.e2e-spec.ts` + +**Frontend**: +- Component: `packages/web-vite/src/components/ReadingProgressBar.tsx` +- Updated: `packages/web-vite/src/components/HighlightSidebar.tsx` +- Updated: `packages/web-vite/src/components/NotebookModal.tsx` +- Updated: `packages/web-vite/src/pages/ReaderPage.tsx` + +### Dependencies: +- ARC-010A (Minimal Reader) ✅ Complete +- ARC-005 (Library Core Mutations) ✅ Complete + +### Effort: +- **Estimate**: 3-4 days +- **Actual**: ~1 week (included refactoring + extensive testing) + +### Status: ✅ **COMPLETED** (Merged November 21, 2025) + +### Impact: +- **User Experience**: More accurate reading progress, robust highlights survive content changes +- **Developer Experience**: Type-safe DI tokens, clean data model +- **Standards Compliance**: W3C Web Annotation alignment enables future interoperability +- **Performance**: Efficient sentinel-based tracking, optimized queries + +--- + +**Document Last Updated**: November 21, 2025 +**Total Completed ARCs**: 17 (including ARC-010) +**Total Completed TDs**: 3 (TD-003, TD-004, TD-006 Phases 1-2) diff --git a/docs/architecture/unified-migration-backlog.md b/docs/architecture/unified-migration-backlog.md index aa29a46e1..90ed7785c 100644 --- a/docs/architecture/unified-migration-backlog.md +++ b/docs/architecture/unified-migration-backlog.md @@ -1,1081 +1,401 @@ # Unified Migration Backlog: Express to NestJS -This backlog consolidates the simplified and original migration strategies into actionable tickets. Each ticket represents a deployable increment that can be tracked in Notion/Todoist. +**Last Updated**: November 21, 2025 -**Key Approach**: Start with a new NestJS service (Node.js 24 LTS) running alongside Express, then migrate features slice-by-slice until we can decommission the old services. +This is the **active working backlog** containing only pending and in-progress items. Completed items have been moved to `unified-migration-backlog-complete.md`. -## 🎯 Current Status & Next Steps - -### ✅ **COMPLETED** (Major Milestone Achieved) - -- **ARC-001**: NestJS Package Setup - Complete infrastructure -- **ARC-002**: Health Checks & Observability - Monitoring ready -- **ARC-003**: Authentication Module - Full auth system with web integration -- **ARC-003B**: Database & Entity Integration - TypeORM entities working -- **ARC-004**: GraphQL Module Setup - Base schema + authentication context working -- **ARC-004B**: Vite Migration (Partial) - Basic library page integrated with GraphQL -- **ARC-005**: Library Core Mutations - Archive, delete, reading progress, folder management -- **ARC-006**: Advanced Search & Filtering - Full-text search, folder filters, sorting -- **ARC-006B**: Performance & UX Optimizations - 26x faster queries, simplified logging, improved UX -- **ARC-007**: Bulk Operations & Multi-select - Select multiple items, bulk actions with transactions -- **ARC-008**: Labels System - Complete label management with filtering -- **ARC-011**: Add Link & Content Ingestion - Save URLs with modal, validation, E2E tests (17 passing) -- **ARC-010A**: Minimal Reader - Basic article reader with sanitization, responsive design, state handling -- **ARC-012** (80% complete): Queue Infrastructure - BullMQ, EventBus, workers, full test coverage (87 unit + 116 E2E) -- **Performance Optimization**: 25-50x faster development + 8-30x faster database queries - -### 🔄 **IN PROGRESS** - -1. **ARC-012**: Queue Integration & Background Processing (80% complete - Phase 5 pending) ⭐ **CURRENT** - - ✅ Phases 1-4 complete (infrastructure, events, workers, integration) - - ⏳ Content fetching implementation (stub needs real readability extraction) - - ⏸️ Phase 5 monitoring deferred (BullMQ Board, metrics, load testing) - -### 🎯 **READY TO START** (Recommended Order) - -1. **ARC-013**: Advanced Content Processing (4-5 days) - Completes ARC-012 content fetching -3. **ARC-009**: Frontend Library Feature Parity (5-7 days) -4. **ARC-010**: Reading Progress & Highlights (3-4 days) -5. **ARC-007B**: Architecture Refinements (1-2 days) - Technical debt cleanup - -### ⏳ **PENDING TESTING** (Lower Priority) - -- Google OAuth integration testing -- Apple OAuth integration testing -- Email verification (pending email service integration) - -### 🎯 **RECOMMENDED NEXT**: ARC-005 Library Core Mutations - -With GraphQL and basic library listing working, implement core mutations (archive, delete, mark-read) to unblock frontend action buttons and establish mutation patterns for remaining features. +**Key Approach**: Start with a new NestJS service (Node.js 22 LTS) running alongside Express, then migrate features slice-by-slice until we can decommission the old services. --- -## ARC-001 NestJS Package Setup ✅ **COMPLETED** - -- **Problem/Objective**: Create the foundational NestJS package and Docker infrastructure to run alongside existing Express API without disruption. -- **Approach**: Bootstrap NestJS application with proper workspace integration and Docker configuration. Tasks: - - [x] Create `packages/api-nest` directory structure - - [x] Initialize NestJS project with `nest new api-nest --skip-git` - - [x] Configure TypeScript with strict settings extending workspace root - - [x] Set up package.json with proper scripts and dependencies - - [x] Create Docker service in docker-compose.yml for new API on port 4001 - - [x] Configure environment loading and basic logging -- **Acceptance Criteria**: ✅ **ALL COMPLETED** - - [x] NestJS application boots without errors on port 4001 - - [x] Docker Compose runs both Express API (4000) and NestJS API (4001) - - [x] TypeScript compilation works with workspace configuration - - [x] Basic logging and environment loading functional -- **Dependencies**: None. -- **Effort Estimate**: 2 days. -- **Status**: ✅ Completed - -## ARC-002 Health Checks & Observability ✅ **COMPLETED** - -- **Problem/Objective**: Establish basic health monitoring and structured logging before migrating business logic. -- **Approach**: Set up comprehensive health checking and observability infrastructure. Tasks: - - [x] Install `@nestjs/terminus` for health checks - - [x] Create `/api/health` endpoint for basic status - - [x] Create `/api/health/deep` endpoint with database and Redis connectivity checks - - [x] Set up request logging middleware matching Express format - - [x] Configure structured logging with consistent error handling -- **Acceptance Criteria**: ✅ **ALL COMPLETED** - - [x] `/api/health` returns 200 with basic status - - [x] `/api/health/deep` checks database and Redis connectivity - - [x] Request/response logging matches Express format - - [x] Error handling returns consistent JSON responses -- **Dependencies**: ARC-001. -- **Effort Estimate**: 1 day. -- **Status**: ✅ Completed - -## ARC-003 Authentication Module ✅ **COMPLETED** - -- **Problem/Objective**: Migrate authentication to NestJS with improved validation while maintaining JWT compatibility with Express. -- **Approach**: Build comprehensive authentication system in NestJS with enhanced security. Tasks: - - [x] Create `AuthModule` with JWT strategy and passport integration - - [x] Implement authentication guards and decorators for route protection - - [x] Create `/api/v2/auth/*` endpoints for login, register, and OAuth flows - - [x] Set up OAuth providers structure (Google, Apple) - - [x] Add rate limiting and security middleware - - [x] Implement comprehensive E2E testing with user personas - - [x] Add Swagger/OpenAPI documentation -- **Acceptance Criteria**: ✅ **ALL COMPLETED** - - [x] `/api/v2/auth/login` works alongside Express `/api/auth/login` - - [x] JWT tokens are compatible between Express and NestJS APIs - - [x] Role-based access control (RBAC) implemented - - [x] Comprehensive test coverage achieved (>90%) -- **Dependencies**: ARC-001, ARC-002. -- **Effort Estimate**: 3 days. -- **Status**: ✅ Completed (3 days actual) - -## ARC-003B Database & Entity Integration ✅ **COMPLETED** - -- **Problem/Objective**: Integrate NestJS with existing PostgreSQL schema without breaking Express API or requiring complex migrations. -- **Approach**: Create TypeORM entities mapping to existing tables using hybrid migration strategy. Tasks: - - [x] Fix User entity to map exactly to existing schema (migrations 0001-0188) - - [x] Create UserProfile entity mapping to `user_profile` table (migration 0019) - - [x] Create UserPersonalization entity mapping to `user_personalization` table (migrations 0008+) - - [x] Create first new migration (0189) for role column using existing Postgrator system - - [x] Update DatabaseModule to include all entities - - [x] Update UserModule with full entity support - - [x] Document repeatable process for future entity migrations -- **Acceptance Criteria**: ✅ **ALL COMPLETED** - - [x] Entities map exactly to existing database schema - - [x] New role column added via traditional migration system - - [x] Both APIs can access same database tables - - [x] Repeatable process documented for future entities -- **Dependencies**: ARC-003. -- **Effort Estimate**: 2 days. -- **Status**: ✅ Completed (1 day actual) - -## ARC-004 GraphQL Module Setup ✅ **COMPLETED** - -- **Problem/Objective**: Set up GraphQL in NestJS to work alongside Express GraphQL without breaking existing clients. -- **Approach**: Establish parallel GraphQL endpoint in NestJS to gradually migrate resolvers from Express. Tasks: - - [x] Install `@nestjs/graphql` and `@nestjs/apollo` packages - - [x] Configure GraphQL module with Apollo Driver on `/api/graphql` path (aligned with Vite + legacy clients) - - [x] Create base GraphQL schema with essential types (User, AuthPayload) - - [x] Implement authentication context middleware to extract JWT tokens - - [x] Create initial resolvers (viewer + session) returning authenticated context - - [x] Add schema introspection and playground for development - - [x] Add Jest e2e coverage for `/api/graphql` viewer/session flows - - [x] Create LibraryModule with LibraryItemEntity mapping to existing `library_item` table - - [x] Implement `libraryItems` query with cursor-based pagination - - [x] Implement `libraryItem(id)` query for single item lookup -- **Acceptance Criteria**: ✅ **ALL COMPLETED** - - [x] GraphQL endpoint accessible at `/api/graphql` - - [x] Authentication context properly extracts user from JWT tokens - - [x] Viewer query returns current user data matching Express format - - [x] Schema introspection works without errors - - [x] Both Express and NestJS GraphQL endpoints function simultaneously - - [x] LibraryItemEntity correctly maps to existing database schema - - [x] Library queries return paginated results with proper type safety -- **Dependencies**: ARC-003B. -- **Effort Estimate**: 2 days. -- **Status**: ✅ Completed - -## ARC-004B Frontend Performance Optimization (Vite Migration) ✅ **FOUNDATION COMPLETE** - -- **Problem/Objective**: Migrate from Next.js to Vite for dramatically improved development experience and build performance. -- **Approach**: Complete frontend migration to Vite + React Router for 50-100x performance gains. Tasks: - - [x] Create Vite configuration with React, TypeScript, and SWC - - [x] Set up React Router for client-side routing with auth guards - - [x] Create packages/web-vite with initial structure - - [x] Configure GraphQL client targeting `/api/graphql` - - [x] Implement authentication store with JWT token management - - [x] Create basic LibraryPage component fetching from NestJS GraphQL - - [x] Integrate `libraryItems` query with pagination - - [x] Create all page stubs (Login, Register, Settings, Reader, Admin) - - [x] Implement protected routes and navigation - - [ ] ~~Implement advanced library features~~ → **Moved to ARC-009** - - [ ] ~~Configure Vite plugins for optimization~~ → **Infrastructure (can be done anytime)** - - [ ] ~~Update build pipeline and Docker~~ → **Infrastructure (can be done anytime)** - - [ ] ~~Update testing configuration~~ → **Infrastructure (can be done anytime)** -- **Acceptance Criteria**: ✅ **FOUNDATION COMPLETE** - - [x] Basic library page loads and displays items - - [x] Authentication flow works with login/logout - - [x] GraphQL queries successfully fetch from NestJS backend - - [x] All routes configured with proper protection - - [x] Dev experience significantly improved (HMR working) - - [ ] ~~Feature parity with legacy library UI~~ → **See ARC-009** - - [ ] ~~Production build optimization~~ → **Infrastructure backlog** -- **Dependencies**: ARC-003, ARC-004. -- **Effort Estimate**: Foundation: 1 week ✅ Complete | Remaining UI features: See ARC-009 -- **Status**: ✅ Foundation Complete - Ready for backend-driven feature development -- **Note**: Remaining UI features naturally roll into ARC-009 after backend APIs are ready (ARC-005 through ARC-008) - -## ARC-005 Library Core Mutations ✅ **COMPLETED** - -- **Problem/Objective**: Implement essential library item mutations to enable basic user actions without content processing. -- **Approach**: Add GraphQL mutations for core library management operations that don't require queue/content processing. This unblocks frontend action buttons and establishes mutation patterns. Tasks: - - **Backend (NestJS):** - - [x] Add mutations to LibraryResolver: - - [x] `archiveLibraryItem(id: String!, archived: Boolean!): LibraryItem!` - - [x] `deleteLibraryItem(id: String!): DeleteResult!` - - [x] `updateReadingProgress(id: String!, progress: ReadingProgressInput!): LibraryItem!` - - [x] `moveLibraryItemToFolder(id: String!, folder: String!): LibraryItem!` - - [x] Implement service methods in LibraryService: - - [x] `archive(userId, itemId, archived)` - update state column - - [x] `delete(userId, itemId)` - soft delete or hard delete based on current folder - - [x] `updateProgress(userId, itemId, progressInput)` - update reading progress fields - - [x] `moveToFolder(userId, itemId, folder)` - update folder column - - [x] Add input types to GraphQL schema: - - [x] `ReadingProgressInput` (topPercent, bottomPercent, anchorIndex) - - [x] `DeleteResult` (success, message) - - [x] Add validation and error handling for all mutations - - [x] Create E2E tests for each mutation covering success and error cases (18 tests, all passing) - - **Frontend (web-vite):** - - [x] Create mutation hooks in packages/web-vite/src/lib/graphql-client.ts: - - [x] `useArchiveItem()` hook - - [x] `useDeleteItem()` hook - - [x] `useUpdateReadingProgress()` hook - - [x] `useMoveToFolder()` hook - - [x] Wire mutations to LibraryPage action buttons - - [x] Add optimistic updates for better UX - - [x] Add success/error toast notifications - - [x] Handle loading states during mutation execution - -- **Acceptance Criteria**: ✅ **ALL COMPLETED** - - [x] Archive button archives/unarchives items successfully - - [x] Delete button removes items from library with confirmation - - [x] Reading progress updates persist correctly - - [x] Move to folder changes item location - - [x] All mutations work with proper authentication - - [x] Error handling displays user-friendly messages - - [x] Optimistic UI updates provide instant feedback - - [x] E2E tests achieve >90% coverage (18/18 passing) - - [x] Mutations maintain data consistency with database -- **Dependencies**: ARC-004, ARC-004B. -- **Effort Estimate**: 3-5 days. -- **Actual Time**: ~1 day -- **Status**: ✅ Completed - -## ARC-006 Advanced Search & Filtering ✅ **COMPLETED** - -- **Problem/Objective**: Implement comprehensive search and filtering capabilities to match legacy system functionality. -- **Approach**: Add full-text search, advanced filters, and sorting to library queries. Tasks: - - **Backend (NestJS):** - - [x] Enhance `libraryItems` query parameters: - - [x] Add `searchQuery: String` for full-text search - - [x] Add `folder: String` filter (inbox, archive, trash, all) - - [x] Add `state: LibraryItemState` filter - - [x] Add `sortBy: String` (savedAt, updatedAt, publishedAt, title, author) - - [x] Add `sortOrder: String` (ASC, DESC) - - [x] Implement full-text search in LibraryService: - - [x] Basic ILIKE search across title/description/author - - [ ] **DEFERRED**: PostgreSQL `ts_vector` full-text search (performance optimization) - - [ ] **DEFERRED**: Support multi-word queries with proper ranking - - [ ] **DEFERRED**: Handle special search operators (in:, is:, label:, has:) - - [x] Add query builder logic for complex filters - - [ ] **TODO**: Optimize database queries with proper indexes - - [x] Add query validation and sanitization - - [x] Create E2E tests for search scenarios (12 new tests, 30/30 passing) - - **Frontend (web-vite):** - - [x] Enhance search box with debounced input (300ms) - - [ ] **DEFERRED**: Add visual query builder UI (optional) - - [ ] **DEFERRED**: Implement search suggestions/typeahead - - [x] Add folder filter tabs (Inbox, Archive, All, Trash) - - [x] Add sort controls (saved date, updated date, published date, title, author) - - [x] Show search result count - - [ ] **DEFERRED**: Add search history/saved searches - - [x] Handle debounced search input - - [x] Add loading indicators during search - -- **Acceptance Criteria**: - - [x] Full-text search returns relevant results (basic ILIKE matching) - - [x] Folder filters correctly scope results - - [x] State filters work correctly (archived, deleted, etc.) - - [x] Sort controls change result ordering - - [ ] **DEFERRED**: Search query syntax matches legacy system (in:inbox, label:tech, etc.) - - [ ] **TODO**: Search performance acceptable (<500ms for typical queries) - needs indexes - - [x] Empty search states display helpful messages - - [x] Search works correctly with pagination - - [ ] **DEFERRED**: Legacy search queries migrate seamlessly -- **Dependencies**: ARC-005. -- **Effort Estimate**: 2-3 days. -- **Actual Time**: ~4 hours -- **Status**: ✅ Completed (with performance optimizations deferred to ARC-006B) - -## ARC-006B Performance & UX Optimizations ✅ **COMPLETED** - -- **Problem/Objective**: Optimize search performance, logging, and UX based on initial implementation feedback. -- **Approach**: Add database indexes, simplify logging, improve debounce behavior, add query monitoring. Tasks: - - **Performance:** - - [x] Add PostgreSQL indexes for search fields (title, author, description, folder, state, savedAt) - - [x] Add pg_trgm extension for fast ILIKE queries - - [x] Add GIN indexes for array columns (labels) - - [x] Created migration 0190 with 8 strategic indexes - - [x] Benchmark query performance and set targets (<200ms for search) - - **Logging:** - - [x] Simplify structured logging format for better readability - - [x] Create dev-friendly format (one-line with key info) - - [x] Keep structured format for production - - [x] Add color coding for log levels - - **Query Monitoring:** - - [x] Create TypeORM query logger to track slow queries - - [x] Add execution time threshold (warn if >500ms) - - [x] Log query execution times in development - - [x] Create QueryTimer utility for manual timing - - **UX Improvements:** - - [x] Fix search debounce to not trigger loading on empty query - - [x] Add "searching..." indicator separate from full page load - - [x] Separate loading vs searching states - - [x] Smart debounce: 300ms for search, 0ms for folder changes - - [x] Show result count prominently - -- **Acceptance Criteria**: ✅ **ALL COMPLETED** - - [x] Search queries execute in <200ms with indexes (tested: ~150ms) - - [x] Logs are readable in terminal without JSON parsing - - [x] Slow queries (>500ms) are logged with details - - [x] Deleting search text doesn't cause jarring reload - - [x] Users can type rapidly without performance issues -- **Dependencies**: ARC-006. -- **Effort Estimate**: 1-2 days. -- **Actual Time**: ~1 day -- **Status**: ✅ Completed -- **Files Created**: - - `packages/db/migrations/0190.do.add_library_item_search_indexes.sql` - - `packages/db/migrations/0190.undo.add_library_item_search_indexes.sql` - - `packages/db/migrations/0190.README.md` - - `packages/api-nest/src/database/query-logger.ts` - - `packages/api-nest/PERFORMANCE_OPTIMIZATIONS.md` -- **Performance Impact**: - - Folder filter: 26x faster (~800ms → ~30ms) - - Text search: 8x faster (~1200ms → ~150ms) - - Sort operations: 30x faster (~600ms → ~20ms) - -## ARC-007 Bulk Operations & Multi-select ✅ **COMPLETED** - -- **Problem/Objective**: Enable power users to perform actions on multiple library items simultaneously. -- **Approach**: Implement bulk mutations that operate on multiple items efficiently. Tasks: - - **Backend (NestJS):** - - [x] Add bulk mutations to LibraryResolver: - - [x] `bulkArchiveItems(itemIds: [String!]!, archived: Boolean!): BulkActionResult!` - - [x] `bulkDeleteItems(itemIds: [String!]!): BulkActionResult!` - - [x] `bulkMoveToFolder(itemIds: [String!]!, folder: String!): BulkActionResult!` - - [x] `bulkMarkAsRead(itemIds: [String!]!): BulkActionResult!` - - [x] Implement bulk operations in LibraryService: - - [x] Support explicit item ID lists - - [x] Use database transactions for atomicity - - [x] Implement batch processing (100 items per batch) - - [x] Handle partial failures gracefully - - [x] Add GraphQL types: - - [x] `BulkActionResult` (success, successCount, failureCount, errors, message) - - [x] Add bulk operation limits (1000 items max) and validation - - [x] Create E2E tests for bulk scenarios (14 tests, all passing) - - **Frontend (web-vite):** - - [x] Implement multi-select mode UI: - - [x] Add checkbox to each library card - - [x] Add "Select All" / "Deselect All" controls - - [x] Show multi-select action bar when items selected - - [x] Add visual indicators for selected items - - [x] Multi-Select toggle button - - [x] Create bulk action buttons: - - [x] Archive/Unarchive selected - - [x] Delete selected - - [x] Move to folder (inbox, archive) - - [x] Mark as read - - [x] Add bulk action confirmation modals - - [x] Handle partial failures gracefully - - [x] Show success/failure counts via toast notifications - - [ ] **DEFERRED**: Keyboard shortcuts for multi-select (Shift+Click, Cmd+A) - - [ ] **DEFERRED**: Query-based selection (all items matching search) - -- **Acceptance Criteria**: ✅ **CORE COMPLETE** - - [x] Users can select multiple items via checkboxes - - [x] Bulk actions execute successfully on selected items - - [x] Bulk operations maintain data consistency (transactions) - - [x] Partial failures are reported clearly - - [x] Multi-select UI functional and intuitive - - [x] Bulk operations have reasonable performance (batched processing) - - [x] Optimistic UI updates provide instant feedback - - [ ] **DEFERRED**: Keyboard shortcuts (future enhancement) - - [ ] **DEFERRED**: Query-based bulk actions (future enhancement) -- **Dependencies**: ARC-005, ARC-006. -- **Effort Estimate**: 2 days. -- **Actual Time**: ~2 hours -- **Status**: ✅ Completed -- **Test Coverage**: 44/44 tests passing (30 existing + 14 new bulk operation tests) - -## ARC-007B Architecture Refinements (Technical Debt) - -- **Problem/Objective**: Address identified architectural concerns and technical debt before adding more complex features. -- **Approach**: Refactor existing code to follow NestJS best practices and improve maintainability. Tasks: - - **Constants & Type Safety:** - - [ ] Create constants file for folder names (`FOLDER_INBOX`, `FOLDER_ARCHIVE`, `FOLDER_TRASH`) - - [ ] Create constants for library item states (extract from enum) - - [ ] Create constants for config keys (all `EnvVariables` references) - - [ ] Replace all magic strings with constants throughout codebase - - [ ] Add TypeScript const assertions for immutability - - **Repository Pattern:** - - [ ] Create `LibraryItemRepository` class extending TypeORM Repository - - [ ] Move all DataSource operations from `LibraryService` to repository - - [ ] Move bulk operations (transaction logic) into repository methods - - [ ] Create `UserRepository` class for user-specific database operations - - [ ] Update services to use repositories exclusively (remove DataSource injections) - - [ ] Update tests to mock repositories instead of DataSource - - **Service Layer Cleanup:** - - [ ] Review `LibraryService` - ensure business logic only, no direct DB queries - - [ ] Review `AuthService` - move seedLibraryItems to dedicated seeding service - - [ ] Ensure consistent error handling patterns across services - - [ ] Add JSDoc comments to public service methods - - **Testing:** - - [ ] Verify all unit tests still pass after refactoring - - [ ] Verify all E2E tests still pass after refactoring - - [ ] Add integration tests for repository methods - -- **Acceptance Criteria**: - - [ ] Zero magic strings in services/resolvers (all constants) - - [ ] Services use repositories exclusively (no DataSource injections) - - [ ] Repository pattern consistently applied across all entities - - [ ] All tests passing (unit, integration, E2E) - - [ ] Code is more maintainable and follows NestJS best practices -- **Dependencies**: ARC-007. -- **Effort Estimate**: 1-2 days. -- **Priority**: Medium (can be done after ARC-008 or ARC-009) -- **Status**: Pending (documented technical debt) - -## ARC-008 Labels System ✅ **COMPLETED** - -- **Problem/Objective**: Implement label management to enable users to organize and filter their library items. -- **Approach**: Create comprehensive label system with CRUD operations and item associations. Tasks: - - **Backend (NestJS):** ✅ **COMPLETE** - - [x] Create Label and EntityLabel entities mapping to existing database schema - - [x] Label entity: id, name, color, description, position, internal, timestamps, userId - - [x] EntityLabel junction table for many-to-many with library items - - [x] Create LabelModule with service and resolver - - [x] Add GraphQL queries: - - [x] `labels: [Label!]!` - list all user's labels ordered by position - - [x] `label(id: String!): Label` - get single label - - [x] Add GraphQL mutations with validation: - - [x] `createLabel(input: CreateLabelInput!): Label!` - with duplicate name check - - [x] `updateLabel(id: String!, input: UpdateLabelInput!): Label!` - with internal label protection - - [x] `deleteLabel(id: String!): DeleteResult!` - with internal label protection - - [x] `setLibraryItemLabels(itemId: String!, labelIds: [String!]!): [Label!]!` - replace item labels - - [x] Update LibraryItemEntity with EntityLabel relation - - [x] Add field resolver for labels in LibraryResolver - - [x] Add comprehensive input validation: - - [x] Label name: 1-100 chars, unique per user - - [x] Color: Hex format (#FF5733) - - [x] Description: 0-500 chars - - [x] Register entities in DatabaseModule - - [x] Schema generation complete with all types and mutations - - [x] Fix label filtering by syncing label_names column when labels are assigned - - [x] Database migration 0191 for labels.updated_at default value - - [ ] **DEFERRED**: E2E tests (testing infrastructure needs updates) - - **Frontend (web-vite):** ✅ **COMPLETE** - - [x] Create Labels management page: - - [x] List all labels with colors - - [x] Create new label form - - [x] Edit label inline - - [x] Delete label with confirmation - - [x] Add label selection UI to library items: - - [x] Label picker dropdown component - - [x] Multi-select label checkboxes - - [x] Visual label chips on cards - - [x] Add label filtering to search: - - [x] Filter by label dropdown - - [x] Show active label filters count - - [x] Clear individual label filters - - [x] Create label management hooks: - - [x] `useLabels()` - fetch all labels - - [x] `useCreateLabel()` - create new label - - [x] `useUpdateLabel()` - update existing label - - [x] `useDeleteLabel()` - delete label - - [x] `useSetLibraryItemLabels()` - assign labels to item - -- **Acceptance Criteria**: ✅ **ALL COMPLETED** - - [x] Users can create, update, and delete labels - - [x] Labels can be assigned to library items - - [x] Multiple labels per item supported - - [x] Label filtering works in search - - [x] Label colors display correctly in UI - - [x] Label deletion handles item associations gracefully (cascade delete) - - [x] Label assignment syncs both entity_labels and label_names columns - - [x] Label names are unique per user - - [x] Label UI provides intuitive dropdown picker -- **Dependencies**: ARC-005, ARC-006. -- **Effort Estimate**: 2-3 days. -- **Actual Time**: ~1 day -- **Status**: ✅ Completed -- **Key Fixes Applied**: - - Fixed LabelPicker to convert label names to UUIDs before API call - - Added schema specification to LibraryItemEntity (`schema: 'omnivore'`) - - Fixed all column name mappings (snake_case vs camelCase) - - Created migration 0191 for `labels.updated_at` default value - - Updated `setLibraryItemLabels` to sync `label_names` column for filtering - - Injected LibraryItemEntity repository into LabelService for column updates - -## ARC-009 Frontend Library Feature Parity - -- **Problem/Objective**: Achieve complete feature parity with legacy library UI for production readiness. -- **Approach**: Implement all remaining UI features and polish to match legacy system. Tasks: - - **Layout & Display:** - - [ ] Implement grid layout view (LibraryGridCard component) - - [ ] Implement list layout view (LibraryListCard component) - - [ ] Add layout toggle button (grid/list) - - [ ] Persist layout preference to localStorage - - [ ] Make layouts responsive (mobile, tablet, desktop) - - [ ] Add thumbnail/cover image display - - [ ] Show reading progress indicators - - [ ] Add state badges (processing, failed, archived) - - **Interactions:** - - [ ] Implement hover actions menu - - [ ] Add context menu (right-click) - - [ ] Add keyboard navigation (j/k, arrows) - - [ ] Add keyboard shortcuts for actions: - - [ ] e = archive/unarchive - - [ ] # = delete - - [ ] l = edit labels - - [ ] t = open notebook - - [ ] - = mark as read - - [ ] Enter = open article - - [ ] Add keyboard shortcut help modal (?) - - **Modals & Dialogs:** - - [ ] Create "Add Link" modal - - [ ] Create "Edit Item" modal (title, description) - - [ ] Create "Upload File" modal with drag-and-drop - - [ ] Create confirmation dialogs for destructive actions - - [ ] Add loading overlays for long operations - - **Polish & UX:** - - [ ] Add proper empty states for each folder - - [ ] Add skeleton loaders for initial page load - - [ ] Add infinite scroll with loading indicators - - [ ] Add error boundaries and error states - - [ ] Add toast notifications for all actions - - [ ] Add optimistic UI updates - - [ ] Implement pinned searches feature - - [ ] Add processing items auto-refresh - - [ ] Add drag-and-drop file upload to page - - **Performance:** - - [ ] Implement virtual scrolling for large lists - - [ ] Optimize re-renders with React.memo - - [ ] Add request deduplication - - [ ] Implement proper cache invalidation - -- **Acceptance Criteria**: - - [ ] All legacy library features work in new UI - - [ ] Keyboard shortcuts match legacy system - - [ ] Layout switching works smoothly - - [ ] Performance acceptable (FCP <1s, smooth scrolling) - - [ ] Mobile experience is fully functional - - [ ] All modals and dialogs work correctly - - [ ] Error states provide helpful guidance - - [ ] Loading states indicate progress clearly - - [ ] Visual design matches or improves on legacy - - [ ] User testing validates feature completeness -- **Dependencies**: ARC-005, ARC-006, ARC-007, ARC-008. -- **Effort Estimate**: 5-7 days. -- **Status**: Pending prior ARCs completion - -## ARC-010A Minimal Reader ✅ **COMPLETED** - -- **Problem/Objective**: Enable users to read saved articles with basic display functionality before implementing advanced features. -- **Approach**: Create simple, clean reader page that displays extracted content without highlights/annotations. This unblocks content extraction testing and delivers core reading value quickly. Tasks: - - **Backend (NestJS):** - - [x] Add `content` field to LibraryItem GraphQL type (HTML content) - - [x] ~~Add `textContent` field~~ - Not needed (readable_content serves this purpose) - - [x] Ensure `libraryItem(id)` query returns content fields - - [x] Add basic content sanitization (DOMPurify on frontend) - - **Frontend (web-vite):** - - [x] Create `/reader/:id` route with ReaderPage component - - [x] Implement reader layout: - - [x] Article header (title, author, date, original URL) - - [x] Content display area with clean typography - - [x] Back to library button - - [ ] ~~Share/actions menu~~ - Deferred to ARC-010 - - [x] Add loading state while fetching content - - [x] Add error state for missing/failed content - - [x] Handle CONTENT_NOT_FETCHED state gracefully (show message) - - [x] Responsive design (mobile + desktop) - - [x] Basic reading styles (font size, line height, max-width) - - [x] Update LibraryPage to link to reader (click title/Read button) - -- **Acceptance Criteria**: ✅ **ALL CORE CRITERIA MET** - - [x] Users can click an item and navigate to reader page - - [x] Content displays with clean, readable typography - - [x] Works on mobile and desktop devices - - [x] Gracefully handles items without content yet - - [x] Back navigation returns to library - - [x] Reader route is protected (requires auth) -- **Dependencies**: None (works with current backend, enhanced by ARC-013) -- **Effort Estimate**: 1-2 days -- **Actual Time**: ~2 hours -- **Status**: ✅ Completed (2025-10-05) -- **Note**: This is a minimal viable reader. Advanced features (highlights, notes, progress) come in ARC-010. -- **Completion Analysis**: See `/docs/architecture/ARC-010A-COMPLETION-ANALYSIS.md` - -## ARC-010 Reading Progress & Highlights - -- **Problem/Objective**: Implement reading progress tracking and highlights/annotations system. -- **Approach**: Build on ARC-010A minimal reader by adding advanced reading features. Tasks: - - **Backend (NestJS):** - - [ ] Create HighlightEntity mapping to existing `highlights` table - - [ ] Create HighlightModule with service and resolver - - [ ] Add GraphQL queries: - - [ ] `highlights(itemId: String!): [Highlight!]!` - get all highlights for item - - [ ] `highlight(id: String!): Highlight` - get single highlight - - [ ] Add GraphQL mutations: - - [ ] `createHighlight(itemId: String!, text: String!, position: Int!, note: String): Highlight!` - - [ ] `updateHighlight(id: String!, text: String, note: String): Highlight!` - - [ ] `deleteHighlight(id: String!): DeleteResult!` - - [ ] `updateReadingProgress(itemId: String!, progress: ReadingProgressInput!): LibraryItem!` - - [ ] Update LibraryItemEntity to include highlights relation - - [ ] Add reading progress sync logic - - [ ] Create E2E tests for highlights and progress tracking - - **Frontend (web-vite):** - - [ ] Create ArticleReader component/page - - [ ] Implement highlight selection UI - - [ ] Add highlight annotation sidebar - - [ ] Implement reading progress tracker - - [ ] Add "Notebook" view showing all highlights - - [ ] Create highlight management hooks: - - [ ] `useHighlights(itemId)` - fetch highlights - - [ ] `useCreateHighlight()` - create highlight - - [ ] `useUpdateHighlight()` - update highlight - - [ ] `useDeleteHighlight()` - delete highlight - - [ ] Sync reading progress automatically - - [ ] Add highlight search and filtering - - [ ] Export highlights functionality - -- **Acceptance Criteria**: - - [ ] Users can create highlights while reading - - [ ] Highlights persist and sync across devices - - [ ] Reading progress tracked automatically - - [ ] Notebook view shows all highlights with context - - [ ] Highlights can have notes/annotations - - [ ] Highlight colors/styles supported - - [ ] Reading position restored on return to article - - [ ] Export highlights to markdown/JSON - - [ ] Highlight search works correctly -- **Dependencies**: ARC-010A (minimal reader as foundation), ARC-005, ARC-009. -- **Effort Estimate**: 3-4 days. -- **Status**: Pending ARC-010A and prior ARCs completion - -## ARC-011 Add Link & Content Ingestion ✅ **COMPLETED** - -- **Problem/Objective**: Implement the core "save to library" functionality with URL parsing and content extraction. -- **Approach**: Build the link saving pipeline. Content extraction deferred to ARC-012 (queue) and ARC-013 (readability). Tasks: - - **Backend (NestJS):** - - [x] Add GraphQL mutation: - - [x] `saveUrl(input: SaveUrlInput!): LibraryItem!` - - [x] Create SaveUrlInput type with url and folder fields - - [x] Add validation (URL format using @IsUrl, duplicate detection) - - [x] Generate unique slugs from URLs with timestamp - - [x] Set initial state to CONTENT_NOT_FETCHED (extraction deferred to ARC-012) - - [x] Create E2E tests for save URL flow (17 tests, all passing) - - [ ] ~~Handle different content types (article, PDF, etc.)~~ → **Deferred to ARC-013** - - [ ] ~~Add rate limiting for URL saving~~ → **Can be added anytime** - - [ ] ~~Implement basic content extraction~~ → **Deferred to ARC-012 (queue) and ARC-013 (readability)** - - **Frontend (web-vite):** - - [x] Add useSaveUrl hook to graphql-client - - [x] Implement "Add Link" modal with URL input - - [x] Add folder selection to save modal (inbox/archive) - - [x] Add content type tabs (Link, PDF, RSS) with "coming soon" for PDF/RSS - - [x] Show save progress indicator (loading spinner) - - [x] Handle save errors gracefully (validation + error messages) - - [x] Add URL validation in UI (client-side validation) - - [x] Show newly saved item in library immediately (refetch after save) - - [x] Integrate modal with "+ Add Article" buttons - - [ ] ~~Add browser extension integration points~~ → **Future enhancement** - - [ ] ~~Folder selection persists preference~~ → **Future UX enhancement** - -- **Acceptance Criteria**: ✅ **ALL CORE CRITERIA MET** - - [x] Users can save URLs to their library - - [x] Duplicate URLs detected and handled (ConflictException) - - [x] Save errors provide helpful messages (validation errors shown in UI) - - [x] Saved items appear in library immediately (refetch on success) - - [x] Folder selection works (inbox/archive dropdown) - - [x] All 17 E2E tests passing (including validation and error cases) - - [ ] ~~Basic content extraction works for common sites~~ → **Deferred to ARC-012/ARC-013** - - [ ] ~~Rate limiting prevents abuse~~ → **Can be added anytime** - - [ ] ~~Browser extension can save URLs~~ → **Future enhancement** -- **Dependencies**: ARC-005. -- **Effort Estimate**: 2-3 days. -- **Status**: ✅ **Completed** (actual: 1 day for MVP focusing on URL saving, content extraction deferred) - -## ARC-012 Queue Integration & Background Processing ⭐ **80% COMPLETE** - -- **Problem/Objective**: Integrate BullMQ queues for robust background processing of content extraction and other async tasks in single-service architecture. -- **Architectural Decisions** (see `/docs/architecture/ARC-012-QUEUE-ARCHITECTURE-DESIGN.md` and `ARC-012-EVENT-AND-REDIS-ANALYSIS.md`): - - **Event Pattern**: Node.js EventEmitter (not full EventManager) for simplicity ✅ - - **Redis Architecture**: Sentinel (master-slave with HA) for BullMQ compatibility ✅ - - **Worker Strategy**: In-process workers (not separate microservice) ✅ - - **Scaling**: Horizontal pod autoscaling with shared Redis ✅ - - **Configuration**: Constants file (no magic strings) ✅ - -- **Approach**: Establish queue infrastructure with event-driven processing. Implementation in 5 phases: - - ### **Phase 1: Infrastructure Setup** ✅ **COMPLETE** - - [x] Install dependencies: `@nestjs/bullmq`, `bullmq`, `ioredis` - - [x] Create `queue.constants.ts` with all queue names, job types, priorities - - [x] Create `QueueModule` with Redis Sentinel configuration - - [x] Set up shared Redis connection (cache + queue) - - [x] Create health check endpoints for queue/Redis (QueueHealthIndicator) - - [x] Add graceful shutdown handling (OnModuleDestroy) - - [x] Fix Redis maxRetriesPerRequest (null for BullMQ blocking operations) - - [x] Fix Jest ESM configuration for bullmq/msgpackr - - [x] **Testing**: Unit tests for QueueModule, health checks (13/13 passing) - - [ ] Add Prometheus metrics integration → **DEFERRED to Phase 5** - - ### **Phase 2: Event System** ✅ **COMPLETE** - - [x] Create `EventBusService` extending EventEmitter - - [x] Define event types in `events.constants.ts` - - [x] Create event data interfaces (type-safe) - - [x] Wire event handlers to queue operations - - [x] Add event emission logging - - [x] **Testing**: Unit tests for EventBusService (13/13 passing) - - ### **Phase 3: Content Processing Queue** ✅ **INFRASTRUCTURE COMPLETE** ⏳ **CONTENT STUB** - - [x] Create `ContentProcessorService` with `@Processor()` decorator - - [x] Implement `@Process('fetch-content')` job handler with **STUB** content fetching - - [x] Add job priority configuration (HIGH, NORMAL, LOW) - - [x] Implement retry logic with exponential backoff (3 attempts) - - [x] Add job deduplication by libraryItemId as jobId - - [x] Add progress tracking (updateProgress at 10%, 20%, 70%, 90%, 100%) - - [x] **Testing**: Unit tests for processor (15/15 passing) - - [ ] **TODO**: Implement real content fetching (readability extraction) → **ARC-013** - - [ ] Configure rate limiting per user → **DEFERRED** (can add later) - - ### **Phase 4: Library Integration** ✅ **COMPLETE** - - [x] Update `saveUrl` mutation to emit ContentSaveRequested event - - [x] Update library item state: PROCESSING → SUCCEEDED/FAILED - - [x] Inject EventBusService into LibraryService - - [x] Add source tracking to SaveUrlInput - - [x] **Testing**: E2E test for full saveUrl → queue → process flow (17/17 passing) - - [ ] Add job status polling endpoint for frontend → **NOT NEEDED** (can query item state) - - [ ] Implement job cancellation endpoint → **DEFERRED** (future enhancement) - - [ ] Add user notification on processing completion/failure → **Event system ready**, UI integration deferred - - ### **Phase 5: Monitoring & Optimization** ⏸️ **DEFERRED** - - [ ] Add BullMQ Board UI endpoint (`/admin/queues`) - - [ ] Implement queue depth metrics (Prometheus) - - [ ] Add job latency histograms - - [ ] Create AlertManager rules for queue backlog - - [ ] Add worker concurrency auto-adjustment - - [ ] Performance profiling and optimization - - [ ] **Testing**: Load test with 100+ concurrent jobs - - ### **Configuration Management (No Magic Strings)** - ```typescript - // queue.constants.ts - export const QUEUE_NAMES = { - CONTENT_PROCESSING: 'content-processing', - NOTIFICATIONS: 'notifications', - POST_PROCESSING: 'post-processing', - } as const - - export const JOB_TYPES = { - FETCH_CONTENT: 'fetch-content', - SEND_NOTIFICATION: 'send-notification', - } as const - - export const JOB_PRIORITY = { - CRITICAL: 1, - HIGH: 5, - NORMAL: 10, - LOW: 20, - } as const - ``` - -- **Testing Requirements**: - - [ ] **Unit Tests**: - - [ ] QueueModule configuration and dependency injection - - [ ] EventBusService event emission and handling - - [ ] ContentProcessorService job processing logic - - [ ] Redis connection management and failover - - [ ] Job priority and deduplication logic - - [ ] **Integration Tests**: - - [ ] Queue → Worker communication - - [ ] Event → Queue → Processing flow - - [ ] Redis Sentinel failover scenarios - - [ ] Graceful shutdown with in-flight jobs - - [ ] **E2E Tests** (see `packages/api-nest/test/queue.e2e-spec.ts`): - - [ ] Complete saveUrl → queue → process → update flow - - [ ] Job retry on failure (3 attempts) - - [ ] Job cancellation by user - - [ ] Rate limiting enforcement - - [ ] Concurrent job processing (50+ jobs) - - [ ] Queue backlog handling - - [ ] **Load Tests**: - - [ ] 100 jobs/minute sustained load - - [ ] Burst traffic (500 jobs in 1 minute) - - [ ] Multiple replica scaling (2x, 3x, 5x) - -- **Acceptance Criteria**: - - [x] API response time <200ms (unchanged from current) ✅ - - [x] Jobs queued and processed reliably (no data loss) ✅ - - [x] Failed jobs retry with exponential backoff (3 attempts) ✅ - - [x] Graceful shutdown completes in-flight jobs (<30s) ✅ - - [x] All tests passing (unit, integration, E2E) - **87 unit + 116 E2E passing** ✅ - - [ ] Queue monitoring UI shows accurate metrics → **Phase 5** - - [ ] Horizontal scaling works (2x replicas = ~2x throughput) → **Future testing** - - [ ] Redis Sentinel failover recovers in <10 seconds → **Future testing** - - [ ] Prometheus metrics exported and alerting configured → **Phase 5** - - [ ] Job throughput: 50+ jobs/hour on single instance → **Needs real content fetching** - - [ ] Real content extraction working → **ARC-013** - -- **Dependencies**: ARC-011 (completed). -- **Effort Estimate**: 3 days (originally estimated). -- **Actual Time**: ~2 days for infrastructure (Phases 1-4), Phase 5 deferred -- **Status**: ✅ **80% Complete** - Infrastructure ready, content fetching stub needs ARC-013 -- **Architecture Docs**: - - Detailed design: `/docs/architecture/ARC-012-QUEUE-ARCHITECTURE-DESIGN.md` - - Event & Redis analysis: `/docs/architecture/ARC-012-EVENT-AND-REDIS-ANALYSIS.md` -- **Key Achievements**: - - ✅ BullMQ integrated with proper Redis configuration - - ✅ Event-driven architecture with EventBusService - - ✅ Worker pattern established with ContentProcessorService - - ✅ Full test coverage (42 queue tests, 17 E2E tests) - - ✅ Clean logger mocking using NestJS .setLogger() pattern - - ✅ Jest ESM configuration fixed for bullmq dependencies - -## ARC-013 Advanced Content Processing - -- **Problem/Objective**: Implement comprehensive content processing including readability extraction, PDF handling, and image optimization. -- **Approach**: Migrate content processing pipeline to NestJS with full feature parity. Tasks: - - [ ] Create ContentProcessorModule with job handlers - - [ ] Integrate readability extraction (readabilityjs package) - - [ ] Implement PDF processing using pdf-handler logic - - [ ] Add EPUB processing support - - [ ] Implement image optimization and thumbnail generation - - [ ] Add content sanitization and security validation - - [ ] Implement retry mechanisms for failed processing - - [ ] Add processing status tracking and progress reporting - - [ ] Handle different content types (web, PDF, EPUB, RSS) - - [ ] Implement error classification and user notifications -- **Acceptance Criteria**: - - [ ] Articles automatically processed when saved - - [ ] Content extraction works correctly for web articles - - [ ] PDF processing maintains existing functionality - - [ ] EPUB files processed correctly - - [ ] Images optimized and thumbnails generated - - [ ] Processing errors handled gracefully with retries - - [ ] Content sanitization prevents XSS and security issues - - [ ] Processing status accurately tracked and reported - - [ ] Users notified of processing failures -- **Dependencies**: ARC-012. -- **Effort Estimate**: 4-5 days. -- **Status**: Pending ARC-012 completion - -## ARC-014 Remaining Feature Migration - -- **Problem/Objective**: Migrate remaining Express features (feeds, integrations, admin) to NestJS. -- **Approach**: Systematically migrate remaining endpoints with feature flag support. Tasks: - - [ ] Create FeedsModule for RSS/Atom feed subscriptions - - [ ] Create IntegrationModule for third-party integrations: - - [ ] Readwise integration - - [ ] Notion integration - - [ ] Webhook endpoints - - [ ] Create DigestModule for email digests - - [ ] Migrate admin utilities and management endpoints - - [ ] Implement feature flags for gradual rollout - - [ ] Add monitoring and logging for migration tracking - - [ ] Create rollback procedures for each feature - - [ ] Update frontend to use NestJS endpoints -- **Acceptance Criteria**: - - [ ] All critical endpoints migrated with identical functionality - - [ ] Frontend successfully uses NestJS endpoints - - [ ] No functionality regression detected in tests - - [ ] Feature flags allow selective rollout and rollback - - [ ] Admin tools work correctly with new backend - - [ ] Integration webhooks maintain compatibility - - [ ] RSS feeds work correctly -- **Dependencies**: ARC-013. -- **Effort Estimate**: 5-7 days. -- **Status**: Pending ARC-013 completion - -## ARC-015 Service Consolidation & Cleanup - -- **Problem/Objective**: Decommission old services and consolidate to single NestJS API. -- **Approach**: Complete migration by removing legacy services and consolidating infrastructure. Tasks: - - [ ] Update docker-compose.yml to remove Express API service - - [ ] Remove separate queue-processor and content-handler containers - - [ ] Update NestJS API to run on port 4000 (production port) - - [ ] Update deployment scripts and CI/CD pipelines - - [ ] Clean up old configuration files and environment variables - - [ ] Update documentation and self-hosting guides - - [ ] Remove legacy code from repository - - [ ] Perform final validation and testing - - [ ] Update monitoring and alerting configurations - - [ ] Create rollback plan if needed -- **Acceptance Criteria**: - - [ ] Single NestJS API handles all functionality on port 4000 - - [ ] Resource usage reduced by 33% (memory) and 75% (services) - - [ ] Deployment process simplified with single service - - [ ] All automated tests pass with new configuration - - [ ] Self-hosting documentation updated and validated - - [ ] No legacy Express code remains in production builds - - [ ] Monitoring and logging work correctly - - [ ] Performance metrics meet or exceed baseline -- **Dependencies**: ARC-014. -- **Effort Estimate**: 2-3 days. -- **Status**: Pending ARC-014 completion +## 🎯 Current Status Summary + +### ✅ **COMPLETED** (16 Major ARCs) +All completed items moved to `unified-migration-backlog-complete.md`. Key achievements: +- **Backend Foundation**: NestJS setup, auth, GraphQL, database integration +- **Core Features**: Library CRUD, search, labels, bulk operations, highlights, reading progress +- **Infrastructure**: Queue system (80%), repository pattern, testing infrastructure +- **Frontend**: Vite migration, library UI, reader page, labels management +- **Technical Debt**: Constants, repository pattern, Testcontainers + factories + +**Test Coverage**: 261 tests (174 E2E + 87 unit), 100% passing + +### 🚨 **CRITICAL ISSUE IDENTIFIED** +- **Test Database Connection**: Tests writing to live database instead of testcontainer +- **Root Cause**: ENV variable mismatch (`TEST_DB_*` vs `TEST_DATABASE_*`) +- **Priority**: Must fix immediately (high risk) +- **Effort**: 2-4 hours --- -## Migration Progress Summary +## 🎯 Active Backlog (Priority Order) -### **Phase 1: Foundation** ✅ Complete -- **ARC-001**: NestJS Package Setup ✅ -- **ARC-002**: Health Checks & Observability ✅ -- **ARC-003**: Authentication Module ✅ -- **ARC-003B**: Database & Entity Integration ✅ -- **ARC-004**: GraphQL Module Setup ✅ +### 🔴 CRITICAL PRIORITY -### **Phase 2: Library Core** ✅ Complete -- **ARC-004B**: Vite Migration (Partial) 🔄 Foundation complete -- **ARC-005**: Library Core Mutations ✅ Complete -- **ARC-006**: Advanced Search & Filtering ✅ Complete -- **ARC-006B**: Performance & UX Optimizations ✅ Complete -- **ARC-007**: Bulk Operations & Multi-select ✅ Complete -- **ARC-008**: Labels System ✅ Complete - -### **Phase 3: Frontend Feature Parity** -- **ARC-009**: Frontend Library Feature Parity -- **ARC-010**: Reading Progress & Highlights - -### **Phase 4: Content Ingestion** -- **ARC-011**: Add Link & Content Ingestion -- **ARC-012**: Queue Integration & Background Processing -- **ARC-013**: Advanced Content Processing - -### **Phase 5: Completion** -- **ARC-014**: Remaining Feature Migration -- **ARC-015**: Service Consolidation & Cleanup +#### FIX-001: Test Database Isolation ⚠️ **CRITICAL** +- **Problem**: E2E tests currently write to live development database instead of testcontainer +- **Root Cause**: `test.config.ts` reads `TEST_DATABASE_*` but global-setup sets `TEST_DB_*` +- **Impact**: Data pollution in development database, potential data loss +- **Fix Required**: + - Update `src/config/test.config.ts` to prioritize `TEST_DB_*` variables + - Add validation to block production database names (`omnivore`, `omnivore_prod`) + - Add safety checks with clear error messages + - Verify all tests use testcontainer +- **Acceptance Criteria**: + - All tests run against ephemeral testcontainer + - Production database names blocked with error + - No data written to development/production databases + - All 261 tests still passing +- **Dependencies**: None (critical fix) +- **Effort Estimate**: 2-4 hours +- **Status**: ⚠️ **IDENTIFIED - NEEDS IMMEDIATE FIX** +- **Files**: + - `packages/api-nest/src/config/test.config.ts` (primary fix) + - `packages/api-nest/test/setup/global-setup.ts` (verify ENV vars) --- -**Total Tickets**: 18 ARCs (added ARC-006B for performance, ARC-007B for technical debt, ARC-010A for minimal reader) -- **Completed**: 11 ARCs (ARC-001 through ARC-008, ARC-010A, ARC-011, plus partial ARC-004B) -- **In Progress**: 1 ARC (ARC-004B foundation complete, remaining in ARC-009) -- **Ready to Start**: 2 ARCs (ARC-012, ARC-007B) -- **Remaining**: 6 ARCs (ARC-007B, ARC-009, ARC-010, ARC-012 through ARC-015) +### 🔴 HIGH PRIORITY -**Effort Estimates:** -- **Completed**: ~17 days estimated (actual: ~8-9 days due to efficiency gains) -- **Tech Debt (ARC-007B)**: 1-2 days (optional, can be deferred) -- **Frontend Parity (ARC-009 to ARC-010)**: 8-11 days -- **Content Ingestion (ARC-011 to ARC-013)**: 9-11 days -- **Completion (ARC-014 to ARC-015)**: 7-10 days -- **Total Remaining**: 25-34 days (5-7 weeks) excluding optional ARC-007B +#### ARC-013: Content Extraction & Processing ⭐ **NEXT UP** +- **Problem/Objective**: Complete the save-to-read flow with actual content extraction (currently stubbed) +- **Impact**: Unblocks core user workflow (save article → read article) +- **Current State**: + - Queue infrastructure ready (ARC-012 at 80%) + - ContentProcessorService has stub implementation + - Event system in place + - SaveUrl mutation working +- **Approach**: Implement real content extraction using Readability.js and related tools -**Next Milestone**: ARC-009 Frontend Library Feature Parity to complete user-facing library experience +**Tasks**: -**Recent Accomplishments**: -- ✅ **ARC-010A Minimal Reader completed** - Basic article reader with clean typography - - Created ReaderPage component with responsive design - - Added content field to GraphQL schema (readable_content mapping) - - Implemented DOMPurify HTML sanitization - - Graceful handling of CONTENT_NOT_FETCHED state - - Title click navigation to reader from library - - Loading, error, and empty states -- ✅ **ARC-011 Add Link & Content Ingestion completed** - Save URLs with validation - - AddLinkModal component with folder selection - - SaveUrl mutation with duplicate detection - - 17/17 E2E tests passing - - Content extraction deferred to ARC-012/013 -- ✅ **ARC-008 Labels System completed** - Full label management with filtering - - Created Labels management page with CRUD operations - - Implemented LabelPicker component with dropdown UI - - Added label filtering to library search - - Fixed label persistence by syncing both entity_labels and label_names columns - - Migration 0191 for labels.updated_at default value -- ✅ Bulk operations with transaction support (44/44 E2E tests passing) -- ✅ Multi-select UI with checkboxes and bulk action bar -- ✅ Migration 0190 adds 8 strategic indexes (26x faster folder filters, 8x faster search) -- ✅ Simplified logging format (one-line, color-coded, readable in terminal) +**Phase 1: Web Article Extraction** (Days 1-2) +- [ ] Install dependencies: + - [ ] `@mozilla/readability` - Content extraction + - [ ] `jsdom` - DOM parsing for Node.js + - [ ] `dompurify` with jsdom - HTML sanitization + - [ ] `turndown` - HTML to Markdown conversion (for plain text) +- [ ] Implement ContentFetcherService: + - [ ] `fetchUrl(url: string): Promise` - HTTP fetch with headers + - [ ] Handle redirects and SSL certificates + - [ ] Set proper User-Agent and timeouts + - [ ] Rate limiting per domain +- [ ] Implement ReadabilityService: + - [ ] `extractArticle(html: string, url: string): Promise
` + - [ ] Clean and sanitize HTML + - [ ] Extract title, author, published date + - [ ] Extract main content with images + - [ ] Generate text excerpt/preview +- [ ] Update ContentProcessorService: + - [ ] Replace stub with real extraction logic + - [ ] Call ContentFetcherService → ReadabilityService + - [ ] Update LibraryItem with extracted content + - [ ] Set state to SUCCEEDED or FAILED + - [ ] Handle extraction errors gracefully -**Identified Technical Debt** (to address in future refactoring): -- 🔧 Magic strings throughout codebase (folder names, config keys, states) → Need referential constants -- 🔧 Direct DataSource usage in services → Should use Repository pattern exclusively -- 🔧 Inconsistent database operation patterns → Consolidate into custom repositories +**Phase 2: Image Processing** (Day 3) +- [ ] Implement ImageProxyService: + - [ ] Download and cache images + - [ ] Resize/optimize images + - [ ] Generate thumbnails + - [ ] Return CDN/proxy URLs +- [ ] Update content HTML with proxied image URLs +- [ ] Handle image extraction failures gracefully -**Known UI Bugs** (to be addressed in ARC-009): -- 🐛 Label dropdown flickers when opened over library item cards (z-index/overlay issue) -- 🐛 Punycode deprecation warnings from transitive dependencies (eslint, typeorm) - cosmetic, non-blocking +**Phase 3: Content Enhancements** (Day 4) +- [ ] Add metadata extraction: + - [ ] OpenGraph tags (og:title, og:description, og:image) + - [ ] Twitter Card metadata + - [ ] JSON-LD structured data + - [ ] Favicon extraction +- [ ] Implement content hash generation (for duplicate detection) +- [ ] Add word count calculation +- [ ] Add reading time estimation + +**Phase 4: Testing & Polish** (Day 5) +- [ ] Create E2E tests: + - [ ] Save URL → extract content → verify in reader + - [ ] Handle extraction failures + - [ ] Handle redirects and SSL issues + - [ ] Verify image proxying + - [ ] Test metadata extraction +- [ ] Add integration tests for each service +- [ ] Performance testing (extraction time targets) +- [ ] Error handling and user feedback + +**Deferred to ARC-014**: +- [ ] PDF content extraction (pdf-parse) +- [ ] RSS feed parsing +- [ ] YouTube video transcripts +- [ ] Twitter thread unrolling + +**Acceptance Criteria**: +- [ ] Save URL extracts article title, author, content, images +- [ ] Extracted content displays correctly in reader +- [ ] Images load through proxy/cache +- [ ] Failed extractions show helpful error messages +- [ ] Content hash prevents duplicates +- [ ] E2E test: Save article → read in reader (full flow) +- [ ] Extraction completes in <10 seconds for typical articles +- [ ] All existing tests still pass (261 tests) + +**Dependencies**: +- ARC-011 (completed - Save URL mutation) +- ARC-012 (80% - Queue infrastructure ready) + +**Effort Estimate**: 4-5 days + +**Status**: ⭐ **READY TO START** (highest priority after FIX-001) + +**Priority**: 🔴 **CRITICAL** - Completes core save-to-read workflow --- -## Implementation Notes +### 🟡 MEDIUM PRIORITY -### **Stable State Philosophy** +#### ARC-009: Frontend Library Feature Parity ⏳ **95% COMPLETE** +- **Problem/Objective**: Achieve full feature parity with legacy library UI +- **Current State**: Core features complete, polish needed +- **Approach**: Complete remaining UI features and polish -Each ARC ticket is designed to reach a **stable, testable, deployable state** before moving to the next. This approach ensures: -- No half-completed features in production -- Easy rollback points if issues arise -- Continuous value delivery to users -- Reduced integration complexity +**Remaining Tasks**: +- [ ] Advanced filters UI: + - [ ] Date range picker (saved date, published date) + - [ ] Content type filter (article, PDF, etc.) + - [ ] Read status filter + - [ ] Has highlights filter +- [ ] Library view modes: + - [ ] Grid view (current default) + - [ ] List view (compact) + - [ ] Magazine view (large cards) + - [ ] View preference persistence +- [ ] Sort options polish: + - [ ] Add "Reading Progress" sort + - [ ] Add "Recently Added" sort + - [ ] Remember last sort preference +- [ ] Keyboard shortcuts: + - [ ] `j/k` - Navigate items + - [ ] `a` - Archive item + - [ ] `e` - Edit labels + - [ ] `r` - Read/open item + - [ ] `x` - Select item + - [ ] `Shift+X` - Select all + - [ ] `/` - Focus search +- [ ] Import/export (deferred to separate ARC): + - [ ] Export library to JSON/CSV + - [ ] Import from Pocket/Instapaper -### **Dependency Flow & Stable States** +**Acceptance Criteria**: +- [ ] All filter combinations work correctly +- [ ] View modes toggle and persist preference +- [ ] Keyboard shortcuts functional and documented +- [ ] Advanced filters performance acceptable +- [ ] UI matches design system +**Dependencies**: None (frontend polish) + +**Effort Estimate**: 2-3 days + +**Status**: 95% complete, polish remaining + +**Priority**: 🟡 **MEDIUM** - UX improvement, not blocking + +--- + +#### ARC-010B: Reading Progress & Highlights (Frontend Polish) +- **Problem/Objective**: Polish highlight and reading progress UI/UX +- **Current State**: Backend complete (ARC-010), basic frontend working +- **Approach**: Enhance UI components for better user experience + +**Tasks**: +- [ ] Highlight creation UI: + - [ ] Improve text selection UX + - [ ] Color picker for highlights + - [ ] Quick annotation input + - [ ] Highlight preview before save +- [ ] Highlight sidebar polish: + - [ ] Group highlights by color + - [ ] Sort options (position, date, color) + - [ ] Search/filter highlights + - [ ] Jump to highlight in text +- [ ] Reading progress UI: + - [ ] Visual progress bar in reader + - [ ] Percentage complete indicator + - [ ] Resume reading from last position + - [ ] Scroll position persistence +- [ ] Notebook improvements: + - [ ] Markdown preview + - [ ] Rich text editor option + - [ ] Autosave indicator + - [ ] Version history (future) + +**Acceptance Criteria**: +- [ ] Highlighting feels smooth and intuitive +- [ ] Progress bar accurately reflects reading position +- [ ] Notebook autosaves without data loss +- [ ] All highlight colors work correctly +- [ ] Performance acceptable with 100+ highlights + +**Dependencies**: ARC-010 (completed - backend) + +**Effort Estimate**: 2-3 days + +**Status**: Backend complete, frontend basic working + +**Priority**: 🟡 **MEDIUM** - Polish, core functionality working + +--- + +### 🟢 LOW PRIORITY (Future Work) + +#### ARC-012 Phase 5: Queue Monitoring & Optimization +- **Problem/Objective**: Add monitoring UI and performance optimizations +- **Current State**: Infrastructure complete (80%), monitoring deferred +- **Approach**: Incremental additions + +**Remaining Tasks** (Phase 5): +- [ ] Add BullMQ Board UI endpoint (`/admin/queues`) +- [ ] Implement Prometheus metrics export: + - [ ] Queue depth by queue + - [ ] Job latency histograms + - [ ] Success/failure rates + - [ ] Worker utilization +- [ ] Create AlertManager rules: + - [ ] Queue backlog threshold + - [ ] Job failure rate spike + - [ ] Worker downtime +- [ ] Add worker concurrency auto-adjustment +- [ ] Performance profiling and optimization +- [ ] Load testing: 100+ concurrent jobs + +**Acceptance Criteria**: +- [ ] BullMQ Board accessible to admins +- [ ] Metrics exported to Prometheus +- [ ] Alerts fire correctly +- [ ] Load test passes with target throughput + +**Dependencies**: ARC-012 Phases 1-4 (complete), ARC-013 (real content processing) + +**Effort Estimate**: 1-2 days + +**Status**: Infrastructure complete, monitoring deferred + +**Priority**: 🟢 **LOW** - Can add incrementally + +--- + +#### ARC-014: Additional Content Types +- **Problem/Objective**: Support PDF, RSS, video content types +- **Approach**: Extend content processing pipeline + +**Tasks**: +- [ ] PDF extraction: + - [ ] Install `pdf-parse` or `pdfjs-dist` + - [ ] Extract text from PDFs + - [ ] Generate PDF thumbnails + - [ ] Handle scanned PDFs (OCR) +- [ ] RSS feed parsing: + - [ ] Install `rss-parser` + - [ ] Subscribe to RSS feeds + - [ ] Auto-import new articles + - [ ] Feed management UI +- [ ] Video transcripts: + - [ ] YouTube transcript API + - [ ] Video metadata extraction + - [ ] Thumbnail extraction +- [ ] Twitter threads: + - [ ] Thread unrolling + - [ ] Author attribution + - [ ] Media preservation + +**Acceptance Criteria**: +- [ ] PDFs extract and display correctly +- [ ] RSS feeds auto-import articles +- [ ] Video content saves with metadata +- [ ] Twitter threads unroll properly + +**Dependencies**: ARC-013 (web article extraction) + +**Effort Estimate**: 5-7 days + +**Status**: Not started + +**Priority**: 🟢 **LOW** - After core web articles working + +--- + +#### TD-006 Phase 3-5: Test Conversion & Documentation +- **Problem/Objective**: Convert E2E tests to unit tests (test pyramid inversion fix) +- **Current State**: Infrastructure ready (Phases 1-2 complete) +- **Approach**: Incremental conversion as services are refactored + +**Remaining Phases**: +- [ ] **Phase 3**: Convert 80-90 E2E tests to unit tests + - [ ] Identify tests that should be unit tests + - [ ] Convert using repository mocks + - [ ] Update factories for unit test usage + - [ ] Maintain E2E tests for critical flows +- [ ] **Phase 4**: Update documentation + - [ ] Create `docs/TESTING.md` guide + - [ ] Document factory patterns + - [ ] Add mocking examples + - [ ] Troubleshooting guide +- [ ] **Phase 5**: Deprecate old utilities + - [ ] Remove TD-001 seed scripts + - [ ] Archive manual test DB setup docs + - [ ] Update CI/CD pipelines + +**Target Test Distribution**: +- Current: 87 unit (33%) / 174 E2E (67%) = **Inverted pyramid** +- Target: 220 unit (85%) / 25 integration (10%) / 15 E2E (5%) = **Proper pyramid** + +**Acceptance Criteria**: +- [ ] Test pyramid corrected (85% unit, 10% integration, 5% E2E) +- [ ] All tests still passing +- [ ] Documentation complete +- [ ] CI/CD pipeline updated + +**Dependencies**: TD-003, TD-006 Phases 1-2 (both complete) + +**Effort Estimate**: 5-7 days (can be done incrementally) + +**Status**: Infrastructure ready, conversion deferred + +**Priority**: 🟢 **LOW** - Technical debt, not blocking features + +--- + +## 📋 Backlog Management Notes + +### Completed Items +All completed ARCs (001-008, 010A, 010, 011, 012 partial) and TDs (003, 004, 006 Phases 1-2) have been moved to `unified-migration-backlog-complete.md` for historical reference. + +### Next Review +- **When**: After ARC-013 completion +- **Focus**: Evaluate ARC-014 priority, frontend polish items +- **Metrics**: Test coverage, performance benchmarks, user feedback + +### Dependencies ``` -Foundation (✅ Complete) -├─ ARC-001: NestJS Setup -├─ ARC-002: Health Checks -├─ ARC-003: Authentication -├─ ARC-003B: Database Entities -└─ ARC-004: GraphQL Module - └─ STABLE STATE: Read-only library listing works - -Library Core (🔄 Current Focus) -├─ ARC-004B: Vite Frontend (ongoing) -├─ ARC-005: Core Mutations ⭐ NEXT -│ └─ STABLE STATE: Archive, delete, mark-read work -├─ ARC-006: Search & Filtering -│ └─ STABLE STATE: Advanced search matches legacy -├─ ARC-007: Bulk Operations -│ └─ STABLE STATE: Multi-select and bulk actions work -└─ ARC-008: Labels System - └─ STABLE STATE: Full label management without content processing - -Frontend Parity -├─ ARC-009: UI Feature Parity -│ └─ STABLE STATE: Library UI matches legacy feature-for-feature -└─ ARC-010: Reading & Highlights - └─ STABLE STATE: Reading experience complete - -Content Ingestion (Can use Express APIs until ready) -├─ ARC-011: Basic URL Saving -│ └─ STABLE STATE: Can save URLs with basic extraction -├─ ARC-012: Queue Integration -│ └─ STABLE STATE: Background processing via queues -└─ ARC-013: Advanced Processing - └─ STABLE STATE: Full content processing parity - -Completion -├─ ARC-014: Remaining Features -│ └─ STABLE STATE: All features migrated -└─ ARC-015: Service Consolidation - └─ STABLE STATE: Single unified service +FIX-001 → [No dependencies, critical fix] +ARC-013 → Requires: ARC-011 ✅, ARC-012 Phase 1-4 ✅ +ARC-009 → Can start anytime (frontend only) +ARC-010B → Requires: ARC-010 ✅ +ARC-012 Phase 5 → Requires: ARC-013 (real jobs to monitor) +ARC-014 → Requires: ARC-013 ✅ +TD-006 Phases 3-5 → Can proceed anytime (incremental) ``` -### **Technical Approach** +### Success Metrics for Next Milestone +- [ ] FIX-001 complete (test isolation) +- [ ] ARC-013 complete (content extraction working) +- [ ] Full save-to-read flow working end-to-end +- [ ] 280+ tests passing (with new ARC-013 tests) +- [ ] User can: register → login → save article → read extracted content → highlight → annotate -- **Parallel Development**: NestJS runs on port 4001 alongside Express on 4000 -- **Feature Flags**: Use environment variables to toggle between implementations -- **Testing**: Comprehensive E2E testing at each ticket boundary (>90% coverage target) -- **Rollback Plan**: Express API remains available during development -- **Backend-First**: Always implement backend before dependent frontend features -- **Data Consistency**: Both APIs share same database during migration -- **No Breaking Changes**: JWT tokens and data formats remain compatible +--- -### **Key Decision Points** - -**Why This Order?** -1. **ARC-005 First**: Enables action buttons in UI, establishes mutation patterns -2. **Search Before Bulk**: Bulk operations need search query syntax -3. **Labels Before Frontend Parity**: Many UI features depend on labels -4. **Frontend Parity Before Content**: Library management can work without new content ingestion -5. **Queue Integration Last**: Most complex, can leverage Express content processing meanwhile - -**Parallel Work Opportunities:** -- ARC-004B (Vite frontend) can progress alongside ARC-005 through ARC-008 -- ARC-010 (Reading/Highlights) can be done in parallel with ARC-011 (if resources allow) -- Documentation and testing improvements can happen continuously +**Document Version**: 2.0 +**Last Updated**: November 21, 2025 +**Managed By**: Development Team +**See Also**: `unified-migration-backlog-complete.md` for completed items diff --git a/docs/architecture/vision-comparison-analysis.md b/docs/architecture/vision-comparison-analysis.md new file mode 100644 index 000000000..6bb647535 --- /dev/null +++ b/docs/architecture/vision-comparison-analysis.md @@ -0,0 +1,515 @@ +# Vision Comparison Analysis: Product Brief vs. Refined Strategy + +**Date**: 2025-01-16 +**Purpose**: Reflect the new understanding against original product documentation + +--- + +## Executive Summary + +**Original Vision** (product-brief.md): "Personal Scholar" - ambitious multi-modal knowledge platform +**Refined Vision** (strategic-vision-2025.md): "Content Inbox + AI Curator" - focused on newsletters, articles, AI triage + +**Key Change**: From "do everything" to "do one thing exceptionally well" - then expand. + +--- + +## Side-by-Side Comparison + +### Product Positioning + +| Aspect | Original Brief | Refined Strategy | Assessment | +|--------|---------------|------------------|------------| +| **Primary Use Case** | Universal knowledge platform for all content types | Content inbox for newsletters + articles with AI triage | ✅ More focused, achievable | +| **Target User** | "People who love to read and learn" (broad) | Ourselves first (dogfooding), then knowledge workers overwhelmed by newsletters | ✅ Clearer target | +| **Main Value Prop** | "Capture everything, learn from everything" | "One inbox for newsletters and articles, AI helps you find what matters" | ✅ More concrete | +| **Differentiator** | Open-source + multi-modal + AI + self-hostable | Open-source + newsletter email ingestion + AI digest | ✅ More defensible | + +### Feature Scope + +| Feature Category | Original Brief | Refined Strategy | Status | +|-----------------|---------------|------------------|---------| +| **Web Articles** | ✅ Save, read, highlight | ✅ Save, read, highlight | ✅ Mostly built | +| **Newsletters** | ✅ Email addresses for subscriptions ⭐ | ✅ Email addresses for subscriptions ⭐ | 🔴 Not built yet | +| **PDF/EPUB** | ✅ Enhanced support with OCR | ✅ Basic support (already working) | ✅ Basic working | +| **Podcasts** | ✅ Transcription, timestamps, highlights | ⏸️ Defer to post-MVP | ⏸️ Deferred | +| **Audiobooks** | ✅ Transcription, sync with eBook text | ⏸️ Defer to post-MVP | ⏸️ Deferred | +| **YouTube Videos** | ✅ Transcripts (was in Omnivore beta!) | ⏸️ Defer to post-MVP | ⏸️ Deferred | +| **Voice Notes** | ✅ Capture and transcribe | ⏸️ Defer to post-MVP | ⏸️ Deferred | +| **AI Summaries** | ✅ Article summarization | ✅ Daily digest with AI summaries ⭐ | 🔴 Not built yet | +| **AI Highlights** | ✅ Auto-highlight key points | ⏸️ Defer (manual highlights first) | ⏸️ Deferred | +| **Semantic Search** | ✅ Vector search with pgvector | ⏸️ Defer (basic search works) | ⏸️ Deferred | +| **RAG/Q&A** | ✅ "What have I learned about X?" | ⏸️ Defer to post-MVP | ⏸️ Deferred | +| **Publishing** | ✅ Public knowledge sharing | ⏸️ Not relevant for solo use | ⏸️ Deferred | +| **Collaboration** | ✅ Shared libraries, team features | ⏸️ Not relevant for solo use | ⏸️ Deferred | + +**Key Insight**: Original brief had **15+ major feature categories**. Refined strategy focuses on **4 core features**: +1. Newsletter email ingestion ⭐ +2. AI digest and triage ⭐ +3. Unified highlights +4. Article reading and saving + +--- + +## Deep Dive: Feature Analysis + +### 1. Newsletter Email Ingestion ⭐ **CRITICAL DIFFERENTIATOR** + +**Original Brief (Section 2: Companion Tools)**: +> "Email Integration: While Omnivore provides an email address to forward newsletters and articles, a desktop tool could integrate with email clients (via plugins or simple mail rules) to automate that." + +**Analysis**: +- ✅ Brief mentioned this but buried it in "companion tools" +- ✅ We've identified this as THE killer feature +- ✅ Omnivore users loved this feature (you fell in love with it) +- 🔴 **Not yet implemented in our codebase** + +**What's Missing in Current Codebase**: +- No EmailModule +- No email parsing service +- No unique email address generation +- No SMTP inbound handling + +**Priority**: **#1 - Build this immediately after completing library UI** + +--- + +### 2. AI Digest & Triage ⭐ **THE NEW DIFFERENTIATOR** + +**Original Brief (Section 1.6: AI-Powered Smart Highlights)**: +> "The Omnivore Digest is a daily summary that used AI to sort and rank your recent items, and make summaries of them." + +**Analysis**: +- ✅ Brief mentioned daily digest +- ✅ Omnivore had this in prototype form +- ➕ **We're expanding this**: not just daily digest, but a full triage workflow +- ➕ **New concept**: "Inbox Zero for Content" - digest → skim → archive in minutes +- 🔴 **Not yet implemented** + +**What We're Adding Beyond Original Brief**: +- Digest view as primary interface (not just newsletter) +- Quick actions from digest (Read/Archive/Delete without opening) +- Smart prioritization (future: AI learns what you care about) +- Batch triage workflow + +**Priority**: **#2 - Build after email ingestion** (needs content flowing in first) + +--- + +### 3. Multi-Modal Content (Podcasts, Audiobooks, Video) + +**Original Brief (Sections 1.1-1.3)**: +> Extensive coverage of: +> - Audiobooks with transcripts and bi-directional sync with eBooks +> - Podcasts with RSS integration, transcription, in-app player +> - YouTube videos with time-synced transcripts + +**Analysis**: +- ✅ Brief was very thorough on multi-modal vision +- ✅ Use cases are compelling (Snipd proves podcast market) +- ⚠️ **Too ambitious for MVP** +- ⏸️ **Deferred to Phase 2** (after we validate core) + +**Why We're Deferring**: +1. Each content type is a separate project (2-3 months each) +2. Transcription costs money (Whisper API or self-hosted) +3. Different UX challenges (audio player, time-stamped highlights) +4. Email + articles already provide huge value + +**When to Revisit**: +- After 2-3 months of daily use with newsletters + articles +- If we personally want podcast transcripts +- If beta users specifically request it + +--- + +### 4. Voice Integration (Voice Notes, Alexa/Google Assistant) + +**Original Brief (Section 1.5 + Section 2.3)**: +> "Voice Notes and Real-Time Audio Clipping" + "Voice Assistant & Smart Speaker Integrations" + +**Analysis**: +- ✅ Brief was comprehensive on voice capture +- ✅ Use cases make sense (hands-free capture) +- ⚠️ **Complex to implement** (device permissions, background audio) +- ⏸️ **Defer indefinitely** (not critical for knowledge workers at desk) + +**Why We're Deferring**: +- Voice capture is harder than it sounds (iOS/Android restrictions) +- Smart speaker integrations require separate skills/actions +- Not a core use case for newsletter reading +- Can always add later if needed + +--- + +### 5. AI-Powered Features (RAG, Semantic Search, Auto-Highlights) + +**Original Brief (Section 1.6: AI-Powered Smart Highlights)**: +> - Auto-highlights (AI suggests key sentences) +> - Semantic tagging (auto-tag content by topics) +> - Related content recommendations +> - "Ask your library" (RAG Q&A) + +**Analysis**: +- ✅ Brief correctly identified AI as differentiator +- ✅ pgvector already set up in database (foundation ready) +- ➕ **We're focusing on practical AI first**: summaries for triage +- ⏸️ **Advanced AI deferred**: RAG, semantic search, auto-highlights + +**Our AI Strategy**: +1. **Phase 1** (MVP): AI summaries for digest (simple, high value) +2. **Phase 2** (post-MVP): Semantic search (leverage pgvector) +3. **Phase 3** (future): RAG Q&A ("What have I learned about X?") +4. **Phase 4** (far future): Auto-highlights, topic clustering, insights + +**Why This Order**: +- Summaries deliver immediate value (save time every day) +- Semantic search requires content corpus (need more saved items first) +- RAG requires significant implementation (vector store, retrieval, prompting) +- Auto-highlights are cool but manual highlighting works fine + +--- + +### 6. Publishing & Social Features + +**Original Brief (End of Section 1.6)**: +> "Export into a long form journal or blog with back links sounds fun. Perhaps the links could be made for public view." + +**Original product-thoughts.md**: +> "Export into a long form journal or blog with back links sounds fun. Perhaps the links could be made for public view." + +**Analysis**: +- ✅ Nice vision for future +- ⚠️ **Not relevant for solo use** (building for ourselves first) +- ⏸️ **Defer indefinitely** (maybe never build) + +**Why We're Not Building This**: +- Publishing is a separate product (Medium, Substack already exist) +- Adds complexity (public pages, permissions, moderation) +- Distraction from core value (content inbox + knowledge capture) +- Can always export to Obsidian/Notion and publish from there + +**If We Ever Build It**: +- Simple export: highlights → markdown → publish wherever +- Maybe: Public highlight collections (like Readwise public pages) +- Not: Full blogging platform + +--- + +### 7. Integrations & Ecosystem + +**Original Brief (Section 2.4: Desktop & Workflow Integrations)**: +> - Obsidian/Logseq plugins +> - Notion integration +> - IFTTT/Zapier workflows +> - Raycast extension (already exists!) + +**Analysis**: +- ✅ Brief correctly prioritized integrations +- ✅ **Already partially built**: Obsidian and Logseq plugins mentioned in codebase +- ✅ **Right approach**: be data layer for ecosystem +- ➕ **We should double down on this** + +**Our Integration Strategy**: +1. **Phase 1** (MVP): Export highlights to markdown (basic) +2. **Phase 2** (post-MVP): Obsidian sync (two-way if possible) +3. **Phase 3** (post-MVP): Webhooks for automation (IFTTT/Zapier) +4. **Phase 4** (post-MVP): Public API for community plugins + +**Why This Matters**: +- Integrations reduce lock-in fear (open-source ethos) +- Ecosystem increases value (network effects) +- Community can build features we don't (plugins for multi-modal, etc.) + +--- + +## Architectural Alignment + +### What Original Brief Got Right Architecturally + +**Brief mentioned these technical approaches**: +1. ✅ Queue system for background processing (BullMQ) → **We built this!** +2. ✅ Open-source speech-to-text (Whisper) → **Deferred but architecture ready** +3. ✅ AI integration (OpenAI, Anthropic) → **Ready to implement** +4. ✅ Vector search (pgvector) → **Already in database!** +5. ✅ Self-hostable with Docker → **Already working** + +**We're in great shape**: The technical foundation matches the brief's vision. + +### What We've Added Architecturally + +**New modules not in original brief**: +1. ✅ **EventBusService** - event-driven architecture for loose coupling +2. ✅ **Vite frontend** - 50-100x faster dev experience than Next.js +3. ✅ **TypeORM with proper entities** - type-safe database layer +4. ✅ **Comprehensive testing** - 203 tests (brief didn't mention testing) +5. ✅ **Performance optimizations** - 26x faster queries (brief didn't address scale) + +**We're more mature architecturally** than the brief envisioned. + +--- + +## Monetization & Open-Source Strategy + +### Original Brief (Section 3: Ethical Monetization) + +**Proposed models**: +1. ✅ Freemium (core free, premium AI/TTS paid) +2. ✅ Donations (Open Collective, GitHub Sponsors) +3. ✅ Affiliate revenue (bookshops, tools) +4. ✅ Self-hosted vs. hosted service (open-core) + +**Our stance**: +- ✅ **Agree with freemium + self-hosted model** +- ✅ **Open-source is non-negotiable** +- ✅ **Privacy-first is core value** +- ➕ **Clarification**: Hosted service = convenience, not lock-in + +**Pricing strategy** (future): +- Free tier: + - Unlimited articles, newsletters, highlights + - Basic search + - Export to markdown +- Premium tier ($5-10/mo): + - AI summaries and digest + - Advanced search (semantic) + - Priority sync + - Higher limits (if needed) +- Self-hosted: Free forever (bring your own OpenAI key) + +**Revenue goal** (far future): +- Cover hosting costs (~$100/mo) +- Cover AI costs (~$50-200/mo depending on users) +- Sustain development (pay for time) +- **Not**: Get rich, maximize growth, venture scale + +--- + +## User Research & Validation + +### Original Brief Assumptions + +**Brief assumed**: +- Large market for multi-modal learning tools +- Users want podcasts + audiobooks + articles unified +- AI-powered research assistant has demand +- Open-source alternatives to Readwise/Pocket needed + +**What we're validating differently**: +- ✅ **Building for ourselves first** (dogfooding) +- ✅ **No user research until we love using it** +- ✅ **Launch small, iterate based on real usage** + +**Why this is better**: +1. Avoids "building what we think users want" +2. Ensures product quality (we're the harshest critics) +3. Faster iteration (no user feedback delays) +4. Clear success metric: Do we use it every day? + +**When to do user research**: +- After 2-3 months of daily personal use +- When considering major new features (multi-modal, etc.) +- After beta launch (10-20 users giving feedback) + +--- + +## Timeline Comparison + +### Original Brief Timeline (Implicit) + +**Brief implied**: +- Horizon 1 (Enhanced Read-It-Later): 6 months +- Horizon 2 (Multi-Modal): 6-12 months +- Horizon 3 (AI Research Assistant): 12-24 months +- **Total: 2-3 years to full vision** + +**Resources assumed**: Small team (2-4 people) + +### Our Revised Timeline + +**Phase 1-6** (MVP for ourselves): +- 4-5 months to daily-use quality +- Solo developer, part-time + +**Phase 7+** (Beta and beyond): +- 2-3 months of beta feedback +- Iterate and polish +- **Total: 6-8 months to public launch** + +**Multi-modal expansion** (if we want it): +- 3-6 months per content type (podcasts, video, etc.) +- Only if validated by personal use or user demand + +**Timeline comparison**: +- Original: 2-3 years to full vision +- Ours: 6-8 months to core product, then evaluate +- **We're being realistic**: Better to ship something great than promise everything + +--- + +## Risk Assessment Comparison + +### Original Brief Risks (Identified) + +**Ethical considerations**: +- ✅ Podcast/audiobook transcription copyright → **Deferred** +- ✅ Self-hosting complexity for users → **We'll address with docs** +- ✅ AI costs at scale → **Premium tier covers this** + +**Technical considerations**: +- ✅ OCR and transcription compute costs → **Deferred to premium** +- ✅ Multi-modal UX complexity → **Deferred to post-MVP** + +### Additional Risks We've Identified + +**New risks**: +1. **Email deliverability**: Will newsletters be delivered to our SMTP endpoint? + - Mitigation: Use SendGrid (proven solution) +2. **AI summary quality**: Will summaries actually be useful? + - Mitigation: Prototype with OpenAI playground first +3. **Dogfooding discipline**: Will we actually use it daily? + - Mitigation: Commit to 2-month trial, document friction +4. **Scope creep**: Will we try to build everything from brief? + - Mitigation: This strategy document, ruthless prioritization + +--- + +## What We're Learning + +### Insights from Comparison + +1. **Original brief was comprehensive but over-ambitious** + - 15+ major features → 2-3 years of work + - We've refocused on 4 core features → 4-5 months + +2. **The killer feature was always newsletters** + - Brief mentioned it but didn't emphasize enough + - We've made it the cornerstone (rightfully so) + +3. **AI should be practical first, magical later** + - Brief jumped to RAG and semantic search + - We're starting with summaries (immediate value) + +4. **Multi-modal can wait** + - Brief led with podcasts/audiobooks + - We're validating core first (newsletters + articles) + +5. **Architecture is solid** + - Brief's technical direction was correct + - We've built a foundation that supports future expansion + +6. **Open-source positioning is right** + - Brief correctly identified differentiation + - We're doubling down: self-hostable, privacy-first, no lock-in + +--- + +## Recommendations + +Based on this comparison analysis: + +### 1. Update Product Brief + +**Create `product-brief-v2.md`**: +- Focus on content inbox (not "Personal Scholar") +- Lead with newsletter email ingestion (the killer feature) +- Position AI digest as differentiator +- Defer multi-modal to "Future Vision" section +- Keep architecture and monetization sections (they're good) + +### 2. Retire Unfocused Documents + +**Archive or delete**: +- Original `product-brief.md` → Save as `product-brief-v1-archive.md` +- `product-thoughts.md` → Useful context but outdated, archive + +**Keep**: +- `strategic-vision-2025.md` → New source of truth +- `unified-migration-backlog.md` → Technical roadmap + +### 3. Align Structurizr Workspace + +**Update `workspace.dsl`**: +- Remove unneeded modules (publishing, collaboration) +- Add EmailModule, DigestModule, AIModule +- Update component descriptions to match refined vision +- Create "current state" vs. "target state" views + +### 4. Update README and Docs + +**Positioning change**: +- Old: "Open-source read-it-later with multi-modal support" +- New: "Open-source content inbox with AI-powered triage" + +**Key messages**: +- "All your newsletters and articles in one place" +- "AI digest shows what matters in minutes" +- "Capture highlights across everything you read" +- "Self-hostable, privacy-first, open-source" + +--- + +## Conclusion + +### What We're Keeping from Original Brief + +✅ **Vision elements**: +- Open-source and self-hostable +- Privacy-first, no data selling +- Newsletter email ingestion +- Unified highlights and knowledge capture +- AI-powered features +- Integrations with PKM tools + +✅ **Technical approach**: +- NestJS architecture +- Queue system (BullMQ) +- Vector search foundation (pgvector) +- OpenAI/Anthropic integration +- Self-hosting with Docker + +### What We're Changing + +🔄 **Scope**: +- From: "Everything" (15+ features) +- To: "Core 4" (email, AI digest, highlights, reading) + +🔄 **Timeline**: +- From: 2-3 years +- To: 4-6 months to MVP + +🔄 **Approach**: +- From: Build for imagined users +- To: Build for ourselves, then share + +### What We're Deferring + +⏸️ **Multi-modal**: Podcasts, audiobooks, YouTube, voice notes +⏸️ **Advanced AI**: RAG, semantic search, auto-highlights +⏸️ **Social**: Publishing, collaboration, sharing + +### The Core Truth + +The original product brief described a **2-3 year vision** for a funded team. + +We're executing a **4-6 month MVP** as a solo developer. + +Both visions are valid. Ours is just more realistic for our resources. + +**The best part**: Our foundation (NestJS, BullMQ, pgvector) supports the full brief's vision. We can expand later if we want. We're just being disciplined about MVP scope. + +--- + +## Next Actions + +Based on this analysis: + +1. ✅ **This document** - Capture comparison +2. [ ] **Update Structurizr** (ARC-016) - Align architecture docs +3. [ ] **Create Product Brief v2** - Focused positioning +4. [ ] **Choose a name** - Rebrand from Omnivore +5. [ ] **Finish library UI** (ARC-009) - Complete foundation +6. [ ] **Build email ingestion** (ARC-017) - The killer feature +7. [ ] **Build AI digest** (ARC-018) - The differentiator + +**Let's ship something great.** diff --git a/packages/api-nest/WEB_ANNOTATION_MIGRATION_SUMMARY.md b/packages/api-nest/WEB_ANNOTATION_MIGRATION_SUMMARY.md new file mode 100644 index 000000000..73035e162 --- /dev/null +++ b/packages/api-nest/WEB_ANNOTATION_MIGRATION_SUMMARY.md @@ -0,0 +1,223 @@ +# Web Annotation Selector Type System Migration + +## Overview + +Migrated from a generic `HighlightSelector` interface to proper **W3C Web Annotation Data Model** types, ensuring type safety across the entire stack while following web standards. + +## Why This Change? + +### The Problem +The original `HighlightSelector` interface didn't match the actual data format: + +```typescript +// OLD: Generic interface (didn't match actual data) +interface HighlightSelector { + type: 'text-quote' | 'range' | 'xpath' | 'css' + value: string + start?: number + end?: number +} +``` + +But the database (migration 0193) and frontend were already using Web Annotation format: +```json +{ + "textQuote": { "exact": "...", "prefix": "...", "suffix": "..." }, + "textPosition": { "start": 0, "end": 100 }, + "domRange": { "startPath": "...", "endPath": "..." } +} +``` + +### The Solution +Implemented proper TypeScript interfaces matching the W3C specification: + +```typescript +// NEW: W3C Web Annotation Data Model types +interface TextQuoteSelector { + exact: string // The exact text being highlighted + prefix?: string // Text before (for disambiguation) + suffix?: string // Text after (for disambiguation) +} + +interface TextPositionSelector { + start: number // Character position start + end: number // Character position end +} + +interface RangeSelector { + startSelector: XPathSelector | CSSSelector + endSelector: XPathSelector | CSSSelector + startOffset?: number + endOffset?: number +} + +interface HighlightSelectors { + textQuote: TextQuoteSelector // REQUIRED (database constraint) + textPosition?: TextPositionSelector + domRange?: RangeSelector +} +``` + +## Benefits + +### 1. **Type Safety** +- No more `any` types +- Compiler enforces correct selector structure +- Auto-completion in IDEs + +### 2. **Standards Compliance** +- Follows W3C Web Annotation specification +- Interoperable with other annotation systems +- Future-proof architecture + +### 3. **Multi-Strategy Anchoring** +- Primary: TextQuote (exact text matching) +- Fallback 1: TextPosition (character positions) +- Fallback 2: DomRange (DOM structure) +- Makes highlights resilient to content changes + +### 4. **Database Alignment** +- Types match database constraint: `selectors->'textQuote' ? 'exact'` +- No runtime type mismatches +- GraphQL schema reflects actual data + +## Files Changed + +### NestJS API (`packages/api-nest/`) + +1. **src/highlight/entities/highlight-selector.interface.ts** + - Replaced generic `HighlightSelector` with W3C-compliant interfaces + - Added comprehensive documentation with W3C spec links + - Defined: `TextQuoteSelector`, `TextPositionSelector`, `RangeSelector`, `HighlightSelectors` + +2. **src/highlight/entities/highlight.entity.ts** + - Changed `selectors` type from `Record` to `HighlightSelectors` + - Added W3C spec reference in comments + +3. **src/highlight/highlight.service.ts** + - Updated selector construction to use `HighlightSelectors` type + - Fallback logic creates proper Web Annotation format + +4. **src/highlight/dto/highlight.type.ts** (GraphQL output type) + - Changed from `Record` to `HighlightSelectors` + - Updated description to reference W3C standard + +5. **src/highlight/dto/highlight-inputs.type.ts** (GraphQL input type) + - Changed `selectors` field to use `HighlightSelectors` type + - Updated description to reference W3C standard + +### Frontend (`packages/web-vite/`) + +✅ **No changes needed!** The frontend already uses the correct types: +- `AnchorTextQuote` (matches `TextQuoteSelector`) +- `AnchorTextPosition` (matches `TextPositionSelector`) +- `AnchorDomRange` (matches `RangeSelector`) +- `AnchoredSelectors` (matches `HighlightSelectors`) + +### Database + +✅ **No migration needed!** The database already: +- Stores selectors as JSONB in Web Annotation format +- Enforces constraint: `selectors->'textQuote' ? 'exact'` +- Has migration 0193 comments documenting the format + +## Testing + +All tests pass with proper typing: +- ✅ `highlight.e2e-spec.ts` - 30/30 tests passing +- ✅ `factories-example.e2e-spec.ts` - 3/3 tests passing +- ✅ Full e2e suite: 173 passed (up from 150) + +## Web Annotation Specification Reference + +The implementation follows the W3C Web Annotation Data Model: +- Specification: https://www.w3.org/TR/annotation-model/ +- Selectors: https://www.w3.org/TR/annotation-model/#selectors +- TextQuote: https://www.w3.org/TR/annotation-model/#text-quote-selector +- TextPosition: https://www.w3.org/TR/annotation-model/#text-position-selector +- Range: https://www.w3.org/TR/annotation-model/#range-selector + +## Example Usage + +### Creating a Highlight (Service) + +```typescript +const selectors: HighlightSelectors = { + textQuote: { + exact: 'This is the highlighted text', + prefix: 'context before ', + suffix: ' context after' + }, + textPosition: { + start: 1234, + end: 1260 + } +} + +const highlight = await highlightService.createHighlight(userId, { + libraryItemId: '...', + quote: 'This is the highlighted text', + selectors // Type-safe! +}) +``` + +### GraphQL Mutation + +```graphql +mutation { + createHighlight(input: { + libraryItemId: "..." + quote: "This is the highlighted text" + selectors: { + textQuote: { + exact: "This is the highlighted text" + prefix: "context before " + suffix: " context after" + } + } + }) { + id + selectors # Returns HighlightSelectors + } +} +``` + +### Frontend (Applying Highlights) + +```typescript +const highlight: AnchoredHighlight = { + id: '...', + color: 'YELLOW', + selectors: { + textQuote: { exact: '...', prefix: '...', suffix: '...' }, + textPosition: { start: 1234, end: 1260 }, + domRange: { startPath: '0/1/2', endPath: '0/1/3', ... } + } +} + +// Multi-strategy anchoring automatically tries: +// 1. domRange (most precise) +// 2. textPosition (fallback) +// 3. textQuote (last resort) +applyHighlights([highlight], rootElement) +``` + +## Migration Checklist + +- [x] Define W3C-compliant TypeScript interfaces +- [x] Update entity types +- [x] Update service types +- [x] Update GraphQL types (input & output) +- [x] Update factory types +- [x] Verify frontend types align +- [x] Run all tests +- [x] Document changes + +## Future Enhancements + +With proper typing in place, we can now: +1. Add XPath and CSS selectors (already in W3C spec) +2. Implement selector refinement strategies +3. Add selector confidence scoring +4. Support annotation fragments +5. Enable cross-document annotations diff --git a/packages/web-vite/DESIGN-SYSTEM-BRIEF.md b/packages/web-vite/DESIGN-SYSTEM-BRIEF.md new file mode 100644 index 000000000..2540df738 --- /dev/null +++ b/packages/web-vite/DESIGN-SYSTEM-BRIEF.md @@ -0,0 +1,838 @@ +# Omnivore Design System Brief +## A Comprehensive Guide for Design Consultants, Researchers, and Contributors + +--- + +## 🎯 Executive Summary + +**Project**: Omnivore - Read-it-later application for saving, organizing, and reading web content +**Goal**: Create a beautiful, functional, and hierarchically pleasing design system that serves power users while remaining accessible to newcomers +**Current State**: Migration from legacy web package to modern Vite-based frontend +**Design Philosophy**: Information-dense yet elegant; prioritize functionality without sacrificing aesthetics + +--- + +## 📋 Table of Contents + +1. [Product Vision & Context](#product-vision--context) +2. [User Personas & Use Cases](#user-personas--use-cases) +3. [Current Design System Analysis](#current-design-system-analysis) +4. [Design Challenges & Opportunities](#design-challenges--opportunities) +5. [Visual Hierarchy Principles](#visual-hierarchy-principles) +6. [Component Design Requirements](#component-design-requirements) +7. [Design Research Questions](#design-research-questions) +8. [Deliverables & Participation Guidelines](#deliverables--participation-guidelines) + +--- + +## 1. Product Vision & Context + +### What is Omnivore? + +Omnivore is a **read-it-later application** that enables users to: +- **Save** web articles, PDFs, and RSS feeds for later reading +- **Organize** content with labels, folders, and search +- **Read** with distraction-free reader mode and progress tracking +- **Highlight** and annotate important passages +- **Sync** across devices with cloud storage + +### Target Audience + +**Primary Users**: +- **Knowledge workers** - Researchers, writers, students who process large volumes of information +- **Avid readers** - People who save 10-100+ articles per week +- **Information curators** - Those who organize and categorize content systematically + +**Usage Patterns**: +- **High-frequency users**: Daily interaction, managing hundreds to thousands of saved items +- **Batch processors**: Save during the day, read in dedicated sessions +- **Cross-device**: Mobile for saving, desktop/tablet for reading + +### Competitive Landscape + +**Direct Competitors**: Pocket, Instapaper, Raindrop.io, Matter +**Differentiators**: Open-source, privacy-focused, powerful organization features, full-text search + +--- + +## 2. User Personas & Use Cases + +### Persona 1: "Research Rachel" +**Profile**: PhD student, 28, saves 30-50 academic articles/week +**Goals**: Organize research by topic, find articles quickly, track reading progress +**Pain Points**: Information overload, difficulty finding saved content, lack of visual organization +**Design Needs**: Clear hierarchy, powerful search/filter, efficient batch operations + +### Persona 2: "Tech Tom" +**Profile**: Software engineer, 35, subscribes to 20+ RSS feeds +**Goals**: Stay current with tech news, save tutorials, reference documentation +**Pain Points**: Too many unread items, hard to prioritize, slow interface +**Design Needs**: Quick scanning, visual density options, fast navigation + +### Persona 3: "Casual Caroline" +**Profile**: Marketing professional, 42, saves 5-10 articles/week +**Goals**: Casual reading, simple organization, clean interface +**Pain Points**: Overwhelming interfaces, too many features, hard to learn +**Design Needs**: Simplified UI, clear affordances, guided workflows + +### Key Use Cases + +**Use Case 1: Rapid Triage** +- User opens library with 100+ unread items +- Needs to quickly scan, decide what to read now vs later vs archive +- **Design requirement**: Visual scanning efficiency, batch actions, clear metadata + +**Use Case 2: Focused Reading** +- User selects article to read from library +- Needs distraction-free reading with progress tracking +- **Design requirement**: Smooth transition, minimal chrome, progress indicators + +**Use Case 3: Research Organization** +- User has articles on 5-6 different topics +- Needs to categorize, tag, and retrieve by topic +- **Design requirement**: Intuitive labeling, multi-select, visual grouping + +**Use Case 4: Cross-Device Sync** +- User saves on mobile during commute +- Reads on desktop at home +- **Design requirement**: Responsive design, consistent experience, sync indicators + +--- + +## 3. Current Design System Analysis + +### 3.1 Legacy System Strengths + +**What Works Well**: +1. **Hover Actions Pattern**: Clean icon-only action bar (📖 📦 🏷️ 🌐 🗑) that appears on hover + - Reduces visual clutter + - Familiar interaction pattern + - Keeps cards compact + +2. **Label Chips**: Color-coded labels with `● Label Name` format + - Visual categorization + - Quick scanning + - Consistent 11px font, 5px border-radius + +3. **Card Hierarchy**: Clear information prioritization + - Metadata (flair icons + time) → Title → Author/Source → Labels + - 2-line title clamp prevents excessive height + - Reading time and saved date prominent + +4. **Dual Layout Modes**: Grid vs List views for different scanning needs + - Grid: 400px max-width, 150px thumbnail, vertical layout + - List: Horizontal layout, 55x55px thumbnail, single-line title + +5. **Dark Theme Foundation**: `#1a1a1a` base with `#2a2a2a` surfaces + - Reduces eye strain for long reading sessions + - Consistent contrast ratios + - Matches modern app trends + +### 3.2 Current System Gaps + +**Areas Needing Improvement**: + +1. **Inconsistent Spacing** + - Mix of hardcoded pixel values and inconsistent gaps + - **Solution**: Design token system with 4px base scale + +2. **Button Visual Weight** + - Primary actions (Read) not sufficiently prominent + - Secondary actions compete for attention + - **Solution**: Clear primary/secondary/tertiary hierarchy + +3. **Label Presentation** + - Current: Icon + "Label" text (redundant) + - Legacy: Icon-only for flair, color chip + text for user labels + - **Solution**: Differentiate system vs user labels + +4. **Density Control** + - One-size-fits-all card size + - No user control over information density + - **Solution**: Compact/comfortable/spacious view modes + +5. **Processing State Feedback** + - Unclear when articles finish processing + - No visual distinction for PROCESSING vs SUCCEEDED states + - **Solution**: Loading skeleton, state badges, toast notifications + +6. **Mobile Responsiveness** + - Hover actions don't work on touch devices + - Cards too large on small screens + - **Solution**: Touch-friendly fallbacks, adaptive layouts + +--- + +## 4. Design Challenges & Opportunities + +### Challenge 1: Information Density vs Readability + +**Problem**: Users want to see many items at once (density) but also need to read metadata easily (clarity) + +**Design Question**: How do we balance information density with visual clarity? + +**Opportunities**: +- User-controlled density settings +- Progressive disclosure (show more on hover/focus) +- Smart defaults based on screen size +- Typography scale that maintains readability at different sizes + +**Research Needed**: +- What metadata is most critical for decision-making? +- How many cards should be visible in viewport? +- What's the optimal title length (1-line, 2-line, 3-line)? + +--- + +### Challenge 2: Action Discoverability vs Visual Clutter + +**Problem**: 5-7 possible actions per card (read, archive, label, open, delete, share, etc.) create visual noise if always visible + +**Design Question**: How do we make actions discoverable without cluttering the interface? + +**Current Solutions**: +- ✅ Hover-to-reveal action bar (works on desktop) +- ❌ No touch device solution +- ❌ Actions not keyboard accessible + +**Opportunities**: +- Context menus (right-click, long-press) +- Swipe gestures on mobile +- Keyboard shortcuts with visual hints +- "More actions" overflow menu + +**Research Needed**: +- Which actions are used most frequently? +- Do users prefer always-visible or hover-reveal? +- How do touch users currently access actions? + +--- + +### Challenge 3: Label System Complexity + +**Problem**: Two types of labels with different purposes: +1. **System labels** (flair): favorite ⭐, pinned 📌, newsletter 📧, feed 📡 +2. **User labels** (tags): custom categories with colors + +**Design Question**: How do we visually distinguish system vs user labels? + +**Current Approach**: +- System labels: Icon-only in metadata row +- User labels: Color chip + text below title + +**Opportunities**: +- Icon-only for both (saves space) +- Different visual treatments (outline vs filled) +- Hierarchical display (important labels first) +- Collapsible label section for cards with many labels + +**Research Needed**: +- Do users understand icon-only labels? +- What's the optimal number of visible labels before "show more"? +- Should labels be clickable for filtering? + +--- + +### Challenge 4: Multi-Selection & Batch Actions + +**Problem**: Users need to operate on 10-100+ items at once, but multi-select mode changes the UI significantly + +**Design Question**: How do we make batch operations efficient without disrupting the browsing experience? + +**Current Issues**: +- Multi-select toggle at top requires conscious activation +- Checkboxes appear on all cards when enabled +- Exit multi-select loses selection + +**Opportunities**: +- Persistent selection across mode toggle +- Keyboard shortcuts (Shift+click for range) +- Smart selection (e.g., "select all unread") +- Floating action bar for selected items + +**Research Needed**: +- How many items do users typically select? +- What batch operations are most common? +- Should selection persist across navigation? + +--- + +### Challenge 5: Reading Progress Visualization + +**Problem**: Progress bars show 0-100% but lack context (started reading vs midway vs nearly done) + +**Design Question**: How do we communicate reading state meaningfully? + +**Current Approach**: +- Thin progress bar at bottom of thumbnail +- Color changes: Blue (started) → Yellow (midway) → Orange (high) → Green (done) + +**Opportunities**: +- Progress percentage text ("45% read") +- Time remaining estimate ("3 min left") +- Visual badges ("In progress", "Finished") +- Calendar heatmap of reading activity + +**Research Needed**: +- Do users care about exact percentage? +- Is color-coding sufficient? +- Should "completed" items be visually distinct? + +--- + +### Challenge 6: State Indicators + +**Problem**: Articles have multiple states (PROCESSING, SUCCEEDED, ARCHIVED, FAILED) that need clear visual communication + +**Design Question**: How do we show state without overwhelming the card design? + +**Current Approach**: +- State shown in metadata row as text +- Toast notification when processing completes +- No visual distinction on card + +**Opportunities**: +- Loading skeleton during PROCESSING +- Success badge/checkmark when SUCCEEDED +- Failure indicator with retry action +- Greyed out archived items + +**Research Needed**: +- Do users understand current state indicators? +- Should failed items be prominently flagged? +- How long should success confirmations be visible? + +--- + +## 5. Visual Hierarchy Principles + +### 5.1 Information Hierarchy (Most to Least Important) + +**Level 1: Primary Information** (Largest, highest contrast) +- Article title +- Thumbnail/cover image + +**Level 2: Supporting Metadata** (Medium size, medium contrast) +- Author name +- Site/source name +- Reading time estimate +- Saved date/time + +**Level 3: Organizational Elements** (Smallest, lower contrast) +- Labels/tags +- Progress indicators +- State badges + +**Level 4: Actions** (Hidden by default, revealed on interaction) +- Read, Archive, Label, Delete, etc. +- Overflow menu for additional actions + +### 5.2 Visual Weight Distribution + +**Card Layout Proportions**: +``` +┌─────────────────────────────┐ +│ Thumbnail (30-40% height) │ ← High visual weight +├─────────────────────────────┤ +│ Metadata Row │ ← Low weight (12px, muted) +│ ⌚ 2 days ago • 5 min │ +├─────────────────────────────┤ +│ Article Title │ ← Highest weight (16px, bold) +│ (1-2 lines, 700 weight) │ +├─────────────────────────────┤ +│ Author | Source │ ← Medium weight (12px, 400) +├─────────────────────────────┤ +│ ● Label1 ● Label2 │ ← Low weight (11px chips) +├─────────────────────────────┤ +│ [Progress Bar 4px] │ ← Very subtle +├─────────────────────────────┤ +│ 2d ago [Hover Actions] │ ← Low weight until hovered +└─────────────────────────────┘ +``` + +### 5.3 Color Hierarchy + +**Primary Colors** (User attention): +- Accent Yellow (`#ffd234`): Branding, primary CTAs +- Primary Blue (`#4a9eff`): Links, read button, selections + +**Secondary Colors** (State communication): +- Success Green (`#4caf50`): Completed reading +- Warning Orange (`#ff9500`): High progress +- Danger Red (`#8b0000`): Delete, errors + +**Neutral Hierarchy** (Information structure): +- Text Primary (`#ffffff`): Titles +- Text Secondary (`#d9d9d9`): Important metadata +- Text Tertiary (`#898989`): Timestamps, labels +- Text Muted (`#666666`): Helper text, placeholders + +**Background Hierarchy** (Depth perception): +- BG Primary (`#1a1a1a`): Main canvas +- BG Secondary (`#2a2a2a`): Cards, elevated surfaces +- BG Tertiary (`#252525`): Hover states +- BG Elevated (`#333333`): Modals, dropdowns + +### 5.4 Typography Hierarchy + +**Scale**: +- Display (24-36px): Page titles +- Heading (16-20px): Card titles, section headers +- Body (14-16px): Readable text +- Caption (11-12px): Metadata, labels +- Micro (10px): Timestamps (use sparingly) + +**Weight Distribution**: +- Bold (700): Card titles, primary actions +- Semibold (600): Section headers, active states +- Medium (500): Metadata, timestamps +- Regular (400): Body text, author info + +**Line Height**: +- Tight (1.25): Titles, headings +- Normal (1.5): Body text +- Relaxed (1.75): Long-form reading + +--- + +## 6. Component Design Requirements + +### 6.1 Library Card Component + +**Must-Have Features**: +- ✅ Thumbnail (with fallback placeholder) +- ✅ Title (1-2 line clamp) +- ✅ Metadata (source, author, time, reading time) +- ✅ Labels (color-coded chips) +- ✅ Progress indicator +- ✅ Action buttons (hover-reveal) +- ✅ Multi-select checkbox + +**Design Specifications**: +- **Maximum width**: 400px (grid), 100% (list) +- **Minimum height**: Flexible, content-dependent +- **Border radius**: 8px (--radius-lg) +- **Shadow**: Subtle elevation on hover +- **Transition**: 200ms ease for all interactions + +**States to Design**: +1. Default (unread, no interaction) +2. Hover (actions revealed, slight lift) +3. Selected (multi-select mode) +4. Processing (loading skeleton or spinner) +5. Failed (error state with retry) +6. Archived (reduced opacity or greyed out) + +**Interaction Patterns**: +- Click title/thumbnail → Open in reader +- Click checkbox → Toggle selection +- Hover card → Reveal actions +- Click action → Perform action with feedback + +**Responsive Behavior**: +- **Desktop (>1024px)**: 3-4 column grid, hover actions +- **Tablet (768-1024px)**: 2-3 column grid, hover actions +- **Mobile (<768px)**: 1-2 column grid or list view, tap for actions menu + +--- + +### 6.2 Action Button System + +**Button Hierarchy**: + +**1. Primary Actions** (High emphasis): +- Read article +- Add to library (for discovery feed) +- Style: Filled, brand color, prominent placement + +**2. Secondary Actions** (Medium emphasis): +- Archive/Unarchive +- Open original +- Style: Icon-only, neutral color, hover-reveal + +**3. Tertiary Actions** (Low emphasis): +- Delete +- Share +- More options (overflow menu) +- Style: Icon-only, neutral color, lower contrast + +**Icon Library**: +- Read: 📖 or ▶️ +- Archive: 📦 or ⬇️ +- Unarchive: ↩ or ⬆️ +- Label: 🏷️ or 🔖 +- Open: 🌐 or ↗️ +- Delete: 🗑 or ✕ +- More: ⋯ or ⋮ + +**Accessibility Requirements**: +- All buttons must have aria-labels +- Keyboard navigation (Tab, Enter, Space) +- Focus indicators (blue ring) +- Tooltips on hover (1s delay) + +--- + +### 6.3 Label/Tag System + +**Label Types**: + +**System Labels** (Flair): +- ⭐ Favorite +- 📌 Pinned +- 📧 Newsletter +- 📡 Feed +- 🔖 Recommended +- Display: Icon-only in metadata row + +**User Labels** (Custom): +- User-defined categories +- Custom colors (from palette) +- Display: `● Label Name` chip format +- Limit: Show 3, "+N more" indicator + +**Design Specs**: +- Height: 20px +- Padding: 4px 7px +- Border-radius: 5px +- Font: 11px, 500 weight +- Background: Semi-transparent label color +- Border: 1px solid (same color, darker) + +**Color Palette** (for user labels): +- Red: `#ef4444` +- Orange: `#f59e0b` +- Yellow: `#ffd234` +- Green: `#10b981` +- Blue: `#3b82f6` +- Purple: `#8b5cf6` +- Pink: `#ec4899` +- Gray: `#6b7280` + +--- + +### 6.4 Progress Indicators + +**Reading Progress Bar**: +- Position: Overlaid on thumbnail or below labels +- Height: 4px +- Style: Rounded ends (--radius-full) +- Color transitions: + - 0%: Gray (`#666`) + - 1-24%: Blue (`#4a9eff`) + - 25-74%: Yellow (`#ffd234`) + - 75-99%: Orange (`#ff9500`) + - 100%: Green (`#4caf50`) + +**Processing State**: +- Option A: Loading skeleton (grey pulsing blocks) +- Option B: Spinner icon in thumbnail area +- Option C: Progress bar with "Processing..." text +- **Question for research**: Which provides clearest feedback? + +--- + +### 6.5 Folder/Filter System + +**3-Tier Navigation**: + +**Tier 1: Top Bar** +- Search input (flexible width, max 600px) +- "+ Add" button (prominent, blue) +- User menu (avatar/icon dropdown) + +**Tier 2: Filters Bar** +- Labels dropdown (🏷️ Labels) +- View toggle (☰ / ⊞) +- Multi-select toggle (☑ Select) +- Sort dropdown (Recent ▾) +- Sort order button (↓ / ↑) + +**Tier 3: Folder Tabs** +- Inbox (default) +- Archive +- Trash +- Selection indicator (right-aligned) + +**Design Specs**: +- Height: 48px per tier +- Padding: 12-16px horizontal +- Gap: 8-12px between elements +- Background: Distinct for each tier + - Tier 1: `#2a2a2a` + - Tier 2: `#252525` + - Tier 3: `#1a1a1a` + +--- + +## 7. Design Research Questions + +### 7.1 User Testing Questions + +**Card Design**: +1. Can you identify what this article is about without reading it? +2. Which elements on the card help you decide whether to read now or later? +3. Where would you click to perform [action]? +4. How quickly can you scan 10 cards and pick one to read? + +**Action Discoverability**: +5. How did you discover the available actions on this card? +6. Which actions do you use most frequently? +7. Is the hover-reveal pattern intuitive or frustrating? +8. How would you perform batch operations (multi-select)? + +**Label System**: +9. What's the difference between the icon labels and text labels? +10. How many labels are too many before it feels cluttered? +11. Would you prefer icon-only labels or text labels? +12. How do you use labels to organize your library? + +**Information Density**: +13. Is there too much/too little information on each card? +14. What metadata would you add/remove? +15. Would you use a compact view option? +16. How important is the thumbnail image? + +### 7.2 A/B Testing Opportunities + +**Test 1: Action Button Styles** +- A: Hover-reveal icon bar (current) +- B: Always-visible primary button + overflow menu +- C: Bottom-anchored floating action bar +- **Measure**: Click-through rate, time to action, user preference + +**Test 2: Label Display** +- A: Icon + text for all labels +- B: Icon-only for system, text for user labels (legacy) +- C: All text-only labels +- **Measure**: Label recognition, scanning speed, user satisfaction + +**Test 3: Card Density** +- A: Compact (no thumbnail, 1-line title) +- B: Comfortable (small thumbnail, 2-line title) [current] +- C: Spacious (large thumbnail, 3-line title + description) +- **Measure**: Scanning efficiency, perceived organization, preference + +**Test 4: Progress Indicator** +- A: Bar only (no text) +- B: Bar + percentage text +- C: Bar + time estimate ("3 min left") +- D: Badge only ("In progress", "Finished") +- **Measure**: Comprehension, usefulness, visual clutter perception + +### 7.3 Analytics to Track + +**Usage Metrics**: +- Most frequently used actions (Read, Archive, Delete, Label, etc.) +- Average time spent in library (scanning vs reading) +- Cards viewed before selecting one +- Multi-select usage frequency +- Label usage patterns (how many labels per item) + +**Performance Metrics**: +- Time to first interaction +- Scroll performance with 100+ cards +- Search speed and accuracy +- Filter/sort responsiveness + +**Engagement Metrics**: +- Reading completion rate (% of articles read to end) +- Return rate (% of saved articles eventually read) +- Organization activity (labeling, archiving frequency) +- Cross-device usage patterns + +--- + +## 8. Deliverables & Participation Guidelines + +### 8.1 What We're Looking For + +**Design Deliverables**: + +**Option 1: Component Designs** +- High-fidelity mockups (Figma, Sketch, or similar) +- Interactive prototypes (preferred) +- Multiple states designed (default, hover, selected, etc.) +- Responsive variations (desktop, tablet, mobile) +- Design rationale document + +**Option 2: Design System Contribution** +- Color palette refinements +- Typography scale optimization +- Spacing/layout system recommendations +- Icon set creation or curation +- Animation/transition guidelines + +**Option 3: User Research** +- Usability test findings (5-10 participants) +- A/B test proposals with success criteria +- User journey maps +- Competitive analysis deep-dive +- Accessibility audit and recommendations + +**Option 4: Conceptual Explorations** +- Alternative navigation patterns +- Novel information visualization +- Gesture/interaction innovation +- AI-assisted organization features +- Cross-device experience design + +### 8.2 Submission Format + +**Required Elements**: +1. **Executive Summary** (1-2 pages) + - Problem statement + - Proposed solution + - Key benefits + +2. **Design Rationale** (2-5 pages) + - User research insights + - Design principles applied + - Trade-off decisions explained + - Accessibility considerations + +3. **Visual Artifacts** (Figma/Sketch files or equivalent) + - Component library + - Example screens + - Interaction flows + - Responsive breakpoints + +4. **Implementation Notes** (Optional but appreciated) + - Technical constraints considered + - CSS/code snippets + - Animation specifications + - Performance considerations + +### 8.3 Evaluation Criteria + +**We'll assess submissions based on**: + +**Functionality** (30%) +- Does it solve the stated problem? +- Are all user needs addressed? +- Is it feasible to implement? +- Does it scale (100s-1000s of items)? + +**Aesthetics** (25%) +- Visual appeal and polish +- Consistent with brand identity +- Modern design trends +- Professional execution + +**Usability** (25%) +- Intuitive interactions +- Clear information hierarchy +- Accessible (WCAG 2.1 AA) +- Keyboard navigation + +**Innovation** (20%) +- Novel approaches +- Thoughtful improvements over legacy +- Consideration of edge cases +- Future-proof thinking + +### 8.4 Participation Details + +**Timeline**: Rolling submissions accepted + +**Compensation**: [To be determined - Open source contribution, paid consultation, or design challenge prizes] + +**License**: All submissions should be compatible with Omnivore's open-source license (AGPL-3.0) + +**Attribution**: Contributors will be credited in the project + +**Communication**: +- Questions: [GitHub Discussions or email] +- Feedback: Provided within 2 weeks of submission +- Iteration: Opportunity to refine based on feedback + +--- + +## 9. Reference Materials + +### 9.1 Current Design System + +**Design Tokens** (CSS Variables): +- See `/packages/web-vite/src/styles/design-tokens.css` +- Complete spacing scale, color palette, typography +- Reference: `/packages/web-vite/DESIGN-TOKENS.md` + +**Legacy System**: +- Explore `/packages/web/components/patterns/LibraryCards/` +- Study hover actions, label chips, card layouts +- Stitches CSS-in-JS patterns + +### 9.2 Brand Assets + +**Colors**: +- Primary: `#ffd234` (Omnivore Yellow) +- Dark theme base: `#1a1a1a` / `#2a2a2a` + +**Typography**: +- Primary font: Inter (Google Fonts) +- Display font: Inter (medium-bold weights) + +**Logo**: [Link to brand assets] + +### 9.3 Competitor Analysis + +**Study these for inspiration**: +- Pocket: Card density, reading progress +- Instapaper: Minimalist aesthetics, typography +- Raindrop.io: Visual organization, tags +- Matter: Social features, clean design +- Readwise Reader: Power user features + +### 9.4 Accessibility Standards + +**Must comply with**: +- WCAG 2.1 Level AA +- Keyboard navigation (all interactive elements) +- Screen reader compatibility (ARIA labels) +- Color contrast ratios (4.5:1 minimum for text) +- Focus indicators (visible on all focusable elements) + +--- + +## 10. Contact & Questions + +**Project Maintainers**: [Contact information] + +**Design Lead**: [Contact information] + +**Community**: +- GitHub: [Repository link] +- Discord: [Server invite] +- Email: [Design feedback email] + +--- + +## Appendix: Key Design Principles + +1. **Information First**: Content and metadata take priority over chrome and decoration + +2. **Progressive Disclosure**: Show essential information immediately, reveal details on interaction + +3. **Consistency Over Novelty**: Familiar patterns create confidence; innovate only where it adds clear value + +4. **Density Options**: Different users have different needs; provide choice without overwhelming + +5. **Performance Matters**: Smooth 60fps interactions, instant feedback, optimistic UI updates + +6. **Accessibility is Non-Negotiable**: Every user deserves full functionality regardless of ability + +7. **Mobile-First Thinking**: Design for touch, scale up to mouse/keyboard + +8. **Data Respect**: Show what matters, hide what doesn't, let users customize + +--- + +**Thank you for your interest in improving Omnivore's design system!** + +We're excited to see your creative solutions and look forward to collaborating with talented designers and researchers who share our vision of building beautiful, functional software that respects users' attention and intelligence. + +--- + +*Document Version*: 1.0 +*Last Updated*: [Current date] +*License*: CC BY 4.0 (Creative Commons Attribution) diff --git a/packages/web-vite/DESIGN-SYSTEM-IMPLEMENTATION-SUMMARY.md b/packages/web-vite/DESIGN-SYSTEM-IMPLEMENTATION-SUMMARY.md new file mode 100644 index 000000000..d8efdfc7c --- /dev/null +++ b/packages/web-vite/DESIGN-SYSTEM-IMPLEMENTATION-SUMMARY.md @@ -0,0 +1,469 @@ +# Design System Implementation Summary + +**Project**: ARC-009B - Design System Overhaul Implementation +**Date**: 2025-01-20 +**Status**: ✅ COMPLETE + +--- + +## Overview + +This document summarizes the implementation of the Omnivore Design System v1.0, based on the Developer Handoff specification and Design System Overhaul Proposal PDFs. + +All work aligns with the ARC-009B backlog item: "Design System Research & Refinement." + +--- + +## ✅ Completed Work + +### 1. Design Tokens Update (design-tokens.css) +**Status**: ✅ COMPLETE + +Updated all design tokens to match Developer Handoff v1.0 specification: + +#### Typography +- ✅ Font family: `'Inter', sans-serif` +- ✅ Font sizes: heading (16px), body (14px), caption (12px), micro (11px) +- ✅ Font weights: regular (400), medium (500), bold (700) +- ✅ Line heights: tight (1.3), normal (1.5) + +#### Colors +- ✅ Accent colors: Brand Yellow (#FFD234), Action Blue (#4A9EFF) +- ✅ State colors: Success (#4CAF50), Warning (#FF9500), Danger (#8B0000) +- ✅ Text colors: Primary (#FFFFFF), Secondary (#D9D9D9), Tertiary (#898989), Muted (#666666) +- ✅ Background colors: Primary (#1a1a1a), Secondary (#2a2a2a), Tertiary (#252525), Elevated (#333333) +- ✅ Border colors with focus states + +#### Spacing & Layout +- ✅ 4px base spacing scale (space-1 through space-16) +- ✅ Border radius values (sm: 4px, md: 5px, lg: 8px) +- ✅ Shadows with proper opacity +- ✅ Transitions (200ms ease-in-out) + +#### Component-Specific Tokens +- ✅ Density-specific padding values (compact/comfortable/spacious) +- ✅ Card thumbnail heights per density mode +- ✅ Title line-clamp values per density +- ✅ Touch target minimums (44px iOS, 48px Android) + +**Files Modified**: +- `/src/styles/design-tokens.css` + +--- + +### 2. Density Prop System +**Status**: ✅ COMPLETE + +Implemented three density modes as specified in Developer Handoff: + +#### Compact Mode +- ✅ No thumbnail display +- ✅ 1-line title clamp +- ✅ Minimal padding (8px) +- ✅ Hide author field +- ✅ Optimized for "Tech Tom" persona (quick scanning) + +#### Comfortable Mode (Default) +- ✅ Medium thumbnail (150px height) +- ✅ 2-line title clamp +- ✅ Moderate padding (12px) +- ✅ Hide author field +- ✅ Balanced information density + +#### Spacious Mode +- ✅ Large thumbnail (180px height) +- ✅ 3-line title clamp +- ✅ Generous padding (16px) +- ✅ Show author field +- ✅ Maximum readability for "Casual Caroline" persona + +**Implementation Details**: +- TypeScript type: `export type CardDensity = 'compact' | 'comfortable' | 'spacious'` +- CSS classes: `.density-compact`, `.density-comfortable`, `.density-spacious` +- Conditional rendering based on density prop +- Responsive adjustments for mobile + +**Files Modified**: +- `/src/components/LibraryItemCard.tsx` +- `/src/styles/LibraryCard.css` + +--- + +### 3. Flair vs Tag Visual Distinction +**Status**: ✅ COMPLETE + +Separated system labels (Flair) from user labels (Tags): + +#### Flair (System Labels) +- ✅ Icon-only display in metadata row +- ✅ FlairBadge component with emoji icons +- ✅ Mapped common labels: Newsletter (📧), RSS (📰), Subscription (🔔), etc. +- ✅ Subtle styling: small rounded badges +- ✅ Tooltip on hover with full label name + +#### Tags (User Labels) +- ✅ Colored chips with text labels +- ✅ Displayed below title/author +- ✅ Maximum 3 visible, "+N more" overflow indicator +- ✅ Hover effects: subtle lift and shadow +- ✅ Colored backgrounds based on user-defined label colors + +**Implementation Details**: +- Extended `Label` interface with `internal?: boolean` property +- Filter labels by `internal` flag: `flairLabels` vs `userTags` +- Separate CSS classes: `.flair-badge` vs `.tag-chip` + +**Files Created**: +- `/src/components/FlairBadge.tsx` +- `/src/styles/FlairBadge.css` + +**Files Modified**: +- `/src/types/api.ts` (added `internal` property to Label interface) +- `/src/components/LibraryItemCard.tsx` (separate rendering logic) +- `/src/styles/LibraryCard.css` (tag chip styles) + +--- + +### 4. CardSkeleton Component +**Status**: ✅ COMPLETE + +Loading placeholder for items in "PROCESSING" state: + +#### Features +- ✅ Shimmer animation for visual feedback +- ✅ Matches LibraryCard structure and dimensions +- ✅ Respects density modes +- ✅ Accessibility: `aria-label="Loading content"` +- ✅ Reduced motion support (disables shimmer) + +#### Animation +- ✅ Pulse animation (opacity fade) +- ✅ Shimmer gradient overlay +- ✅ Respects `prefers-reduced-motion` preference + +**Implementation Details**: +- Conditional rendering in LibraryItemCard: `if (item.state === 'PROCESSING') return ` +- Density-aware skeleton elements (thumbnail, title, tags) +- Performance: CSS animations only (no JavaScript) + +**Files Created**: +- `/src/components/CardSkeleton.tsx` +- `/src/styles/CardSkeleton.css` + +**Files Modified**: +- `/src/components/LibraryItemCard.tsx` (conditional rendering) + +--- + +### 5. Multi-Select Floating Action Bar +**Status**: ✅ COMPLETE + +Batch operations UI for multi-select mode: + +#### Features +- ✅ Floating bar with selected item count +- ✅ Batch actions: Add Labels, Archive, Delete +- ✅ Clear selection button +- ✅ Exit multi-select mode button +- ✅ Keyboard support: Escape key to exit +- ✅ Accessibility: `role="toolbar"`, descriptive aria-labels + +#### Responsive Design +- ✅ Desktop: Centered bottom bar +- ✅ Mobile: Full-width bottom bar, icon-only buttons +- ✅ Touch targets meet minimum size requirements + +#### Visual Design +- ✅ Elevated background with shadow +- ✅ Slide-up fade-in animation +- ✅ Color-coded actions (primary, secondary, danger) +- ✅ Hover states with subtle lift effect + +**Implementation Details**: +- useEffect hook for Escape key listener +- Conditional rendering based on `selectedCount` +- Mobile: Hide text labels, show icons only +- Focus management for keyboard navigation + +**Files Created**: +- `/src/components/MultiSelectActionBar.tsx` +- `/src/styles/MultiSelectActionBar.css` + +--- + +### 6. Accessibility Compliance (WCAG 2.1 AA) +**Status**: ✅ COMPLETE + +All components meet WCAG 2.1 Level AA standards: + +#### Keyboard Navigation +- ✅ All interactive elements keyboard accessible +- ✅ Logical tab order +- ✅ Visible focus indicators (2px blue outline) +- ✅ No keyboard traps + +#### ARIA & Semantics +- ✅ All icon-only buttons have `aria-label` attributes +- ✅ Semantic HTML elements (`

`, `