Repository navigation
Save PDFs as searchable documents (upload, paste, or link) - #611
Merged
Merged
Conversation
Item.sourceFileKey holds the original file a document was made from (an uploaded PDF) so it can be downloaded; fileKey stays the cover image. Classified in the file-key inventory, so delete, reclaim and the image proxy cover it. file_unreadable / document_too_long give damaged or over-long PDFs a clear failure message. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A new import-pdf task renders each page with mupdf (WebAssembly, AGPL, loaded on demand in Trigger.dev) and keeps a page's embedded text layer as its text, so analyze-document only OCRs scanned pages. The document pipeline is shared: title, cover analysis, search text and enrichment work as they do for scans. OCR is capped at 30 pages per document and charged to the daily ingestion allowance per scanned page; past either, pages are kept without text and counted on meta.ocrSkippedPages. A damaged, encrypted or 1,000+ page PDF fails without retrying. POST /api/v1/items/documents/pdf saves an uploaded PDF (25MB cap) as a processing document; retrying a PDF document re-runs the import, which skips rendering once pages exist. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Every upload path accepts a PDF (25MB) alongside images and saves it as a document. Pasting a copied file onto the dashboard now uploads it too. A PDF shows a placeholder card until its pages render, downloads as the original PDF, and the owner sees how many scanned pages search can't find when the OCR cap was hit. pdf_uploaded / pdf_upload_rejected track the flow. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The pager and its load error were hard-coded white on the detail pane, which is the theme background — invisible in light mode. They now use the theme's foreground colours, and the story renders on it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
README features, roadmap and services (Vision and OpenAI now also OCR scanned PDF pages; without Trigger.dev PDFs aren't imported), the .env.example Vision note, and a Saving PDFs help page linked from the help index, nav and the scanning guide. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Covers picking a PDF in the Add item dialog (saved as a document with the PDF as its source and a placeholder card), pasting one onto the dashboard, and rejecting one over 25MB before it uploads. Rendering runs in Trigger.dev, outside E2E, and is covered by the import-pdf tests. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
classify-url hands a URL that serves a PDF (by content type, or a .pdf path served as a generic file; checked on HEAD and again on GET for servers that skip HEAD) to a new handlePdfUrl. It downloads the PDF through safeFetch under the 25MB cap, stores it as the item's source file and triggers import-pdf, so it renders and reads like an upload. An extension capture of the browser's PDF viewer fetches the PDF itself. Re-capturing a URL-saved PDF replaces it whole: reclaimReplacedStorage now also deletes its pages and their files and clears its source and cover, so nothing leaks and the storage counter stays exact. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
pdf_imported (pages, scanned pages, pages OCR'd, whether it was a retry) and pdf_import_failed (reason) follow pdf_uploaded, so the upload to import funnel and why imports fail are measurable. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
All reported issues were addressed across 63 files
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
…-add # Conflicts: # README.md # app/src/components/document/document-pages.tsx # app/trigger/reclaim-item-storage.ts
An import only saves its pages if the item still holds the PDF it rendered, so one still running when a URL is re-captured can't bring back the old pages. The OCR allowance is charged once per PDF and kept on meta.ocrPagesCharged, so a retry after a failed enqueue doesn't pay again. Analysis re-reads meta just before saving its skipped-page count rather than writing back a snapshot from before the analysis. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- Recognise the browser's PDF viewer from the captured DOM alone (Chrome's lone embed, Firefox's pdf.js shell), so a real page at a .pdf path or one embedding a PDF keeps its capture - Prefer Content-Disposition's UTF-8 filename* over the ASCII fallback - Remove the stored PDF if saving the item fails, so retries don't orphan it - Enqueue the import through enqueueUserProcessing, like uploads Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- A page MuPDF fails to parse or render fails the import as unreadable (no retry) instead of unknown, and its pixmap is always freed - Show Download PDF before pages exist, e.g. while importing or after a failed import - Don't upload a pasted file from the dashboard while a dialog is open - Strip path separators from an uploaded PDF's name on download - Make the garbled-text test reach the garble check, and guard the PDF fixture's password against mupdf's option syntax Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
All reported issues were addressed across 66 files
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
…anup analyze-document now carries the PDF it was started for: if a URL re-capture replaced it meanwhile, the run saves nothing and doesn't fail the item, even when the replacement broke it (e.g. deleted its pages). A failed page upload stops rendering and lets in-flight uploads finish before the import cleans up, so none can land after its key is removed. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Search results now include sourceFileKey, so a PDF found by search shows and downloads as a PDF. Retrying a PDF document, including one saved from a URL, reruns its import from the stored file instead of downloading and re-capturing the URL. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- Count supplementary private-use and C1 control characters as garbled when deciding whether to trust a page's text layer - Say the daily limit can also leave scanned pages unread, in the viewer note and the help page - Wait for hydration before pasting in the E2E test: the button is server-rendered before the paste listener is attached, which raced on CI Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
PDFs become searchable
documentitems, reusing the scanned-document pipeline. You can upload one from the file picker, by drag and drop, or by pasting it onto the dashboard (25MB cap), or save a link that points to one. A newimport-pdfTrigger.dev task renders each page with mupdf (WebAssembly, AGPL) and uses the page's embedded text where it has some.analyze-documentthen only OCRs the scanned pages, at most 30 per document, each counted against the daily ingestion allowance. Titling, cover analysis, search text and enrichment work as they do for scans, and the original PDF is kept in the newItem.sourceFileKeyfor Download PDF. A damaged, password-protected or 1,000+ page PDF fails with a clear reason and isn't retried.Checklist
pdf_uploaded,pdf_upload_rejected,pdf_imported,pdf_import_failed; the funnel insight is still to build once events arrive.env.example, in-app help,CONTRIBUTING.md, onboarding checklist), or N/A — README,.env.exampleand a new Saving PDFs help page; no new milestone, since milestones are a DB enum and that would need another migrationcaptureServerException/ the error boundary, or N/Alearnskill, or N/A — N/ANotes
20260929204337_add_pdf_documents: addsitems.source_file_key, plus thefile_unreadableanddocument_too_longfailure reasons.mupdf: loaded on demand insideimport-pdfonly, and markedexternalintrigger.config.tsbecause it loads its WebAssembly from beside its own module. A dry-run Trigger.dev deploy build succeeds.import-pdfgets its owndocument-importqueue so long PDFs don't hold image-analysis slots, and retries on asmall-2xmachine if it runs out of memory.reclaimReplacedStoragenow also deletes the old pages and their files.itemsbucket's allowed MIME types includeapplication/pdf. The bucket migration sets it, but it'sON CONFLICT DO NOTHING.🤖 Generated with Claude Code
Summary by cubic
PDFs can now be saved as searchable
documentitems — uploaded from the file picker, by drag and drop, or pasting onto the dashboard (25MB cap), or via a URL that serves a PDF. The original file is kept so it can be re-downloaded, and Download PDF works even before the pages exist.A new
import-pdfTrigger.dev task renders each page withmupdf(WebAssembly) and uses the page's embedded text layer where one exists;analyze-documentonly OCRs the scanned pages, at most 30 per document, each charged once per PDF against the daily ingestion allowance. Past the cap, pages stay viewable but unsearchable and the owner is told how many. Damaged, password-protected, or 1,000+ page PDFs fail with a clear reason and aren't retried; retrying a failed PDF re-runs the import rather than bare analysis. A saved link to a PDF is detected by content type (or.pdfpath with a generic type), and the browser's own PDF viewer is recognized from the page alone — a capture of a real page keeps its markup. An import only saves its pages if the item still holds the PDF it rendered, so a re-capture can't bring back the old pages.Notes for merge and deploy
20260929204337_add_pdf_documentsaddsitems.source_file_keyand thefile_unreadable/document_too_longfailure reasons.mupdf(AGPL, WebAssembly) is markedexternalintrigger.config.tsand loaded on demand only insideimport-pdf, which runs on its owndocument-importqueue.meta.ocrPagesCharged, so a retry after a failed enqueue doesn't pay twice.reclaimReplacedStoragenow also deletes the old pages and their files, removing the stored PDF if the save fails so retries don't orphan it.itemsbucket's allowed MIME types includeapplication/pdf— the bucket migration sets it, but only viaON CONFLICT DO NOTHING.source_file_key, and retrying a URL-saved PDF re-imports from its stored file instead of downloading the URL again.Written for commit daffed8. Summary will update on new commits.