Option to directly pass the PDF as bytes to process_pdf instead of reading from disk - #67
Closed
jangoepfert wants to merge 1 commit into
Closed
Option to directly pass the PDF as bytes to process_pdf instead of reading from disk#67jangoepfert wants to merge 1 commit into
jangoepfert wants to merge 1 commit into
Conversation
… of loading the PDF from disk
lfoppiano
added a commit
that referenced
this pull request
Aug 15, 2026
- Archive streaming without full decompression (this is half true now, because the client needs the files on the disk.., but it will be automatically fixed with PR #67). - Glob patterns instead of a bare directory, improve the flexibility - S3 input streaming can be also done using S3:// - Atomic writing, move the file after it was fully written so that if the client is killed there is no risk of half-made output file
Member
|
Took over in #124 |
lfoppiano
added a commit
that referenced
this pull request
Aug 15, 2026
The archive and s3 streaming (#117) shipped with a known detour: every entry was written to a temporary directory only so that process_pdf could open it again from a path, with the commit itself noting this would go away once PR #67 landed. It has landed, so this plugs the two together: archive entries and loose s3 objects are now read straight into memory and posted from there, named after the entry (or the s3 basename), and nothing but the results ever touches the disk. process_batch accepts the in-memory documents alongside paths - an entry goes by the name it carries, and since process_pdf returns that same name, the result lands on the same output file it would have as a path. The one input that still takes the temp-dir route is processCitationList, whose .txt files are read by process_txt from a path. The archive tests asserted on the temp dirs the posts came from, which no longer exist; they now assert on what actually crossed the wire - each entry posted once, under its archive name, with its own bytes - plus explicitly that mkdtemp is never called on the pdf path. Completes what #117 left pending on #67.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hi, it would be nice to be able to pass a PDF file that is already loaded in memory directly to Grobid. In my use case, users upload PDFs using an API, which are then parsed directly without writing the content to files first. The parsing result is then added to a database. As process_pdf already returns the parsing result without writing it to a file, adding the option to pass a PDF as bytes to process_pdf would allow for a workflow where everything is kept in-memory.
Thank you for your great work!