Repository navigation
Round of fixes 3 - #143
Merged
Merged
Round of fixes 3 #143
Conversation
A document the server answers 503 to is re-posted every sleep_time seconds for as long as the server stays busy. The retry runs in the pool's worker threads, and the interrupted main thread waited for them: the client kept posting after Ctrl-C and only a second Ctrl-C or a kill stopped it. The busy wait is now an Event wait. When a batch is interrupted the event is set and queued work is cancelled, so waiting workers return at once and the process exits with 130. An interrupted document leaves no error marker. The event is cleared at the start of the next batch.
The service URL was the server plus "/api/<service>", so --server https://host/ asked for https://host//api/isalive. A Hugging Face Space answers that with 400, and the client reported a server that was up as unavailable. Trailing slashes are now stripped when the service URL is built; a path prefix (https://host/grobid/) is kept.
Missing AWS credentials were reported and suggested a fix only when they surfaced while listing a prefix or glob. A single s3:// zip is not listed but opened, and the opening error was logged as "Could not open archive", followed by "No eligible files found", and the command exited 0. A missing-credentials error now stops the run from the archive too, so the command line reports it, suggests --no-sign-request and exits 1. Any other error opening an archive is still logged, and that archive is skipped.
Two inputs of one run could map to the same output: paper.pdf and paper.PDF in one directory, two directories listed together that both hold x.pdf, or two archives that both hold doc.pdf. The second document overwrote the first, while the summary counted both as processed. For archives it was skipped as "already existed". Each output file is now claimed by the first input of the run that maps to it. A later input gets its extension or its archive's name added to the stem (paper.pdf.grobid.tei.xml, doc.batch1.grobid.tei.xml), with -2, -3... if that is taken too, and a warning names both inputs. Inputs are met in the same order on every run, so a rerun finds the same names. Archive entries are claimed before the resume check, so a renamed entry that is finished is skipped without being read. Inputs that do not collide keep their names.
ping() is documented as returning (is_up, status), but on a refused connection, a DNS failure or a timeout it let the constructor's ServerUnavailableException through. It now returns (False, 0) in that case. The constructor still raises when it checks the server.
A 200 with an empty body was written out as a 0-byte TEI file. Every later run then took it for a finished document, so the document was never retried, and --json produced nothing for it. process_pdf and process_txt now return such an answer as a 500 with an explanatory text. The CLI records it as <name>_500.txt and retries it on the next run.
The "Found N file(s)" lines and the summary are printed to stdout and the logs go to stderr. Redirected to a file, stdout is block-buffered, so in `grobid_client ... > run.log 2>&1` the "Found" line landed after every error of the run. The prints now flush.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes
Stop retrying a busy server on Ctrl-C
If the server kept answering 503, the client said "Interrupted" but kept re-posting until a second Ctrl-C. The first Ctrl-C now exits right away with code 130 and leaves no error files.
Accept a server URL written with a trailing slash
--server https://host/produced//api/..., which some servers (a Hugging Face Space among them) reject with 400, so a working server was reported as down. Trailing slashes are now stripped; a path such as/grobid/is kept.Stop and suggest
--no-sign-requestwhen an s3 zip needs credentialsA single
s3://zip read without credentials logged "No eligible files" and exited 0. The run now stops, suggests--no-sign-requestand exits 1. Other errors opening an archive still just skip that archive.Give each input of a run its own output file
paper.pdfandpaper.PDF, or two archives both holdingdoc.pdf, wrote to one file: the second overwrote the first or was skipped as already done. The first input keeps the name and the other gets its extension or archive name added, with a warning:paper.pdf.grobid.tei.xmldoc.batch1.grobid.tei.xmlReruns find the same names, and files without a clash keep theirs.
ping(): report an unreachable server instead of raisingIt returns
(False, 0)instead of raisingServerUnavailableException. The constructor still raises when it checks the server.Treat an empty 200 answer as an error
It was saved as an empty TEI file that every later run treated as finished, so the document was never retried. It is now recorded as
<name>_500.txtand retried on the next run.Flush progress lines so a log file keeps them in order
With
grobid_client ... > run.log 2>&1, the "Found N file(s)" line appeared after all the errors. It now appears first.Not changed: the client still exits 0 when documents fail, and when a glob matches nothing.
Testing