Skip to content

Round of fixes 3 - #143

Merged
lfoppiano merged 7 commits into
masterfrom
fix/client-round-3
Nov 10, 2026
Merged

lfoppiano merged 7 commits into
masterfrom
fix/client-round-3

Conversation

@lfoppiano

@lfoppiano lfoppiano commented Nov 10, 2026 •

Copy link
Copy Markdown
Member

Fixes

  1. Stop retrying a busy server on Ctrl-C
    If the server kept answering 503, the client said "Interrupted" but kept re-posting until a second Ctrl-C. The first Ctrl-C now exits right away with code 130 and leaves no error files.

  2. Accept a server URL written with a trailing slash
    --server https://host/ produced //api/..., which some servers (a Hugging Face Space among them) reject with 400, so a working server was reported as down. Trailing slashes are now stripped; a path such as /grobid/ is kept.

  3. Stop and suggest --no-sign-request when an s3 zip needs credentials
    A single s3:// zip read without credentials logged "No eligible files" and exited 0. The run now stops, suggests --no-sign-request and exits 1. Other errors opening an archive still just skip that archive.

  4. Give each input of a run its own output file
    paper.pdf and paper.PDF, or two archives both holding doc.pdf, wrote to one file: the second overwrote the first or was skipped as already done. The first input keeps the name and the other gets its extension or archive name added, with a warning:

    • paper.pdf.grobid.tei.xml
    • doc.batch1.grobid.tei.xml

    Reruns find the same names, and files without a clash keep theirs.

  5. ping(): report an unreachable server instead of raising
    It returns (False, 0) instead of raising ServerUnavailableException. The constructor still raises when it checks the server.

  6. Treat an empty 200 answer as an error
    It was saved as an empty TEI file that every later run treated as finished, so the document was never retried. It is now recorded as <name>_500.txt and retried on the next run.

  7. Flush progress lines so a log file keeps them in order
    With grobid_client ... > run.log 2>&1, the "Found N file(s)" line appeared after all the errors. It now appears first.

Not changed: the client still exits 0 when documents fail, and when a glob matches nothing.

Testing

  • 27 new tests; the suite goes from 342 to 369 and passes on Python 3.8, 3.12 and 3.13.
  • Each commit passes the tests on its own, so the branch can be merged without squash.
  • mypy and flake8 (CI's checks) are clean.
  • Every fix was also checked live against a GROBID 0.9.2-SNAPSHOT server.

A document the server answers 503 to is re-posted every sleep_time
seconds for as long as the server stays busy. The retry runs in the
pool's worker threads, and the interrupted main thread waited for
them: the client kept posting after Ctrl-C and only a second Ctrl-C
or a kill stopped it.

The busy wait is now an Event wait. When a batch is interrupted the
event is set and queued work is cancelled, so waiting workers return
at once and the process exits with 130. An interrupted document
leaves no error marker. The event is cleared at the start of the
next batch.
The service URL was the server plus "/api/<service>", so
--server https://host/ asked for https://host//api/isalive. A Hugging
Face Space answers that with 400, and the client reported a server
that was up as unavailable. Trailing slashes are now stripped when
the service URL is built; a path prefix (https://host/grobid/) is
kept.
Missing AWS credentials were reported and suggested a fix only when
they surfaced while listing a prefix or glob. A single s3:// zip is
not listed but opened, and the opening error was logged as "Could not
open archive", followed by "No eligible files found", and the command
exited 0.

A missing-credentials error now stops the run from the archive too, so
the command line reports it, suggests --no-sign-request and exits 1.
Any other error opening an archive is still logged, and that archive
is skipped.
Two inputs of one run could map to the same output: paper.pdf and
paper.PDF in one directory, two directories listed together that both
hold x.pdf, or two archives that both hold doc.pdf. The second document
overwrote the first, while the summary counted both as processed. For
archives it was skipped as "already existed".

Each output file is now claimed by the first input of the run that maps
to it. A later input gets its extension or its archive's name added to
the stem (paper.pdf.grobid.tei.xml, doc.batch1.grobid.tei.xml), with
-2, -3... if that is taken too, and a warning names both inputs.

Inputs are met in the same order on every run, so a rerun finds the same
names. Archive entries are claimed before the resume check, so a
renamed entry that is finished is skipped without being read. Inputs
that do not collide keep their names.
ping() is documented as returning (is_up, status), but on a refused
connection, a DNS failure or a timeout it let the constructor's
ServerUnavailableException through. It now returns (False, 0) in that
case. The constructor still raises when it checks the server.
A 200 with an empty body was written out as a 0-byte TEI file. Every
later run then took it for a finished document, so the document was
never retried, and --json produced nothing for it.

process_pdf and process_txt now return such an answer as a 500 with
an explanatory text. The CLI records it as <name>_500.txt and retries
it on the next run.
The "Found N file(s)" lines and the summary are printed to stdout and
the logs go to stderr. Redirected to a file, stdout is block-buffered,
so in `grobid_client ... > run.log 2>&1` the "Found" line landed after
every error of the run. The prints now flush.
@lfoppiano
lfoppiano merged commit e1b9495 into master Nov 10, 2026
15 checks passed
@lfoppiano
lfoppiano deleted the fix/client-round-3 branch November 10, 2026 20:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant