Skip to content

Fix round of bugs - part 2 - #142

Merged
lfoppiano merged 14 commits into
masterfrom
fix/client-round-2
Nov 9, 2026
Merged

lfoppiano merged 14 commits into
masterfrom
fix/client-round-2

Conversation

@lfoppiano

Copy link
Copy Markdown
Member
  • Converters: do not configure the root logger at import time
  • JSON: skip comments and processing instructions when computing ref offsets
  • JSON: keep every reference that carries any information
  • JSON process_directory: pass the converter settings to parallel workers
  • validate_json_refs: count files with bad references as invalid; search subdirectories
  • JSON: normalise abstract text so its reference offsets are exact
  • Markdown: show reference years given as attributes, and arXiv ids
  • Markdown: keep table cells inside their cell and row
  • Format CLI: answer --help, fix the documented commands, add tests
  • CLI: fail on a missing --input; take only XML for processCitationPatentST36
  • Archives: skip finished entries before reading them
  • CLI: exit cleanly on Ctrl-C and say how to resume
  • Read public S3 buckets without credentials: --no-sign-request
  • Add end-to-end tests of the command line against a stand-in GROBID

Both converter modules called logging.basicConfig(level=INFO) when
imported, and grobid_client imports TEI2LossyJSON, so any program using
the library got a root handler at INFO. It also made the format CLIs'
setup_logging() a no-op: basicConfig does nothing once the root logger
has a handler, so --verbose and the CLI log format were ignored.
TEI2LossyJSON's logger did not propagate either, so an application's
handlers never received its warnings.
…fsets

get_text() gives the passage its text and leaves out XML comments and
processing instructions, but the offset walk counted every string node.
Markup of that kind before a reference shifted its offsets, and the
validation assert then aborted the conversion of the whole document.
References were kept only when they had a title, authors, a journal or
one of a few identifiers. Entries GROBID describes with only an arXiv id,
a URL, editors or a publisher were dropped, which left the in-text refs
pointing at them without a target in references[].
The parallel worker built a default converter, so validate_refs was
ignored. It also read each file with open(path, 'r') -- the locale's
encoding -- before converting it, and threw the result away. Where the
locale is not UTF-8 (cp1252 on Windows) that read fails on common bytes
such as those of 'Á' or a closing double quote: 2 of the 9 sample
documents cannot be decoded that way, and they came back as None.
…h subdirectories

A file's 'valid' flag was never cleared, so a file holding invalid
references was reported as valid ("Files: 2/2 valid") and its errors
were missing from the report; only the exit code noticed. Only the top
level of the directory was searched, while the client writes the output
of a recursive run into subdirectories. A JSON file that is not an
object, or an abstract set to null, no longer aborts the whole run.
Reference offsets are computed on whitespace-normalised text, as for the
body, but an abstract passage carried the raw paragraph or sentence
text. An abstract with a line break or a double space before a citation
had offsets pointing a few characters off; on GROBID 0.9.2 output this
hit the abstracts of 3 of 128 passages, and the sdo/ietf flavor of a
ResNet paper had 4 wrong refs. Body and abstract now share one
normalisation helper.

At sentence level the abstract is a list of paragraphs, each a list of
sentences; validate_json_refs only read flat lists, so these refs were
never checked. It now reads both shapes.
GROBID usually writes a reference date as an attribute only
(<date type="published" when="2019"/>). The year was read from the
element text, so it was missing from nearly every reference: 50 of the
51 dates in the sample document have no text. The published date is now
preferred over other dates and read from @when first.

arXiv identifiers were collected but never rendered; they now appear as a
link to the abstract page.
A '|' in a cell ended the cell early and a line break in a cell ended the
row. Pipes are now escaped, whitespace inside a cell is collapsed, and
rows are padded to the widest row, since renderers drop the cells past
the separator row's column count.
`python -m grobid_client.format --help` was reported as an unknown
converter. The examples in each converter's --help ran
`python -m grobid_client.format.TEI2Markdown`, which imports the module
and exits 0 without converting anything; they now use the menu form.
Unknown-converter errors go to stderr. The format command lines had no
tests; they now cover both converters to file and stdout, bad inputs,
the menu, and the commands the help text documents.
…ntST36

A mistyped --input logged "Input path does not exist" and exited 0, so a
script could not tell it from a run with nothing left to do; it now
exits 1. Globs and s3:// inputs are still resolved later, and ~ is
expanded as processing does.

processCitationPatentST36 accepted PDFs as well as ST.36 XML. GROBID
answers a PDF there with 204 No Content, so every PDF in the folder left
an empty <name>_204.txt error marker.
process_batch skips an entry whose TEI already exists, but only once it
holds the entry's bytes, so resuming an archive run read every finished
entry again first. For an s3:// zip that is a download: rerunning on a
public 1.3 GB corpus zip with 330 documents done spent over a minute
fetching them again before reaching new work. Entries whose outputs are
all there (or, with --skip_errors, that left an error marker) are now
filtered out before anything is read; an entry with its TEI but a
missing JSON or Markdown is still read so those get regenerated.
Interrupting a run printed a long traceback from deep inside the S3 or
HTTP stack. Outputs are written atomically and finished documents are
skipped on the next run, so the CLI now says that and exits with 130.
Listing and reading s3:// inputs used the default credential chain only,
so a public bucket such as s3://digitalcorpora/... could not be read on a
machine without AWS credentials: the run stopped with "Unable to locate
credentials". --no-sign-request (s3_no_sign_request=True in the API)
sends unsigned requests, as the AWS CLI option of the same name does.
Listing and range reads now share one boto3 client, and a missing
credentials error suggests the option.
Every other client test mocks the HTTP layer. These run grobid_client in
a subprocess against a small local HTTP server that answers like GROBID,
and check what lands on disk: the input tree is mirrored (they fail on
the code before #126), JSON and Markdown sit next to the TEI, a failure
leaves its error marker, zip and tar.gz archives behave like
directories, --input-list keeps the relative layout, and re-runs skip
finished documents, regenerate a missing JSON, honour --skip_errors and
--force.
@lfoppiano
lfoppiano merged commit 67c902a into master Nov 9, 2026
15 checks passed
@lfoppiano
lfoppiano deleted the fix/client-round-2 branch November 9, 2026 13:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant