Repository navigation
Fix round of bugs - part 2 - #142
Merged
Merged
Conversation
lfoppiano
commented
Nov 9, 2026
Member
- Converters: do not configure the root logger at import time
- JSON: skip comments and processing instructions when computing ref offsets
- JSON: keep every reference that carries any information
- JSON process_directory: pass the converter settings to parallel workers
- validate_json_refs: count files with bad references as invalid; search subdirectories
- JSON: normalise abstract text so its reference offsets are exact
- Markdown: show reference years given as attributes, and arXiv ids
- Markdown: keep table cells inside their cell and row
- Format CLI: answer --help, fix the documented commands, add tests
- CLI: fail on a missing --input; take only XML for processCitationPatentST36
- Archives: skip finished entries before reading them
- CLI: exit cleanly on Ctrl-C and say how to resume
- Read public S3 buckets without credentials: --no-sign-request
- Add end-to-end tests of the command line against a stand-in GROBID
Both converter modules called logging.basicConfig(level=INFO) when imported, and grobid_client imports TEI2LossyJSON, so any program using the library got a root handler at INFO. It also made the format CLIs' setup_logging() a no-op: basicConfig does nothing once the root logger has a handler, so --verbose and the CLI log format were ignored. TEI2LossyJSON's logger did not propagate either, so an application's handlers never received its warnings.
…fsets get_text() gives the passage its text and leaves out XML comments and processing instructions, but the offset walk counted every string node. Markup of that kind before a reference shifted its offsets, and the validation assert then aborted the conversion of the whole document.
References were kept only when they had a title, authors, a journal or one of a few identifiers. Entries GROBID describes with only an arXiv id, a URL, editors or a publisher were dropped, which left the in-text refs pointing at them without a target in references[].
The parallel worker built a default converter, so validate_refs was ignored. It also read each file with open(path, 'r') -- the locale's encoding -- before converting it, and threw the result away. Where the locale is not UTF-8 (cp1252 on Windows) that read fails on common bytes such as those of 'Á' or a closing double quote: 2 of the 9 sample documents cannot be decoded that way, and they came back as None.
…h subdirectories
A file's 'valid' flag was never cleared, so a file holding invalid
references was reported as valid ("Files: 2/2 valid") and its errors
were missing from the report; only the exit code noticed. Only the top
level of the directory was searched, while the client writes the output
of a recursive run into subdirectories. A JSON file that is not an
object, or an abstract set to null, no longer aborts the whole run.
Reference offsets are computed on whitespace-normalised text, as for the body, but an abstract passage carried the raw paragraph or sentence text. An abstract with a line break or a double space before a citation had offsets pointing a few characters off; on GROBID 0.9.2 output this hit the abstracts of 3 of 128 passages, and the sdo/ietf flavor of a ResNet paper had 4 wrong refs. Body and abstract now share one normalisation helper. At sentence level the abstract is a list of paragraphs, each a list of sentences; validate_json_refs only read flat lists, so these refs were never checked. It now reads both shapes.
GROBID usually writes a reference date as an attribute only (<date type="published" when="2019"/>). The year was read from the element text, so it was missing from nearly every reference: 50 of the 51 dates in the sample document have no text. The published date is now preferred over other dates and read from @when first. arXiv identifiers were collected but never rendered; they now appear as a link to the abstract page.
A '|' in a cell ended the cell early and a line break in a cell ended the row. Pipes are now escaped, whitespace inside a cell is collapsed, and rows are padded to the widest row, since renderers drop the cells past the separator row's column count.
`python -m grobid_client.format --help` was reported as an unknown converter. The examples in each converter's --help ran `python -m grobid_client.format.TEI2Markdown`, which imports the module and exits 0 without converting anything; they now use the menu form. Unknown-converter errors go to stderr. The format command lines had no tests; they now cover both converters to file and stdout, bad inputs, the menu, and the commands the help text documents.
…ntST36 A mistyped --input logged "Input path does not exist" and exited 0, so a script could not tell it from a run with nothing left to do; it now exits 1. Globs and s3:// inputs are still resolved later, and ~ is expanded as processing does. processCitationPatentST36 accepted PDFs as well as ST.36 XML. GROBID answers a PDF there with 204 No Content, so every PDF in the folder left an empty <name>_204.txt error marker.
process_batch skips an entry whose TEI already exists, but only once it holds the entry's bytes, so resuming an archive run read every finished entry again first. For an s3:// zip that is a download: rerunning on a public 1.3 GB corpus zip with 330 documents done spent over a minute fetching them again before reaching new work. Entries whose outputs are all there (or, with --skip_errors, that left an error marker) are now filtered out before anything is read; an entry with its TEI but a missing JSON or Markdown is still read so those get regenerated.
Interrupting a run printed a long traceback from deep inside the S3 or HTTP stack. Outputs are written atomically and finished documents are skipped on the next run, so the CLI now says that and exits with 130.
Listing and reading s3:// inputs used the default credential chain only, so a public bucket such as s3://digitalcorpora/... could not be read on a machine without AWS credentials: the run stopped with "Unable to locate credentials". --no-sign-request (s3_no_sign_request=True in the API) sends unsigned requests, as the AWS CLI option of the same name does. Listing and range reads now share one boto3 client, and a missing credentials error suggests the option.
Every other client test mocks the HTTP layer. These run grobid_client in a subprocess against a small local HTTP server that answers like GROBID, and check what lands on disk: the input tree is mirrored (they fail on the code before #126), JSON and Markdown sit next to the TEI, a failure leaves its error marker, zip and tar.gz archives behave like directories, --input-list keeps the relative layout, and re-runs skip finished documents, regenerate a missing JSON, honour --skip_errors and --force.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.