test(parity): a divergence must fail the job, not just print - #51
Merged
Merged
Conversation
The parity suite could not fail. Every comparison was recorded with addResult and rendered into the report, but nothing ever asserted on one: no t.Error, and TestMain forced a non-zero exit only when *zero* checks ran. So a real divergence printed [FAIL], lowered the percentage in the report, and exited 0. That makes the job in ci.yml untrue as described -- "Parity is this project's entire claim, so a change that breaks it fails before review rather than after a merge to main." It did not. It reported. Measured, on a tree carrying two genuine divergences (missing config made to exit 0 instead of 1): without this change 76/78, "FAILURES: 2 checks failed", exit 0 with this change 76/78, same report, exit 1 and a clean tree is 78/78 exit 0 both ways, so this tightens the gate without moving the bar. This is not hypothetical. `pre-commit run --files a b` was broken in every release while the headline read 78/78 -- the suite had no multi-path case, and even if it had, only the filesystem assertion discriminates: both binaries exit 1, one because the hook failed and one because it could not parse. Refs #43
blairham
force-pushed
the
parity-gate
branch
from
September 8, 2026 01:28
143203b to
79a2d08
Compare
The four --files comparisons added in #50 are checks like any other, so the headline and the parity table have to count them. Left stale, the number in the docs and the number the suite produces drift apart, which is the exact failure this file exists to prevent.
16 of 19 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Found while adding a regression check for #50. This is the reason that bug shipped.
The parity suite could not fail
Every comparison is recorded with
addResultand rendered into the report, but nothing ever asserts on one — not.Erroranywhere in the suite, andTestMainforces a non-zero exit only when zero checks ran. A real divergence printed[FAIL], lowered the percentage, wrote it into the JSON artifact, and exited 0.Which makes the job description in
ci.ymluntrue as written:It did not. It reported.
Measured
On a tree carrying two genuine divergences (missing config made to exit 0 instead of 1):
76/78 — FAILURES: 2 checks failed76/78 — FAILURES: 2 checks failedAnd a clean tree is
78/78exit 0 both ways — so this tightens the gate without moving the bar. Verified against real upstream 4.6.2 withPARITY_REQUIRE=1.Not hypothetical
pre-commit run --files a bhas been broken in every release while the headline read 78/78 (#50). The suite had no multi-path case — and even once one exists, only the filesystem assertion discriminates: both binaries exit 1, one because the hook failed and one because it could not parse. A gate that cannot fail plus a check that cannot discriminate is how a 100% parity claim and a broken common flag coexist.docs/parity.mdnow says a divergence fails the run, and says plainly that this was not always true.Verification
make checkclean,make lint0 issues, clean parity run exits 0 at 78/78.Refs #43