story-015: A test that cannot fail must not count as validation

[jerodw]

Summary

Five stories shipped "X is unchanged" tests that stopped checking anything the moment their own story merged. Each asserted git diff HEAD -- <paths> is empty — and on a committed tree that diff is empty for every path, so the assertion held no matter what the story did.

Demonstrated rather than argued: story-009's test_the_definitions_this_story_injects_are_unchanged asserted schemas/ was unchanged, and passed on a branch that added schemas/manifest.json.

The idiom had been accumulating since story-007 and was written five times by five separate tester runs — once after the rule against it was recorded in .harness/docs/ARCHITECTURE.md and injected into every stage. Prose guidance was already tried here and did not hold. This story therefore ships a mechanical check as well as guidance, and repairs the four instances already on main.

Changes

• tests/conftest.py — one shared baseline resolution, story_commit_range(validation_file, repo), resolving a story's run commit as the commit that added its validation file and the baseline as that commit's parent. Bounded at both ends, so it survives a rebase or a squash, stays correct as later commits stack on top, and degrades to the working tree while a story is still in flight. It raises rather than returning a range that cannot differ.
• tests/test_baseline_honesty.py (new) — an AST scan over every module in tests/, flagging a git invocation that targets the repository root and carries a HEAD-derived revision or a working-tree status query. Exactly one module is exempt, by name, and it is the one holding the shared resolution.
• tests/test_story_007_validation.py, …008…, …009…, …010… — the four merged instances repointed at the shared resolution. Each keeps the subject and strictness it was written with, with one authorized exception: the two test_no_committed_story_artifact_was_edited assertions are narrowed to modifications and deletions, because each story's own artifact was added in its own run commit and an addition was never an edit.
• prompts/tester.md, prompts/verifier.md — the general rule the mechanical check cannot cover: an assertion claiming an absence needs a negative control, because it "passes when the property holds and it passes just as happily when the test is looking in the wrong place." Positive assertions fail loudly on their own and need nothing extra. The verifier gets the corresponding requirement.
• .harness/docs/ARCHITECTURE.md — the resolution, the check, and what the check does and does not cover.

Testing

685 passed on Python 3.14 and on Python 3.10.

The verifier did not accept the repairs on description. Its decisive check was to clone the repository, commit all nine changed files as a single run commit, and re-verify there — since the story's entire subject is assertions that change behavior once the story commits:

• Ranges are live. In the post-commit clone, each repaired file resolved to its own story's real run commit against that commit's parent — story-007 → 06eaa477 (11 files changed), story-008 → 14f47eac (7 files), story-009 → ff808499 (26 files). The ranges report real changes; the guarded paths are empty because those stories genuinely left them alone, not because the comparison is degenerate.
• The regression set is caught. Each of the four pre-repair files yields exactly 1 flag before repair and 0 after, plus story-013's archived copy under .harness/runs-archive/.
• The check itself can fail. Appending subprocess.run(['git', '-C', str(REPO_ROOT), 'diff', 'HEAD']) to tests/test_story_011_validation.py turned test_no_module_in_the_suite_resolves_a_dishonest_baseline red; restoring the file turned it green.

Notes for review

• This is the first story the clean-clone check governed, and it passed. clean-clone-result.json records exit_code: 0, 685 passed, on python_version: 3.10.20 — the suite run in a fresh clone with the story committed into it, on the oldest Python CI tests. I had predicted this story might trip it, since it is repairing exactly the class of defect the check exists to catch. It did not, and the verifier's independent post-commit clone agrees.
• The mechanical check is deliberately narrow, and the verifier named the gap. It recognizes repository-root targeting only through -C <root> or cwd=<root>, which is what the acceptance criterion enumerates. A git call written with neither — subprocess.run(['git', 'diff', 'HEAD', ...]) — inherits the process working directory and would not be flagged. Worth knowing before relying on it as a complete guard.
• One thing that cannot be exercised yet: whether the four repaired assertions stay non-vacuous for future stories that legitimately edit the guarded paths. That depends on commits that do not exist. The bounding at both ends makes it structurally independent of later commits and the synthetic-history tests exercise the mechanism, but no real future commit has tested it.
• The implementer created tests/test_baseline_honesty.py under the story's stage_exceptions grant — the check is this story's deliverable, not validation of it. Independence is preserved: the tester wrote tests/test_story_015_validation.py separately and validates the check, the shared resolution, and the four repairs.

🤖 Generated with Claude Code