From fe4bf357d6ff3b626161f9c085df8c2b9d174b0e Mon Sep 17 00:00:00 2001 From: Mike Crowe Date: Mon, 17 Aug 2026 05:54:36 -0400 Subject: [PATCH] Split the templates out of gauntlet.md, and split "skipped layers" three ways gauntlet.md had become two documents read at different moments: how to run each layer, and the fill-in forms for SPEC and EVIDENCE. The file is loaded in full either way. `references/templates.md` now holds the Gherkin scenario template, the SPEC template, the EVIDENCE template, and the tracker roll-up; gauntlet.md keeps a pointer and drops 74 lines. Four fields come with it, each earning its place by the same test -- its absence lets a reader believe something false: - "Skipped layers" becomes "Layers not run as specified", split into N-A / UNAVAILABLE / SUBSTITUTED. One list collapses "no such surface here", "tool missing, nothing ran", and "something else ran, here is its blind spot" into one word. Those are three different confidence claims that currently read identically. - A Dismissed-findings section, one line each naming the command or file:line that disproves the finding. A fix carries its own evidence; a dismissal carries none, so "not a real problem" and "did not check" are indistinguishable. - A Structural-blind-spot line, naming the layer this project cannot run at all -- otherwise it reads as absent rather than accepted. - A `## Revisions` section in SPEC. The spec is required to be append-only and revised visibly, and there was nowhere to write the revision. The demo's evidence.md carried the exact conflation this fixes: mutmut replaced by tools/mutants.py (SUBSTITUTED) and shellcheck never run (UNAVAILABLE) sat under one "Skipped layers" heading, reading as the same claim. Relabelled; no numbers change, and evidence.md is not in source_state.sh's hashed set, so the recorded tree hash still holds. Not included, per the review on #10: the per-layer wall-clock column, and the Config header line that depended on the deferred config format. Co-Authored-By: Claude Opus 5 --- demo-rate-limiter/evidence.md | 18 ++- skills/old-coder/SKILL.md | 5 +- skills/old-coder/references/gauntlet.md | 78 +-------- skills/old-coder/references/templates.md | 195 +++++++++++++++++++++++ 4 files changed, 214 insertions(+), 82 deletions(-) create mode 100644 skills/old-coder/references/templates.md diff --git a/demo-rate-limiter/evidence.md b/demo-rate-limiter/evidence.md index 57255b1..a646c42 100644 --- a/demo-rate-limiter/evidence.md +++ b/demo-rate-limiter/evidence.md @@ -92,13 +92,17 @@ Status legend: pass / fail / unverified / n-a. - Scenario suite alone: **22/22**. The headline mutation score is carried entirely by the scenario tests. -## Skipped layers - -- Tool-based mutation (mutmut): unverified compatibility with Python 3.14; - replaced with the scripted manual procedure (`tools/mutants.py`, 22 mutants). -- Shell lint (shellcheck) for the four scripts that implement half the gates: - **not run**, no tool installed. Every Python file gets three static layers - and the shell gets none. Known gap, raised by verification round 4. +## Layers not run as specified + +- **SUBSTITUTED — tool-based mutation (mutmut):** unverified compatibility with + Python 3.14; the scripted manual procedure ran instead (`tools/mutants.py`, + 22 mutants). What it cannot detect: the mutant list is hand-written, so + unlike a tool generating mutants from the syntax tree it can only test + weaknesses somebody thought of in advance. +- **UNAVAILABLE — shell lint (shellcheck)** for the four scripts that implement + half the gates: no tool installed, and nothing ran in its place. Every Python + file gets three static layers and the shell gets none. Known gap, raised by + verification round 4. ## Independent verification diff --git a/skills/old-coder/SKILL.md b/skills/old-coder/SKILL.md index a2b1351..1f0c33e 100644 --- a/skills/old-coder/SKILL.md +++ b/skills/old-coder/SKILL.md @@ -71,7 +71,8 @@ implementation files: wrong, say so explicitly and revise it visibly — never silently drift. - **Write the spec to a file and name it by absolute path.** A relative path is not clickable in a terminal, so the human cannot open the one artifact they - are being asked to approve. Same for EVIDENCE when you get there. + are being asked to approve. Same for EVIDENCE when you get there. The SPEC + and Gherkin templates are in `references/templates.md`. ### 2. RED — prove each test can fail @@ -175,7 +176,7 @@ such excuse: you chose them, so choose real bugs. ### 6. EVIDENCE — the only thing the human reads after code End with a report the human can trust without opening a single source file -(template in `references/gauntlet.md`): +(template in `references/templates.md`): - The approved spec, with each behavior mapped to the test that verifies it. - Each gauntlet layer: the command run, and its actual result (pasted numbers, diff --git a/skills/old-coder/references/gauntlet.md b/skills/old-coder/references/gauntlet.md index 4d06fd1..84fe7f5 100644 --- a/skills/old-coder/references/gauntlet.md +++ b/skills/old-coder/references/gauntlet.md @@ -183,77 +183,9 @@ into a vacuous pass. Prove each home-grown check can fail with a one-off negative control (feed it a known-bad fixture; make its input unreadable) and record the control in EVIDENCE's honest notes. -## Gherkin scenario template (for the SPEC step) - -```gherkin -Feature: - Scenario: - Given - When - Then - - Scenario: - Given ... - When - Then -``` - -Each scenario maps 1:1 to at least one automated test; name the test after the -scenario so the evidence report's spec→test mapping is mechanical. - -## Evidence report template (for the EVIDENCE step) - -```markdown -## Evidence Report — (Tier <1|2|3>) - -- Spec approval: -- Source state: — persist the - computation as a script (e.g. tools/source_state.sh); a hash recipe written - in prose is working-directory-sensitive and will fail to reproduce -- Toolchain: -- Entry point: -- Independent verification: - **against the final source state** — a state no verifier saw is - `not performed` however many rounds preceded it (Tier 3; protocol in - `verifier.md`) - -### Spec → Test mapping -Status is one of: **pass / fail / unverified / n-a**. A row mapped to -"skipped: " must carry unverified or n-a — never pass. - -| Scenario | Test | Status | -|---|---|---| -| | :: | pass | -| Must NOT: | | pass \| unverified | +## Templates -### Gauntlet (final fresh run) -| Layer | Command | Result | -|---|---|---| -| Tests | | passed, 0 failed | -| Types | | 0 errors | -| Lint | | 0 warnings | -| Changed-line coverage | | / changed lines (list any misses) | -| Mutation | | / killed | -| Property-based | | properties, examples each | -| Real execution | | | -| Supply chain | | 0 known vulns; new deps: none (or list, each ↔ SPEC justification) | -| Suite health | | randomized order (seed ), all passed | - -### Independent verification (never omit; see verifier.md) -- Verifier: ; fresh context; which inputs it received; - what correlation that breaks and what it does not. -- Rounds: (cap ); verdict per round, each against the state it saw. -- Grading: who classified each finding behavioural vs description, and who - approved stopping. -- Attacked: . -- Findings: behavioural (fixed, then re-verified in a new context) vs - description/mapping (fixed and disclosed, no new round). -- Fixed after the last verified state, therefore unverified: . - -### Skipped layers -- : (or "none") - -### Honest notes -- -``` +The Gherkin scenario template, the SPEC template, the EVIDENCE report template, +and the tracker roll-up live in `references/templates.md`. This file is read +while building the gauntlet; that one is read while writing the two artifacts +the human reads. diff --git a/skills/old-coder/references/templates.md b/skills/old-coder/references/templates.md new file mode 100644 index 0000000..43d08b1 --- /dev/null +++ b/skills/old-coder/references/templates.md @@ -0,0 +1,195 @@ +# SPEC and EVIDENCE Templates + +The fill-in forms for the two artifacts the human actually reads. The layers +they reference are defined in `gauntlet.md`. + +## SPEC template (for the SPEC step) + +Written before any implementation file is touched. + +```markdown +# SPEC — + +- Tier: <1|2|3> +- Issue: +- Setup plan: + - Tools to install: + - Git: + - Files the gauntlet will add, **by path**: `tools/mutants.py` (mutation + layer), `tools/gauntlet.sh` (entry point) — mark either "already exists, + reused" — plus any fixture or harness file + - New dependencies: + +## Scenarios + + +## Must NOT +- + +## Revisions +- +``` + +`## Revisions` is where "the spec is append-only, revise it visibly" stops being +a promise. Without somewhere to write the revision, a spec that changed +mid-task and a spec that never changed are the same document, and a reader +cannot tell which one they are holding. + +Commit `SPEC.md` at approval where the repo's git conventions allow it — the +setup plan is where that was authorized. Once the approved spec is a commit, +later drift is literally a `git diff`. Without a durable spec, a compaction +loses the approved contract while the code it authorized remains, and nobody +can check whether a scenario was quietly dropped from the EVIDENCE mapping. + +**If the human rejects the spec**, revise `SPEC.md` in place, add the reason to +`## Revisions`, and re-request approval. Do not delete it and start clean — +what the human turned down, and why, is the most useful thing in the file. + +### Gherkin scenario template + +```gherkin +Feature: + Scenario: + Given + When + Then + + Scenario: + Given ... + When + Then +``` + +Each scenario maps 1:1 to at least one automated test; name the test after the +scenario so the evidence report's spec→test mapping is mechanical. + +## Evidence report template (for the EVIDENCE step) + +```markdown +## Evidence Report — (Tier <1|2|3>) + +- Spec approval: +- Source state: — persist the + computation as a script (e.g. tools/source_state.sh); a hash recipe written + in prose is working-directory-sensitive and will fail to reproduce +- Toolchain: +- Entry point: +- Independent verification: + **against the final source state** — a state no verifier saw is + `not performed` however many rounds preceded it (Tier 3; protocol in + `verifier.md`) +- Tracker: + +### Spec → Test mapping +Status is one of: **pass / fail / unverified / n-a**. A row mapped to +"skipped: " must carry unverified or n-a — never pass. + +| Scenario | Test | Status | +|---|---|---| +| | :: | pass | +| Must NOT: | | pass \| unverified | + +### Gauntlet (final fresh run) +| Layer | Command | Result | +|---|---|---| +| Tests | | passed, 0 failed | +| Types | | 0 errors | +| Lint | | 0 warnings | +| Changed-line coverage | | / changed lines (list any misses) | +| Mutation | | / killed | +| Property-based | | properties, examples each | +| Real execution | | | +| Supply chain | | 0 known vulns; new deps: none (or list, each ↔ SPEC justification) | +| Suite health | | randomized order (seed ), all passed | + +### Independent verification (never omit; see verifier.md) +- Verifier: ; fresh context; which inputs it received; + what correlation that breaks and what it does not. +- Rounds: (cap ); verdict per round, each against the state it saw. +- Grading: who classified each finding behavioural vs description, and who + approved stopping. +- Attacked: . +- Findings: behavioural (fixed, then re-verified in a new context) vs + description/mapping (fixed and disclosed, no new round). +- Fixed after the last verified state, therefore unverified: . + +### Layers not run as specified +Split by status, because they mean different things to a reader: +- **N-A (this project has no such surface):** +- **UNAVAILABLE (tool missing):** +- **SUBSTITUTED:** +- (or "none") + +### Dismissed review findings +Fixes are self-evidencing; dismissals are not. One line each: +- — dismissed because . +- (or "none — every finding was fixed or accepted as a known limit") + +### Structural blind spot +- + +### Honest notes +- +``` + +**Why "layers not run" is split three ways.** One "skipped" list collapses +three states a reader has to tell apart: there is no such surface in this +project, versus the surface exists but the tool was missing and nothing ran, +versus something else ran and here is what it cannot detect. Those are very +different confidence claims and they read identically as "skipped". The third +is the dangerous one: `SUBSTITUTED` may never be written as a pass. Two repeat +runs in place of randomized order is not "suite health: stable" — it is +`SUBSTITUTED (2 repeat runs — cannot detect whole-suite order dependence)`. A +reader who cannot tell a substitute from the real layer reads "found nothing" +where the truth is "did not look with that instrument". `N-A` is not a +degraded run at all: three `N-A` layers describe the project, and EVIDENCE +should say so rather than leaving a reader to count absences. + +**Why dismissals need a line each.** A fix carries its own evidence — the test +that now passes. A dismissal carries none: "not a real problem" is +indistinguishable from "did not check". Naming the command, `file:line`, or +test that disproves the finding is the same rule the verification protocol +already applies to attack lists — say what you tried, not only what you found. + +**Why name the blind spot.** A layer a project cannot run at all otherwise +reads as absent rather than accepted. Stating it converts a silent gap into a +known limit the reader can price in. + +## Tracker roll-up (only when the SPEC names an issue) + +Tracker linkage is **bidirectional and tracker-agnostic**: the SPEC header +carries an issue ID, and on completion a short roll-up goes back to that issue. +No hard dependency on any particular tracker — if the SPEC's `Issue` field says +`none`, this step does not exist. + +Write it to `ROLLUP.md` alongside the spec and evidence files. Post it to the +issue only if an approver says so; with nobody present, leave it on disk and +say so in EVIDENCE. A hosted tracker notifies people and cannot be un-sent, so +it gets the same gate as a commit. + +```markdown +- Built: +- Left undone: +- Traps for the next task: +- Evidence: +``` + +**Why it is short, and why it is not a copy of EVIDENCE.** The two serve +different readers, and keeping them distinct is what stops them drifting into +rival sources of truth: + +| | Reader | Obligation | +|---|---|---| +| EVIDENCE | the human reviewing **this** change | complete — every layer, every number, every limit | +| Tracker roll-up | whoever picks up the **next** task | short — what changed the ground under them, and where to find the rest | + +A tracker whose notes are append-only by API gives the spec's no-silent-drift +property for free: an earlier note cannot be quietly rewritten to match a later +story. Where the tracker permits editing, that property is not there and the +git-commit-at-SPEC mechanism remains the enforcement — the roll-up is a +convenience for the next reader, never the authoritative record.