diff --git a/demo-rate-limiter/evidence.md b/demo-rate-limiter/evidence.md index 57255b1..a646c42 100644 --- a/demo-rate-limiter/evidence.md +++ b/demo-rate-limiter/evidence.md @@ -92,13 +92,17 @@ Status legend: pass / fail / unverified / n-a. - Scenario suite alone: **22/22**. The headline mutation score is carried entirely by the scenario tests. -## Skipped layers - -- Tool-based mutation (mutmut): unverified compatibility with Python 3.14; - replaced with the scripted manual procedure (`tools/mutants.py`, 22 mutants). -- Shell lint (shellcheck) for the four scripts that implement half the gates: - **not run**, no tool installed. Every Python file gets three static layers - and the shell gets none. Known gap, raised by verification round 4. +## Layers not run as specified + +- **SUBSTITUTED — tool-based mutation (mutmut):** unverified compatibility with + Python 3.14; the scripted manual procedure ran instead (`tools/mutants.py`, + 22 mutants). What it cannot detect: the mutant list is hand-written, so + unlike a tool generating mutants from the syntax tree it can only test + weaknesses somebody thought of in advance. +- **UNAVAILABLE — shell lint (shellcheck)** for the four scripts that implement + half the gates: no tool installed, and nothing ran in its place. Every Python + file gets three static layers and the shell gets none. Known gap, raised by + verification round 4. ## Independent verification diff --git a/skills/old-coder/SKILL.md b/skills/old-coder/SKILL.md index a2b1351..1f0c33e 100644 --- a/skills/old-coder/SKILL.md +++ b/skills/old-coder/SKILL.md @@ -71,7 +71,8 @@ implementation files: wrong, say so explicitly and revise it visibly — never silently drift. - **Write the spec to a file and name it by absolute path.** A relative path is not clickable in a terminal, so the human cannot open the one artifact they - are being asked to approve. Same for EVIDENCE when you get there. + are being asked to approve. Same for EVIDENCE when you get there. The SPEC + and Gherkin templates are in `references/templates.md`. ### 2. RED — prove each test can fail @@ -175,7 +176,7 @@ such excuse: you chose them, so choose real bugs. ### 6. EVIDENCE — the only thing the human reads after code End with a report the human can trust without opening a single source file -(template in `references/gauntlet.md`): +(template in `references/templates.md`): - The approved spec, with each behavior mapped to the test that verifies it. - Each gauntlet layer: the command run, and its actual result (pasted numbers, diff --git a/skills/old-coder/references/gauntlet.md b/skills/old-coder/references/gauntlet.md index 4d06fd1..84fe7f5 100644 --- a/skills/old-coder/references/gauntlet.md +++ b/skills/old-coder/references/gauntlet.md @@ -183,77 +183,9 @@ into a vacuous pass. Prove each home-grown check can fail with a one-off negative control (feed it a known-bad fixture; make its input unreadable) and record the control in EVIDENCE's honest notes. -## Gherkin scenario template (for the SPEC step) - -```gherkin -Feature: - Scenario: - Given - When - Then - - Scenario: - Given ... - When - Then -``` - -Each scenario maps 1:1 to at least one automated test; name the test after the -scenario so the evidence report's spec→test mapping is mechanical. - -## Evidence report template (for the EVIDENCE step) - -```markdown -## Evidence Report — (Tier <1|2|3>) - -- Spec approval: -- Source state: — persist the - computation as a script (e.g. tools/source_state.sh); a hash recipe written - in prose is working-directory-sensitive and will fail to reproduce -- Toolchain: -- Entry point: -- Independent verification: - **against the final source state** — a state no verifier saw is - `not performed` however many rounds preceded it (Tier 3; protocol in - `verifier.md`) - -### Spec → Test mapping -Status is one of: **pass / fail / unverified / n-a**. A row mapped to -"skipped: " must carry unverified or n-a — never pass. - -| Scenario | Test | Status | -|---|---|---| -| | :: | pass | -| Must NOT: | | pass \| unverified | +## Templates -### Gauntlet (final fresh run) -| Layer | Command | Result | -|---|---|---| -| Tests | | passed, 0 failed | -| Types | | 0 errors | -| Lint | | 0 warnings | -| Changed-line coverage | | / changed lines (list any misses) | -| Mutation | | / killed | -| Property-based | | properties, examples each | -| Real execution | | | -| Supply chain | | 0 known vulns; new deps: none (or list, each ↔ SPEC justification) | -| Suite health | | randomized order (seed ), all passed | - -### Independent verification (never omit; see verifier.md) -- Verifier: ; fresh context; which inputs it received; - what correlation that breaks and what it does not. -- Rounds: (cap ); verdict per round, each against the state it saw. -- Grading: who classified each finding behavioural vs description, and who - approved stopping. -- Attacked: . -- Findings: behavioural (fixed, then re-verified in a new context) vs - description/mapping (fixed and disclosed, no new round). -- Fixed after the last verified state, therefore unverified: . - -### Skipped layers -- : (or "none") - -### Honest notes -- -``` +The Gherkin scenario template, the SPEC template, the EVIDENCE report template, +and the tracker roll-up live in `references/templates.md`. This file is read +while building the gauntlet; that one is read while writing the two artifacts +the human reads. diff --git a/skills/old-coder/references/templates.md b/skills/old-coder/references/templates.md new file mode 100644 index 0000000..43d08b1 --- /dev/null +++ b/skills/old-coder/references/templates.md @@ -0,0 +1,195 @@ +# SPEC and EVIDENCE Templates + +The fill-in forms for the two artifacts the human actually reads. The layers +they reference are defined in `gauntlet.md`. + +## SPEC template (for the SPEC step) + +Written before any implementation file is touched. + +```markdown +# SPEC — + +- Tier: <1|2|3> +- Issue: +- Setup plan: + - Tools to install: + - Git: + - Files the gauntlet will add, **by path**: `tools/mutants.py` (mutation + layer), `tools/gauntlet.sh` (entry point) — mark either "already exists, + reused" — plus any fixture or harness file + - New dependencies: + +## Scenarios + + +## Must NOT +- + +## Revisions +- +``` + +`## Revisions` is where "the spec is append-only, revise it visibly" stops being +a promise. Without somewhere to write the revision, a spec that changed +mid-task and a spec that never changed are the same document, and a reader +cannot tell which one they are holding. + +Commit `SPEC.md` at approval where the repo's git conventions allow it — the +setup plan is where that was authorized. Once the approved spec is a commit, +later drift is literally a `git diff`. Without a durable spec, a compaction +loses the approved contract while the code it authorized remains, and nobody +can check whether a scenario was quietly dropped from the EVIDENCE mapping. + +**If the human rejects the spec**, revise `SPEC.md` in place, add the reason to +`## Revisions`, and re-request approval. Do not delete it and start clean — +what the human turned down, and why, is the most useful thing in the file. + +### Gherkin scenario template + +```gherkin +Feature: + Scenario: + Given + When + Then + + Scenario: + Given ... + When + Then +``` + +Each scenario maps 1:1 to at least one automated test; name the test after the +scenario so the evidence report's spec→test mapping is mechanical. + +## Evidence report template (for the EVIDENCE step) + +```markdown +## Evidence Report — (Tier <1|2|3>) + +- Spec approval: +- Source state: — persist the + computation as a script (e.g. tools/source_state.sh); a hash recipe written + in prose is working-directory-sensitive and will fail to reproduce +- Toolchain: +- Entry point: +- Independent verification: + **against the final source state** — a state no verifier saw is + `not performed` however many rounds preceded it (Tier 3; protocol in + `verifier.md`) +- Tracker: + +### Spec → Test mapping +Status is one of: **pass / fail / unverified / n-a**. A row mapped to +"skipped: " must carry unverified or n-a — never pass. + +| Scenario | Test | Status | +|---|---|---| +| | :: | pass | +| Must NOT: | | pass \| unverified | + +### Gauntlet (final fresh run) +| Layer | Command | Result | +|---|---|---| +| Tests | | passed, 0 failed | +| Types | | 0 errors | +| Lint | | 0 warnings | +| Changed-line coverage | | / changed lines (list any misses) | +| Mutation | | / killed | +| Property-based | | properties, examples each | +| Real execution | | | +| Supply chain | | 0 known vulns; new deps: none (or list, each ↔ SPEC justification) | +| Suite health | | randomized order (seed ), all passed | + +### Independent verification (never omit; see verifier.md) +- Verifier: ; fresh context; which inputs it received; + what correlation that breaks and what it does not. +- Rounds: (cap ); verdict per round, each against the state it saw. +- Grading: who classified each finding behavioural vs description, and who + approved stopping. +- Attacked: . +- Findings: behavioural (fixed, then re-verified in a new context) vs + description/mapping (fixed and disclosed, no new round). +- Fixed after the last verified state, therefore unverified: . + +### Layers not run as specified +Split by status, because they mean different things to a reader: +- **N-A (this project has no such surface):** +- **UNAVAILABLE (tool missing):** +- **SUBSTITUTED:** +- (or "none") + +### Dismissed review findings +Fixes are self-evidencing; dismissals are not. One line each: +- — dismissed because . +- (or "none — every finding was fixed or accepted as a known limit") + +### Structural blind spot +- + +### Honest notes +- +``` + +**Why "layers not run" is split three ways.** One "skipped" list collapses +three states a reader has to tell apart: there is no such surface in this +project, versus the surface exists but the tool was missing and nothing ran, +versus something else ran and here is what it cannot detect. Those are very +different confidence claims and they read identically as "skipped". The third +is the dangerous one: `SUBSTITUTED` may never be written as a pass. Two repeat +runs in place of randomized order is not "suite health: stable" — it is +`SUBSTITUTED (2 repeat runs — cannot detect whole-suite order dependence)`. A +reader who cannot tell a substitute from the real layer reads "found nothing" +where the truth is "did not look with that instrument". `N-A` is not a +degraded run at all: three `N-A` layers describe the project, and EVIDENCE +should say so rather than leaving a reader to count absences. + +**Why dismissals need a line each.** A fix carries its own evidence — the test +that now passes. A dismissal carries none: "not a real problem" is +indistinguishable from "did not check". Naming the command, `file:line`, or +test that disproves the finding is the same rule the verification protocol +already applies to attack lists — say what you tried, not only what you found. + +**Why name the blind spot.** A layer a project cannot run at all otherwise +reads as absent rather than accepted. Stating it converts a silent gap into a +known limit the reader can price in. + +## Tracker roll-up (only when the SPEC names an issue) + +Tracker linkage is **bidirectional and tracker-agnostic**: the SPEC header +carries an issue ID, and on completion a short roll-up goes back to that issue. +No hard dependency on any particular tracker — if the SPEC's `Issue` field says +`none`, this step does not exist. + +Write it to `ROLLUP.md` alongside the spec and evidence files. Post it to the +issue only if an approver says so; with nobody present, leave it on disk and +say so in EVIDENCE. A hosted tracker notifies people and cannot be un-sent, so +it gets the same gate as a commit. + +```markdown +- Built: +- Left undone: +- Traps for the next task: +- Evidence: +``` + +**Why it is short, and why it is not a copy of EVIDENCE.** The two serve +different readers, and keeping them distinct is what stops them drifting into +rival sources of truth: + +| | Reader | Obligation | +|---|---|---| +| EVIDENCE | the human reviewing **this** change | complete — every layer, every number, every limit | +| Tracker roll-up | whoever picks up the **next** task | short — what changed the ground under them, and where to find the rest | + +A tracker whose notes are append-only by API gives the spec's no-silent-drift +property for free: an earlier note cannot be quietly rewritten to match a later +story. Where the tracker permits editing, that property is not there and the +git-commit-at-SPEC mechanism remains the enforcement — the roll-up is a +convenience for the next reader, never the authoritative record.