Fail closed when a gauntlet layer never runs

demo-rate-limiter/evidence.md

@@ -1,24 +1,24 @@
 # Evidence Report — Sliding-Window Rate Limiter (Tier 3)
 
-- Spec approval: **obtained** for REVISION 4 (2026-08-09), REVISION 5 and
-  REVISION 6 (2026-08-18) — the human approved each contract change before
-  implementation. Earlier revisions (2026-07-25, 2026-07-27) were autonomous
-  and are still unapproved; treat them as the weaker part of the spec.
+- Spec approval: **obtained** for REVISION 4 (2026-08-09) and REVISION 5–7
+  (2026-08-18) — the human approved each contract change before implementation.
+  Earlier revisions (2026-07-25, 2026-07-27) were autonomous and are still
+  unapproved; treat them as the weaker part of the spec.
 - Independent verification: **not performed against the final source state
-  `4734451`.** Six earlier rounds were performed; the last verified state
+  `42528d9`.** Six earlier rounds were performed; the last verified state
   `d0b506c` returned `failed`, and the fixes made since — one of them
   behavioural — are disclosed below as unverified. This report is finalized as
   a **declared downgrade**, not on the strength of a passing verdict. A
   verdict attaches to the state a verifier actually saw, and no verifier has
   seen this one.
-- Source state: source commit `4734451`; sha256 tree hash
-  `dbccc212daa35442` — reproduce both with `./tools/source_state.sh` from any
+- Source state: source commit `42528d9`; sha256 tree hash
+  `5aa96ec5487c957c` — reproduce both with `./tools/source_state.sh` from any
   directory. When a binding is produced the tree hash is the required content
   identity; the source commit is provenance and is supplied only where
   complete history is available, so a shallow checkout reports
   `(unavailable: shallow history)` and a no-Git archive reports `(no git)`,
   both alongside this same tree hash. No error path emits a binding at all.
-  The script separately reports current HEAD; commits after `4734451` that
+  The script separately reports current HEAD; commits after `42528d9` that
   touch only this report or other out-of-scope paths preserve the source
   commit and tree hash. The manifest includes `.github/workflows`, which
   decides whether the gauntlet runs in CI at all.
@@ -27,13 +27,14 @@
 - Entry point: `./tools/gauntlet.sh` reruns every layer below.
 
 All numbers are from one final fresh run of the entry point, executed
-2026-08-18 at source commit `4734451` after the last code edit.
+2026-08-18 at source commit `42528d9` after the last code edit.
 
 `spec.md` was deliberately pruned back to a contract before REVISION 5
 (339 → 255 lines). Every clause, invariant, obligation and failure-model row
 survived; what was removed is the per-revision forensics, which lives in the
-honest notes below and in git. REVISION 5 adds the approved source-binding
-contract and tests without changing rate-limiter behaviour.
+honest notes below and in git. REVISION 5–7 add the approved source-binding,
+provenance and gauntlet-orchestration contracts without changing rate-limiter
+behaviour.
 
 ## Spec → Test mapping
 
@@ -72,12 +73,14 @@ Status legend: pass / fail / unverified / n-a.
 | REVISION 5: source binding is reproducible and fail-closed | test_source_state.py (ignored artifacts, staged/unstaged/untracked/deleted inputs, clean clone, no-Git archive, arbitrary cwd, evidence-only commit) | pass |
 | REVISION 6: truncated history withholds provenance, never invents it | test_source_state.py::test_shallow_history_withholds_provenance (exact marker, shallow HEAD, tree equal to the full clone, empty stderr) | pass |
 | REVISION 6: covered error scenarios pin their reason and emit no binding | test_source_state.py (Git dirty, Git deletion, Git untracked, no-Git missing input, no-Git empty scope — each asserts the reason and `stdout == ""`) | pass |
+| REVISION 7: omitted or failed gauntlet work cannot report green | test_gauntlet_orchestration.sh (omitted layer, failing command with exact rc and stopped sentinel, unknown layer, duplicate layer, complete-manifest positive control) | pass |
 
 ## Gauntlet (final fresh run: `./tools/gauntlet.sh`)
 
 | Layer | Command | Result |
 |---|---|---|
-| Checker self-test | `sh tools/test_gauntlet_checks.sh` (first layer; asserts the must-not scan fails on a planted pattern, passes on a clean tree, and fails closed with a distinct rc 2 when the scan itself breaks) | 3/3 expectations ok |
+| Orchestration self-test | `sh tools/test_gauntlet_orchestration.sh` (omitted layer, command failure, unknown layer, duplicate layer, and complete-manifest positive control against the real helper) | 5/5 scenarios, 13/13 expectations ok |
+| Checker self-test | `sh tools/test_gauntlet_checks.sh` (asserts the must-not scan fails on a planted pattern, passes on a clean tree, and fails closed with a distinct rc 2 when the scan itself breaks) | 3/3 expectations ok |
 | Source-state self-test | `pytest -q tests/test_source_state.py` (negative controls for the covered fail-closed scenarios; shallow/full-history, clean clone and no-Git archive comparisons) | 9/9 passed |
 | Mutation harness negative control | `python tools/mutants.py --negative-control` (a killer and a strictly-equivalent mutant of identical size under one pinned mtime) | C1 KILLED, C2 SURVIVED — ok |
 | Tests | `pytest -q --cov=ratelimiter` | 50 passed, 0 failed |
@@ -89,7 +92,7 @@ Status legend: pass / fail / unverified / n-a.
 | Real execution | `python examples/demo.py` (real `time.monotonic`) | burst of 5 → `[True, True, True, False, False]`; other key unaffected; allowed again after window |
 | Supply chain | `pip-audit -r requirements-dev.txt` | no known vulnerabilities; runtime dependencies: **none** (stdlib only; `threading` is stdlib) |
 | Secret scan | must-not scan in `tools/gauntlet.sh` over src, tests, tools, examples, spec.md, pyproject.toml, requirements-dev.txt and `../.github` | clean, no matches |
-| Source binding | `tools/source_state.sh` (last gauntlet layer) | source commit `4734451`; tree `dbccc212daa35442`; current HEAD is reported separately |
+| Source binding | `tools/source_state.sh` (last gauntlet layer) | source commit `42528d9`; tree `5aa96ec5487c957c`; current HEAD is reported separately |
 | License check | — | n-a: zero runtime dependencies, nothing redistributed beyond this repo's own MIT code |
 | Suite health | pytest-randomly (order shuffled every run) | 50 passed in randomized order, 10/10 consecutive runs |
 
@@ -109,7 +112,7 @@ Status legend: pass / fail / unverified / n-a.
   22 mutants). What it cannot detect: the mutant list is hand-written, so
   unlike a tool generating mutants from the syntax tree it can only test
   weaknesses somebody thought of in advance.
-- **UNAVAILABLE — shell lint (shellcheck)** for the four scripts that implement
+- **UNAVAILABLE — shell lint (shellcheck)** for the six scripts that implement
   half the gates: no tool installed, and nothing ran in its place. Every Python
   file gets three static layers and the shell gets none. Known gap, raised by
   verification round 4.
@@ -162,10 +165,42 @@ independently verified**:
 - REVISION 5 and the reproducible, fail-closed source-state mechanism in
   commits `86bfcf4` and `d45cc2f`;
 - REVISION 6 and the shallow-history provenance repair in commits `3e45e16`
-  and `49e8762`, plus the historical CI wording correction in `4734451`.
+  and `49e8762`, plus the historical CI wording correction in `4734451`;
+- REVISION 7 and the fail-closed gauntlet orchestration in commits `5b8dc1b`
+  and `42528d9`.
 
 ## Honest notes
 
+- **The gauntlet previously authenticated headings, not completed work.** The
+  issue investigation removed the committed mutation invocation while keeping
+  its heading; on a clean tree the entry point ran no mutant, exited zero and
+  printed `all layers green`. The RED implementation reproduced that class of
+  failure with a deliberately fail-open helper: 8 of 13 expectations failed,
+  including an omitted `mutation` layer returning zero and printing green.
+  REVISION 7 puts the expected manifest in a real sourced helper, couples each
+  command to completion through `run_layer`, and makes the all-green message a
+  result of the final audit. Its five-scenario control now passes 13/13
+  expectations. This catches an accidental omission of a registered layer
+  invocation; it does not make a coordinated edit to both the manifest and
+  caller impossible.
+
+- **One REVISION 7 control was shallower than the four beside it.** Scenario 5
+  of `test_gauntlet_orchestration.sh` placed `2>&1` outside the command
+  substitution, so the subshell's stderr leaked to the terminal rather than
+  being captured. It asserts only stdout, so no expectation was wrong; but a
+  regression printing the all-green message while also complaining about a
+  missing or failed layer on stderr would have passed that scenario alone.
+  Found in review, not by the control. Fixed in `42528d9`; the five scenarios
+  still pass 13/13 and the fail-open mutation battery still turns them red. The
+  contract in REVISION 7 is unchanged, so this carries no new spec revision —
+  only a rebind, because `tools/` is inside the hashed source scope.
+
+- **REVISION 7 keeps application and instrument assurance distinct.** The
+  coverage and mutation targets remain `src/ratelimiter`; they were not widened
+  across `tools/`. Home-grown trust-chain tools instead carry targeted negative
+  controls for demonstrated fail-open modes. Those controls prove their named
+  cases only, not the correctness of every path through each tool.
+
 - **The first REVISION 6 evidence rebind was internally stale.** Commit
   `83004bf` updated the headline source state but left four REVISION 5 values in
   the gauntlet table: 6 rather than 9 source-state tests, 47 rather than 50

demo-rate-limiter/spec.md

@@ -340,6 +340,57 @@ reporting it wrongly.
   configuration second; evidence rebinding third. Independent verification
   remains `not performed` unless a separate verifier inspects the final state.
 
+## REVISION 7 — fail-closed gauntlet orchestration (Tier 3)
+
+Approved 2026-08-18. This revision repairs a demonstrated fail-open defect in
+the gauntlet entry point; it does not change rate-limiter runtime behaviour or
+its public API.
+
+Before this revision, deleting a layer command while leaving its heading could
+make `tools/gauntlet.sh` print that heading, perform no work for the layer, exit
+zero and announce that every layer was green. This was reproduced by deleting
+the committed mutation invocation: no mutant ran, but the gauntlet still
+reported success.
+
+### Behaviour
+
+- The gauntlet has a fixed manifest of expected layers and records a layer as
+  complete only after all commands for that layer succeed.
+- A layer command that fails stops the gauntlet immediately, preserves its
+  non-zero status and names the failed layer. Later layers do not run.
+- A successful command sequence that omits any expected layer fails at the
+  final audit and names every missing layer.
+- Unknown and duplicate layer completions fail instead of silently changing or
+  overstating the run.
+- The all-green message is emitted only by the final completion audit, after
+  every expected layer has completed exactly once.
+
+### Must NOT do
+
+- Do not use a printed heading as evidence that a layer ran.
+- Do not rely on `set -e` to stop a command placed on the left side of `&&` or
+  inside another conditional context.
+- Do not extend application coverage or mutation gates across all of `tools/`
+  as a substitute for a control aimed at this orchestration failure mode.
+- Do not claim that one negative control proves a checker recognizes every
+  violation; each control proves only its named known-bad case.
+
+### Setup plan
+
+- Work on branch `codex/issue-13-gauntlet-orchestration`, preserving unrelated
+  untracked assets in the user's checkout.
+- Add `tools/gauntlet_layers.sh` for the expected-layer manifest, execution
+  wrapper and final audit; add `tools/test_gauntlet_orchestration.sh` with
+  controls for an omitted layer and a failed command, plus unknown and
+  duplicate registrations; modify `tools/gauntlet.sh` to use the helper.
+- Clarify the reusable assurance boundary in
+  `skills/old-coder/references/gauntlet.md`: targeted negative controls guard
+  identified fail-open modes in trust-chain tooling, while application
+  coverage and mutation remain scoped to the subject under test.
+- No new dependency. Commit cadence: this approved SPEC first; tests plus
+  implementation second; evidence rebinding third. Independent verification
+  remains `not performed` unless a separate verifier inspects the final state.
+
 ## Revision history
 
 Revisions 1–3 (2026-07-25 → 07-27) were made autonomously during the original

demo-rate-limiter/tools/gauntlet.sh

@@ -9,26 +9,31 @@ find . -name __pycache__ -type d -prune -exec rm -rf {} +
 PY=.venv/bin
 
 . tools/must_not_match.sh
+. tools/gauntlet_layers.sh
 
-echo "=== checker self-test ==="
-sh tools/test_gauntlet_checks.sh
+run_layer orchestration-self-test sh tools/test_gauntlet_orchestration.sh
 
-echo "=== source-state self-test ==="
-"$PY/pytest" -q tests/test_source_state.py
+run_layer checker-self-test sh tools/test_gauntlet_checks.sh
+
+run_layer source-state-self-test "$PY/pytest" -q tests/test_source_state.py
 
-echo "=== tests + coverage ==="
 # --cov-fail-under makes this layer a gate. Without it the layer printed a
 # percentage and exited 0 no matter how far coverage fell: a fail-open layer
 # inside a gauntlet whose first line promises to fail on the first broken one.
-"$PY/pytest" -q --cov=ratelimiter --cov-report=term-missing --cov-fail-under=100
-echo "=== types ==="
-"$PY/mypy" src tests examples tools
-echo "=== lint + format ==="
-"$PY/ruff" check .
-"$PY/ruff" format --check .
-echo "=== supply chain ==="
-"$PY/pip-audit" -r requirements-dev.txt
-echo "=== must-not scans ==="
+run_layer tests-coverage \
+  "$PY/pytest" -q --cov=ratelimiter --cov-report=term-missing --cov-fail-under=100
+
+run_layer types "$PY/mypy" src tests examples tools
+
+layer_lint_format() {
+  "$PY/ruff" check . || return $?
+  "$PY/ruff" format --check . || return $?
+}
+run_layer lint-format layer_lint_format
+
+run_layer supply-chain "$PY/pip-audit" -r requirements-dev.txt
+
+layer_must_not_scans() {
 # Matches usage forms, not the word: `time\.` alone missed `from time import
 # sleep`. Deliberately not a bare word-boundary match on `time`, which fires
 # on conftest's own "No real time in tests" docstring, on `timestamps`, on
@@ -38,22 +43,25 @@ echo "=== must-not scans ==="
 # Scope is narrower than the Must NOT's ambition: `Event.wait(timeout=)` and
 # `Thread.join(timeout=)` are NOT matched. They are declared in spec.md as an
 # exception rather than excluded here, because a pattern cannot decide intent.
-must_not_match 'import[[:space:]]+time|from[[:space:]]+time[[:space:]]+import|time\.[a-zA-Z_]|datetime|sleep[[:space:]]*\(|perf_counter[[:space:]]*\(|monotonic[[:space:]]*\(' tests
+  must_not_match 'import[[:space:]]+time|from[[:space:]]+time[[:space:]]+import|time\.[a-zA-Z_]|datetime|sleep[[:space:]]*\(|perf_counter[[:space:]]*\(|monotonic[[:space:]]*\(' tests || return $?
 # Bracketed letters stop the pattern literal from matching itself. The path
 # list now includes CI config and metadata: workflows are where credentials
 # actually appear, and scanning only src/tests/tools/examples missed them.
-must_not_match 'api[_-]?key|s[e]cret|pass[w]ord|t[o]ken|private[_ -]?key|BEGIN[[:space:]]+[A-Z ]*PRIVATE' \
-  src tests tools examples spec.md pyproject.toml requirements-dev.txt ../.github
-echo "must-not scans clean"
-echo "=== mutation ==="
+  must_not_match 'api[_-]?key|s[e]cret|pass[w]ord|t[o]ken|private[_ -]?key|BEGIN[[:space:]]+[A-Z ]*PRIVATE' \
+    src tests tools examples spec.md pyproject.toml requirements-dev.txt ../.github || return $?
+  echo "must-not scans clean"
+}
+run_layer must-not-scans layer_must_not_scans
+
 # Negative control first: a killer and a strictly-equivalent mutant of
 # identical size under one pinned mtime. If bytecode ever leaks between runs,
 # the equivalent one inherits the killer's verdict and the whole kill count is
 # inflated — silently, and only ever upward.
-"$PY/python" tools/mutants.py --negative-control
-"$PY/python" tools/mutants.py
-echo "=== real execution ==="
-"$PY/python" examples/demo.py
-echo "=== source state ==="
-tools/source_state.sh
-echo "=== gauntlet: all layers green ==="
+run_layer mutation-control "$PY/python" tools/mutants.py --negative-control
+run_layer mutation "$PY/python" tools/mutants.py
+
+run_layer real-execution "$PY/python" examples/demo.py
+
+run_layer source-state tools/source_state.sh
+
+finish_gauntlet

demo-rate-limiter/tools/gauntlet_layers.sh

@@ -0,0 +1,57 @@
+#!/bin/sh
+# Fail-closed execution and completion accounting for the gauntlet entry point.
+GAUNTLET_EXPECTED_LAYERS="orchestration-self-test checker-self-test source-state-self-test tests-coverage types lint-format supply-chain must-not-scans mutation-control mutation real-execution source-state"
+GAUNTLET_COMPLETED_LAYERS=""
+
+run_layer() {
+  if [ "$#" -lt 2 ]; then
+    echo "FAIL: run_layer requires a layer name and command" >&2
+    return 2
+  fi
+
+  layer=$1
+  shift
+
+  case " $GAUNTLET_EXPECTED_LAYERS " in
+    *" $layer "*) ;;
+    *)
+      echo "FAIL: unknown layer '$layer'" >&2
+      return 2
+      ;;
+  esac
+
+  case " $GAUNTLET_COMPLETED_LAYERS " in
+    *" $layer "*)
+      echo "FAIL: duplicate layer '$layer'" >&2
+      return 2
+      ;;
+  esac
+
+  printf '=== %s ===\n' "$layer"
+  if "$@"; then
+    GAUNTLET_COMPLETED_LAYERS="$GAUNTLET_COMPLETED_LAYERS $layer"
+    return 0
+  else
+    rc=$?
+    printf "FAIL: layer '%s' failed (rc=%s)\n" "$layer" "$rc" >&2
+    return "$rc"
+  fi
+}
+
+finish_gauntlet() {
+  missing=0
+  for layer in $GAUNTLET_EXPECTED_LAYERS; do
+    case " $GAUNTLET_COMPLETED_LAYERS " in
+      *" $layer "*) ;;
+      *)
+        echo "FAIL: missing layer '$layer'" >&2
+        missing=1
+        ;;
+    esac
+  done
+
+  if [ "$missing" -ne 0 ]; then
+    return 1
+  fi
+  echo "=== gauntlet: all layers green ==="
+}