From 905e9894cf4726082ac9aefd96f91751b63364f8 Mon Sep 17 00:00:00 2001 From: "Li, Amazing Ang" Date: Tue, 18 Aug 2026 16:49:07 +0800 Subject: [PATCH 1/5] Specify fail-closed gauntlet orchestration --- demo-rate-limiter/spec.md | 51 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 51 insertions(+) diff --git a/demo-rate-limiter/spec.md b/demo-rate-limiter/spec.md index 422550d..32e6952 100644 --- a/demo-rate-limiter/spec.md +++ b/demo-rate-limiter/spec.md @@ -340,6 +340,57 @@ reporting it wrongly. configuration second; evidence rebinding third. Independent verification remains `not performed` unless a separate verifier inspects the final state. +## REVISION 7 — fail-closed gauntlet orchestration (Tier 3) + +Approved 2026-08-18. This revision repairs a demonstrated fail-open defect in +the gauntlet entry point; it does not change rate-limiter runtime behaviour or +its public API. + +Before this revision, deleting a layer command while leaving its heading could +make `tools/gauntlet.sh` print that heading, perform no work for the layer, exit +zero and announce that every layer was green. This was reproduced by deleting +the committed mutation invocation: no mutant ran, but the gauntlet still +reported success. + +### Behaviour + +- The gauntlet has a fixed manifest of expected layers and records a layer as + complete only after all commands for that layer succeed. +- A layer command that fails stops the gauntlet immediately, preserves its + non-zero status and names the failed layer. Later layers do not run. +- A successful command sequence that omits any expected layer fails at the + final audit and names every missing layer. +- Unknown and duplicate layer completions fail instead of silently changing or + overstating the run. +- The all-green message is emitted only by the final completion audit, after + every expected layer has completed exactly once. + +### Must NOT do + +- Do not use a printed heading as evidence that a layer ran. +- Do not rely on `set -e` to stop a command placed on the left side of `&&` or + inside another conditional context. +- Do not extend application coverage or mutation gates across all of `tools/` + as a substitute for a control aimed at this orchestration failure mode. +- Do not claim that one negative control proves a checker recognizes every + violation; each control proves only its named known-bad case. + +### Setup plan + +- Work on branch `codex/issue-13-gauntlet-orchestration`, preserving unrelated + untracked assets in the user's checkout. +- Add `tools/gauntlet_layers.sh` for the expected-layer manifest, execution + wrapper and final audit; add `tools/test_gauntlet_orchestration.sh` with + controls for an omitted layer and a failed command, plus unknown and + duplicate registrations; modify `tools/gauntlet.sh` to use the helper. +- Clarify the reusable assurance boundary in + `skills/old-coder/references/gauntlet.md`: targeted negative controls guard + identified fail-open modes in trust-chain tooling, while application + coverage and mutation remain scoped to the subject under test. +- No new dependency. Commit cadence: this approved SPEC first; tests plus + implementation second; evidence rebinding third. Independent verification + remains `not performed` unless a separate verifier inspects the final state. + ## Revision history Revisions 1–3 (2026-07-25 → 07-27) were made autonomously during the original From 5b8dc1b632541297b683be8a76488a5e7af7d03c Mon Sep 17 00:00:00 2001 From: "Li, Amazing Ang" Date: Tue, 18 Aug 2026 16:51:45 +0800 Subject: [PATCH 2/5] Fail closed when a gauntlet layer is skipped --- demo-rate-limiter/tools/gauntlet.sh | 60 +++++----- demo-rate-limiter/tools/gauntlet_layers.sh | 57 ++++++++++ .../tools/test_gauntlet_orchestration.sh | 103 ++++++++++++++++++ skills/old-coder/references/gauntlet.md | 11 ++ 4 files changed, 205 insertions(+), 26 deletions(-) create mode 100755 demo-rate-limiter/tools/gauntlet_layers.sh create mode 100755 demo-rate-limiter/tools/test_gauntlet_orchestration.sh diff --git a/demo-rate-limiter/tools/gauntlet.sh b/demo-rate-limiter/tools/gauntlet.sh index b3e9e3f..96c8bea 100755 --- a/demo-rate-limiter/tools/gauntlet.sh +++ b/demo-rate-limiter/tools/gauntlet.sh @@ -9,26 +9,31 @@ find . -name __pycache__ -type d -prune -exec rm -rf {} + PY=.venv/bin . tools/must_not_match.sh +. tools/gauntlet_layers.sh -echo "=== checker self-test ===" -sh tools/test_gauntlet_checks.sh +run_layer orchestration-self-test sh tools/test_gauntlet_orchestration.sh -echo "=== source-state self-test ===" -"$PY/pytest" -q tests/test_source_state.py +run_layer checker-self-test sh tools/test_gauntlet_checks.sh + +run_layer source-state-self-test "$PY/pytest" -q tests/test_source_state.py -echo "=== tests + coverage ===" # --cov-fail-under makes this layer a gate. Without it the layer printed a # percentage and exited 0 no matter how far coverage fell: a fail-open layer # inside a gauntlet whose first line promises to fail on the first broken one. -"$PY/pytest" -q --cov=ratelimiter --cov-report=term-missing --cov-fail-under=100 -echo "=== types ===" -"$PY/mypy" src tests examples tools -echo "=== lint + format ===" -"$PY/ruff" check . -"$PY/ruff" format --check . -echo "=== supply chain ===" -"$PY/pip-audit" -r requirements-dev.txt -echo "=== must-not scans ===" +run_layer tests-coverage \ + "$PY/pytest" -q --cov=ratelimiter --cov-report=term-missing --cov-fail-under=100 + +run_layer types "$PY/mypy" src tests examples tools + +layer_lint_format() { + "$PY/ruff" check . || return $? + "$PY/ruff" format --check . || return $? +} +run_layer lint-format layer_lint_format + +run_layer supply-chain "$PY/pip-audit" -r requirements-dev.txt + +layer_must_not_scans() { # Matches usage forms, not the word: `time\.` alone missed `from time import # sleep`. Deliberately not a bare word-boundary match on `time`, which fires # on conftest's own "No real time in tests" docstring, on `timestamps`, on @@ -38,22 +43,25 @@ echo "=== must-not scans ===" # Scope is narrower than the Must NOT's ambition: `Event.wait(timeout=)` and # `Thread.join(timeout=)` are NOT matched. They are declared in spec.md as an # exception rather than excluded here, because a pattern cannot decide intent. -must_not_match 'import[[:space:]]+time|from[[:space:]]+time[[:space:]]+import|time\.[a-zA-Z_]|datetime|sleep[[:space:]]*\(|perf_counter[[:space:]]*\(|monotonic[[:space:]]*\(' tests + must_not_match 'import[[:space:]]+time|from[[:space:]]+time[[:space:]]+import|time\.[a-zA-Z_]|datetime|sleep[[:space:]]*\(|perf_counter[[:space:]]*\(|monotonic[[:space:]]*\(' tests || return $? # Bracketed letters stop the pattern literal from matching itself. The path # list now includes CI config and metadata: workflows are where credentials # actually appear, and scanning only src/tests/tools/examples missed them. -must_not_match 'api[_-]?key|s[e]cret|pass[w]ord|t[o]ken|private[_ -]?key|BEGIN[[:space:]]+[A-Z ]*PRIVATE' \ - src tests tools examples spec.md pyproject.toml requirements-dev.txt ../.github -echo "must-not scans clean" -echo "=== mutation ===" + must_not_match 'api[_-]?key|s[e]cret|pass[w]ord|t[o]ken|private[_ -]?key|BEGIN[[:space:]]+[A-Z ]*PRIVATE' \ + src tests tools examples spec.md pyproject.toml requirements-dev.txt ../.github || return $? + echo "must-not scans clean" +} +run_layer must-not-scans layer_must_not_scans + # Negative control first: a killer and a strictly-equivalent mutant of # identical size under one pinned mtime. If bytecode ever leaks between runs, # the equivalent one inherits the killer's verdict and the whole kill count is # inflated — silently, and only ever upward. -"$PY/python" tools/mutants.py --negative-control -"$PY/python" tools/mutants.py -echo "=== real execution ===" -"$PY/python" examples/demo.py -echo "=== source state ===" -tools/source_state.sh -echo "=== gauntlet: all layers green ===" +run_layer mutation-control "$PY/python" tools/mutants.py --negative-control +run_layer mutation "$PY/python" tools/mutants.py + +run_layer real-execution "$PY/python" examples/demo.py + +run_layer source-state tools/source_state.sh + +finish_gauntlet diff --git a/demo-rate-limiter/tools/gauntlet_layers.sh b/demo-rate-limiter/tools/gauntlet_layers.sh new file mode 100755 index 0000000..9df39ad --- /dev/null +++ b/demo-rate-limiter/tools/gauntlet_layers.sh @@ -0,0 +1,57 @@ +#!/bin/sh +# Fail-closed execution and completion accounting for the gauntlet entry point. +GAUNTLET_EXPECTED_LAYERS="orchestration-self-test checker-self-test source-state-self-test tests-coverage types lint-format supply-chain must-not-scans mutation-control mutation real-execution source-state" +GAUNTLET_COMPLETED_LAYERS="" + +run_layer() { + if [ "$#" -lt 2 ]; then + echo "FAIL: run_layer requires a layer name and command" >&2 + return 2 + fi + + layer=$1 + shift + + case " $GAUNTLET_EXPECTED_LAYERS " in + *" $layer "*) ;; + *) + echo "FAIL: unknown layer '$layer'" >&2 + return 2 + ;; + esac + + case " $GAUNTLET_COMPLETED_LAYERS " in + *" $layer "*) + echo "FAIL: duplicate layer '$layer'" >&2 + return 2 + ;; + esac + + printf '=== %s ===\n' "$layer" + if "$@"; then + GAUNTLET_COMPLETED_LAYERS="$GAUNTLET_COMPLETED_LAYERS $layer" + return 0 + else + rc=$? + printf "FAIL: layer '%s' failed (rc=%s)\n" "$layer" "$rc" >&2 + return "$rc" + fi +} + +finish_gauntlet() { + missing=0 + for layer in $GAUNTLET_EXPECTED_LAYERS; do + case " $GAUNTLET_COMPLETED_LAYERS " in + *" $layer "*) ;; + *) + echo "FAIL: missing layer '$layer'" >&2 + missing=1 + ;; + esac + done + + if [ "$missing" -ne 0 ]; then + return 1 + fi + echo "=== gauntlet: all layers green ===" +} diff --git a/demo-rate-limiter/tools/test_gauntlet_orchestration.sh b/demo-rate-limiter/tools/test_gauntlet_orchestration.sh new file mode 100755 index 0000000..acb750c --- /dev/null +++ b/demo-rate-limiter/tools/test_gauntlet_orchestration.sh @@ -0,0 +1,103 @@ +#!/bin/sh +# Negative controls for the gauntlet's layer-completion contract. +cd "$(dirname "$0")/.." + +failures=0 + +expect_rc() { + want=$1; got=$2; what=$3 + if [ "$got" -eq "$want" ]; then + echo " ok: $what (rc=$got)" + else + echo " NOT OK: $what (want rc $want, got rc $got)" + failures=$((failures + 1)) + fi +} + +expect_contains() { + output=$1; needle=$2; what=$3 + case "$output" in + *"$needle"*) echo " ok: $what" ;; + *) + echo " NOT OK: $what (missing: $needle)" + failures=$((failures + 1)) + ;; + esac +} + +expect_absent() { + output=$1; needle=$2; what=$3 + case "$output" in + *"$needle"*) + echo " NOT OK: $what (unexpected: $needle)" + failures=$((failures + 1)) + ;; + *) echo " ok: $what" ;; + esac +} + +# 1. Omitting an expected layer must make the final audit fail and name it. +output=$({ + . tools/gauntlet_layers.sh + for layer in $GAUNTLET_EXPECTED_LAYERS; do + [ "$layer" = mutation ] || run_layer "$layer" true + done + finish_gauntlet +} 2>&1) +rc=$? +expect_rc 1 "$rc" "omitted layer fails the final audit" +expect_contains "$output" "missing layer 'mutation'" "omitted layer is named" +expect_absent "$output" "all layers green" "omitted layer cannot report green" + +# 2. A broken command must preserve its status, name its layer and stop before +# later work. This specifically guards against relying on set -e through an +# AND-list or another conditional context. +output=$(sh -c ' + set -e + . tools/gauntlet_layers.sh + run_layer mutation sh -c "exit 7" + echo LATER-LAYER-RAN +' 2>&1) +rc=$? +expect_rc 7 "$rc" "failed command status is preserved" +expect_contains "$output" "layer 'mutation' failed (rc=7)" "failed layer is named" +expect_absent "$output" "LATER-LAYER-RAN" "failed command stops later work" +expect_absent "$output" "all layers green" "failed command cannot report green" + +# 3. A typo in a layer name must not create a new, unrequired layer. +output=$(sh -c ' + . tools/gauntlet_layers.sh + run_layer mutatoin true +' 2>&1) +rc=$? +expect_rc 2 "$rc" "unknown layer fails" +expect_contains "$output" "unknown layer 'mutatoin'" "unknown layer is named" + +# 4. Re-running one cheap layer must not stand in for a missing layer. +output=$(sh -c ' + . tools/gauntlet_layers.sh + run_layer mutation true + run_layer mutation true +' 2>&1) +rc=$? +expect_rc 2 "$rc" "duplicate layer fails" +expect_contains "$output" "duplicate layer 'mutation'" "duplicate layer is named" + +# 5. The controls are not a script that can only fail: one successful command +# for every expected layer reaches the one legitimate all-green message. +output=$( + . tools/gauntlet_layers.sh + for layer in $GAUNTLET_EXPECTED_LAYERS; do + run_layer "$layer" true + done + finish_gauntlet +) 2>&1 +rc=$? +expect_rc 0 "$rc" "complete manifest passes" +expect_contains "$output" "=== gauntlet: all layers green ===" "complete manifest reports green" + +if [ "$failures" -ne 0 ]; then + echo "FAIL: $failures orchestration expectation(s) violated" + exit 1 +fi +echo "orchestration self-test clean" diff --git a/skills/old-coder/references/gauntlet.md b/skills/old-coder/references/gauntlet.md index 1b70ca6..253b22c 100644 --- a/skills/old-coder/references/gauntlet.md +++ b/skills/old-coder/references/gauntlet.md @@ -183,6 +183,17 @@ into a vacuous pass. Prove each home-grown check can fail with a one-off negative control (feed it a known-bad fixture; make its input unreadable) and record the control in EVIDENCE's honest notes. +Keep the assurance boundary explicit: application coverage and mutation target +the subject under test; do not widen them across every orchestration script by +default. Protect home-grown tools in the gauntlet's trust chain with targeted +negative controls for identified fail-open modes, and pin the failure reason, +not merely a non-zero status. A control proves only its named known-bad case, +not the whole tool. For the entry point itself, bind execution to completion: +maintain a fixed expected-layer manifest, record each layer only after its +commands succeed, and audit the manifest before printing success. Do not use a +heading as evidence that a layer ran, and do not rely on `set -e` through `&&` +or another conditional context; handle the command status explicitly. + ## Templates The Gherkin scenario template, the SPEC template, and the EVIDENCE report From 9f4fae3dd3b272216a1208fbf8d0127c56afe5df Mon Sep 17 00:00:00 2001 From: "Li, Amazing Ang" Date: Tue, 18 Aug 2026 16:55:28 +0800 Subject: [PATCH 3/5] Rebind evidence after gauntlet orchestration repair --- demo-rate-limiter/evidence.md | 53 +++++++++++++++++++++++++---------- 1 file changed, 38 insertions(+), 15 deletions(-) diff --git a/demo-rate-limiter/evidence.md b/demo-rate-limiter/evidence.md index 0f8a016..9292a0f 100644 --- a/demo-rate-limiter/evidence.md +++ b/demo-rate-limiter/evidence.md @@ -1,24 +1,24 @@ # Evidence Report — Sliding-Window Rate Limiter (Tier 3) -- Spec approval: **obtained** for REVISION 4 (2026-08-09), REVISION 5 and - REVISION 6 (2026-08-18) — the human approved each contract change before - implementation. Earlier revisions (2026-07-25, 2026-07-27) were autonomous - and are still unapproved; treat them as the weaker part of the spec. +- Spec approval: **obtained** for REVISION 4 (2026-08-09) and REVISION 5–7 + (2026-08-18) — the human approved each contract change before implementation. + Earlier revisions (2026-07-25, 2026-07-27) were autonomous and are still + unapproved; treat them as the weaker part of the spec. - Independent verification: **not performed against the final source state - `4734451`.** Six earlier rounds were performed; the last verified state + `5b8dc1b`.** Six earlier rounds were performed; the last verified state `d0b506c` returned `failed`, and the fixes made since — one of them behavioural — are disclosed below as unverified. This report is finalized as a **declared downgrade**, not on the strength of a passing verdict. A verdict attaches to the state a verifier actually saw, and no verifier has seen this one. -- Source state: source commit `4734451`; sha256 tree hash - `dbccc212daa35442` — reproduce both with `./tools/source_state.sh` from any +- Source state: source commit `5b8dc1b`; sha256 tree hash + `0bf97ed089bb4775` — reproduce both with `./tools/source_state.sh` from any directory. When a binding is produced the tree hash is the required content identity; the source commit is provenance and is supplied only where complete history is available, so a shallow checkout reports `(unavailable: shallow history)` and a no-Git archive reports `(no git)`, both alongside this same tree hash. No error path emits a binding at all. - The script separately reports current HEAD; commits after `4734451` that + The script separately reports current HEAD; commits after `5b8dc1b` that touch only this report or other out-of-scope paths preserve the source commit and tree hash. The manifest includes `.github/workflows`, which decides whether the gauntlet runs in CI at all. @@ -27,13 +27,14 @@ - Entry point: `./tools/gauntlet.sh` reruns every layer below. All numbers are from one final fresh run of the entry point, executed -2026-08-18 at source commit `4734451` after the last code edit. +2026-08-18 at source commit `5b8dc1b` after the last code edit. `spec.md` was deliberately pruned back to a contract before REVISION 5 (339 → 255 lines). Every clause, invariant, obligation and failure-model row survived; what was removed is the per-revision forensics, which lives in the -honest notes below and in git. REVISION 5 adds the approved source-binding -contract and tests without changing rate-limiter behaviour. +honest notes below and in git. REVISION 5–7 add the approved source-binding, +provenance and gauntlet-orchestration contracts without changing rate-limiter +behaviour. ## Spec → Test mapping @@ -72,12 +73,14 @@ Status legend: pass / fail / unverified / n-a. | REVISION 5: source binding is reproducible and fail-closed | test_source_state.py (ignored artifacts, staged/unstaged/untracked/deleted inputs, clean clone, no-Git archive, arbitrary cwd, evidence-only commit) | pass | | REVISION 6: truncated history withholds provenance, never invents it | test_source_state.py::test_shallow_history_withholds_provenance (exact marker, shallow HEAD, tree equal to the full clone, empty stderr) | pass | | REVISION 6: covered error scenarios pin their reason and emit no binding | test_source_state.py (Git dirty, Git deletion, Git untracked, no-Git missing input, no-Git empty scope — each asserts the reason and `stdout == ""`) | pass | +| REVISION 7: omitted or failed gauntlet work cannot report green | test_gauntlet_orchestration.sh (omitted layer, failing command with exact rc and stopped sentinel, unknown layer, duplicate layer, complete-manifest positive control) | pass | ## Gauntlet (final fresh run: `./tools/gauntlet.sh`) | Layer | Command | Result | |---|---|---| -| Checker self-test | `sh tools/test_gauntlet_checks.sh` (first layer; asserts the must-not scan fails on a planted pattern, passes on a clean tree, and fails closed with a distinct rc 2 when the scan itself breaks) | 3/3 expectations ok | +| Orchestration self-test | `sh tools/test_gauntlet_orchestration.sh` (omitted layer, command failure, unknown layer, duplicate layer, and complete-manifest positive control against the real helper) | 5/5 scenarios, 13/13 expectations ok | +| Checker self-test | `sh tools/test_gauntlet_checks.sh` (asserts the must-not scan fails on a planted pattern, passes on a clean tree, and fails closed with a distinct rc 2 when the scan itself breaks) | 3/3 expectations ok | | Source-state self-test | `pytest -q tests/test_source_state.py` (negative controls for the covered fail-closed scenarios; shallow/full-history, clean clone and no-Git archive comparisons) | 9/9 passed | | Mutation harness negative control | `python tools/mutants.py --negative-control` (a killer and a strictly-equivalent mutant of identical size under one pinned mtime) | C1 KILLED, C2 SURVIVED — ok | | Tests | `pytest -q --cov=ratelimiter` | 50 passed, 0 failed | @@ -89,7 +92,7 @@ Status legend: pass / fail / unverified / n-a. | Real execution | `python examples/demo.py` (real `time.monotonic`) | burst of 5 → `[True, True, True, False, False]`; other key unaffected; allowed again after window | | Supply chain | `pip-audit -r requirements-dev.txt` | no known vulnerabilities; runtime dependencies: **none** (stdlib only; `threading` is stdlib) | | Secret scan | must-not scan in `tools/gauntlet.sh` over src, tests, tools, examples, spec.md, pyproject.toml, requirements-dev.txt and `../.github` | clean, no matches | -| Source binding | `tools/source_state.sh` (last gauntlet layer) | source commit `4734451`; tree `dbccc212daa35442`; current HEAD is reported separately | +| Source binding | `tools/source_state.sh` (last gauntlet layer) | source commit `5b8dc1b`; tree `0bf97ed089bb4775`; current HEAD is reported separately | | License check | — | n-a: zero runtime dependencies, nothing redistributed beyond this repo's own MIT code | | Suite health | pytest-randomly (order shuffled every run) | 50 passed in randomized order, 10/10 consecutive runs | @@ -109,7 +112,7 @@ Status legend: pass / fail / unverified / n-a. 22 mutants). What it cannot detect: the mutant list is hand-written, so unlike a tool generating mutants from the syntax tree it can only test weaknesses somebody thought of in advance. -- **UNAVAILABLE — shell lint (shellcheck)** for the four scripts that implement +- **UNAVAILABLE — shell lint (shellcheck)** for the six scripts that implement half the gates: no tool installed, and nothing ran in its place. Every Python file gets three static layers and the shell gets none. Known gap, raised by verification round 4. @@ -162,10 +165,30 @@ independently verified**: - REVISION 5 and the reproducible, fail-closed source-state mechanism in commits `86bfcf4` and `d45cc2f`; - REVISION 6 and the shallow-history provenance repair in commits `3e45e16` - and `49e8762`, plus the historical CI wording correction in `4734451`. + and `49e8762`, plus the historical CI wording correction in `4734451`; +- REVISION 7 and the fail-closed gauntlet orchestration in commit `5b8dc1b`. ## Honest notes +- **The gauntlet previously authenticated headings, not completed work.** The + issue investigation removed the committed mutation invocation while keeping + its heading; on a clean tree the entry point ran no mutant, exited zero and + printed `all layers green`. The RED implementation reproduced that class of + failure with a deliberately fail-open helper: 8 of 13 expectations failed, + including an omitted `mutation` layer returning zero and printing green. + REVISION 7 puts the expected manifest in a real sourced helper, couples each + command to completion through `run_layer`, and makes the all-green message a + result of the final audit. Its five-scenario control now passes 13/13 + expectations. This catches an accidental omission of a registered layer + invocation; it does not make a coordinated edit to both the manifest and + caller impossible. + +- **REVISION 7 keeps application and instrument assurance distinct.** The + coverage and mutation targets remain `src/ratelimiter`; they were not widened + across `tools/`. Home-grown trust-chain tools instead carry targeted negative + controls for demonstrated fail-open modes. Those controls prove their named + cases only, not the correctness of every path through each tool. + - **The first REVISION 6 evidence rebind was internally stale.** Commit `83004bf` updated the headline source state but left four REVISION 5 values in the gauntlet table: 6 rather than 9 source-state tests, 47 rather than 50 From 42528d951b7d3df7b9468346ff4ff8eefcb3b9bd Mon Sep 17 00:00:00 2001 From: "Li, Amazing Ang" Date: Tue, 18 Aug 2026 17:25:21 +0800 Subject: [PATCH 4/5] Capture stderr in the complete-manifest control MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Scenario 5 placed `2>&1` outside the command substitution, so the subshell's stderr leaked to the terminal instead of being captured — unlike the four controls above it. It asserts only stdout, so nothing was wrong today, but a regression that printed the all-green message while also emitting a missing- or failed-layer complaint on stderr would have passed this scenario on its own. The other controls still catch that; this restores the layer of depth they were written to have. Co-Authored-By: Claude Opus 5 --- demo-rate-limiter/tools/test_gauntlet_orchestration.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/demo-rate-limiter/tools/test_gauntlet_orchestration.sh b/demo-rate-limiter/tools/test_gauntlet_orchestration.sh index acb750c..a0eed4a 100755 --- a/demo-rate-limiter/tools/test_gauntlet_orchestration.sh +++ b/demo-rate-limiter/tools/test_gauntlet_orchestration.sh @@ -85,13 +85,13 @@ expect_contains "$output" "duplicate layer 'mutation'" "duplicate layer is named # 5. The controls are not a script that can only fail: one successful command # for every expected layer reaches the one legitimate all-green message. -output=$( +output=$({ . tools/gauntlet_layers.sh for layer in $GAUNTLET_EXPECTED_LAYERS; do run_layer "$layer" true done finish_gauntlet -) 2>&1 +} 2>&1) rc=$? expect_rc 0 "$rc" "complete manifest passes" expect_contains "$output" "=== gauntlet: all layers green ===" "complete manifest reports green" From 5c9d48af8c851a8ed9013554be211cf81a632841 Mon Sep 17 00:00:00 2001 From: "Li, Amazing Ang" Date: Tue, 18 Aug 2026 17:29:47 +0800 Subject: [PATCH 5/5] Rebind evidence after the control-depth fix MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Source state is now `42528d9` / `5aa96ec5487c957c`. The 50-test randomized suite was rerun 10/10 at this state rather than inherited, and the orchestration control was re-verified at 13/13 with the fail-open mutation battery still turning it red. Records that scenario 5 was found shallow by review rather than by the control, and that the REVISION 7 contract is unchanged — this is a rebind, not a new revision, because `tools/` is inside the hashed scope. Co-Authored-By: Claude Opus 5 --- demo-rate-limiter/evidence.md | 26 +++++++++++++++++++------- 1 file changed, 19 insertions(+), 7 deletions(-) diff --git a/demo-rate-limiter/evidence.md b/demo-rate-limiter/evidence.md index 9292a0f..d208e68 100644 --- a/demo-rate-limiter/evidence.md +++ b/demo-rate-limiter/evidence.md @@ -5,20 +5,20 @@ Earlier revisions (2026-07-25, 2026-07-27) were autonomous and are still unapproved; treat them as the weaker part of the spec. - Independent verification: **not performed against the final source state - `5b8dc1b`.** Six earlier rounds were performed; the last verified state + `42528d9`.** Six earlier rounds were performed; the last verified state `d0b506c` returned `failed`, and the fixes made since — one of them behavioural — are disclosed below as unverified. This report is finalized as a **declared downgrade**, not on the strength of a passing verdict. A verdict attaches to the state a verifier actually saw, and no verifier has seen this one. -- Source state: source commit `5b8dc1b`; sha256 tree hash - `0bf97ed089bb4775` — reproduce both with `./tools/source_state.sh` from any +- Source state: source commit `42528d9`; sha256 tree hash + `5aa96ec5487c957c` — reproduce both with `./tools/source_state.sh` from any directory. When a binding is produced the tree hash is the required content identity; the source commit is provenance and is supplied only where complete history is available, so a shallow checkout reports `(unavailable: shallow history)` and a no-Git archive reports `(no git)`, both alongside this same tree hash. No error path emits a binding at all. - The script separately reports current HEAD; commits after `5b8dc1b` that + The script separately reports current HEAD; commits after `42528d9` that touch only this report or other out-of-scope paths preserve the source commit and tree hash. The manifest includes `.github/workflows`, which decides whether the gauntlet runs in CI at all. @@ -27,7 +27,7 @@ - Entry point: `./tools/gauntlet.sh` reruns every layer below. All numbers are from one final fresh run of the entry point, executed -2026-08-18 at source commit `5b8dc1b` after the last code edit. +2026-08-18 at source commit `42528d9` after the last code edit. `spec.md` was deliberately pruned back to a contract before REVISION 5 (339 → 255 lines). Every clause, invariant, obligation and failure-model row @@ -92,7 +92,7 @@ Status legend: pass / fail / unverified / n-a. | Real execution | `python examples/demo.py` (real `time.monotonic`) | burst of 5 → `[True, True, True, False, False]`; other key unaffected; allowed again after window | | Supply chain | `pip-audit -r requirements-dev.txt` | no known vulnerabilities; runtime dependencies: **none** (stdlib only; `threading` is stdlib) | | Secret scan | must-not scan in `tools/gauntlet.sh` over src, tests, tools, examples, spec.md, pyproject.toml, requirements-dev.txt and `../.github` | clean, no matches | -| Source binding | `tools/source_state.sh` (last gauntlet layer) | source commit `5b8dc1b`; tree `0bf97ed089bb4775`; current HEAD is reported separately | +| Source binding | `tools/source_state.sh` (last gauntlet layer) | source commit `42528d9`; tree `5aa96ec5487c957c`; current HEAD is reported separately | | License check | — | n-a: zero runtime dependencies, nothing redistributed beyond this repo's own MIT code | | Suite health | pytest-randomly (order shuffled every run) | 50 passed in randomized order, 10/10 consecutive runs | @@ -166,7 +166,8 @@ independently verified**: commits `86bfcf4` and `d45cc2f`; - REVISION 6 and the shallow-history provenance repair in commits `3e45e16` and `49e8762`, plus the historical CI wording correction in `4734451`; -- REVISION 7 and the fail-closed gauntlet orchestration in commit `5b8dc1b`. +- REVISION 7 and the fail-closed gauntlet orchestration in commits `5b8dc1b` + and `42528d9`. ## Honest notes @@ -183,6 +184,17 @@ independently verified**: invocation; it does not make a coordinated edit to both the manifest and caller impossible. +- **One REVISION 7 control was shallower than the four beside it.** Scenario 5 + of `test_gauntlet_orchestration.sh` placed `2>&1` outside the command + substitution, so the subshell's stderr leaked to the terminal rather than + being captured. It asserts only stdout, so no expectation was wrong; but a + regression printing the all-green message while also complaining about a + missing or failed layer on stderr would have passed that scenario alone. + Found in review, not by the control. Fixed in `42528d9`; the five scenarios + still pass 13/13 and the fail-open mutation battery still turns them red. The + contract in REVISION 7 is unchanged, so this carries no new spec revision — + only a rebind, because `tools/` is inside the hashed source scope. + - **REVISION 7 keeps application and instrument assurance distinct.** The coverage and mutation targets remain `src/ratelimiter`; they were not widened across `tools/`. Home-grown trust-chain tools instead carry targeted negative