From b4728a5513854479330c15dcf7fb568d3f91449e Mon Sep 17 00:00:00 2001 From: vk Date: Tue, 1 Sep 2026 20:07:08 +0530 Subject: [PATCH 01/10] Add SAMPLES=N rate mode, cost capture and a named-not-called counter ATTEMPTS cannot answer "what fraction of the time does this skill fire". It early-stops on the first hit, so it measures "landed within 3 tries" and biases upward -- the file has said so above case_max_turns since 2026-08-23, and that early-stopping bias is what read encode-lessons-lint as a dead skill when it was turn-starved. Every before/after claim about a description needs a real rate, so SAMPLES=N runs N independent, non-retrying invocations and prints raw k/N. Default 0: with SAMPLES unset every path, output line and exit code is unchanged, and run_case's body is untouched -- the branch is a guard prepended to it. Two numbers ship alongside k/N because k/N alone is not interpretable here. total_cost_usd, read from the terminal result event with no `origin` (the same discriminator top_level_subtype already uses to avoid a subagent's result line). Nothing in this file counted a billed call before spending it; a full default run is up to 80 invocations and no line of output said so. named-not-called, from the assistant's prose. Measured 2026-08-27 in tests/evals/collision/RESULTS.md: when the tool a skill needs is denied, the model names the skill in prose and never calls Skill, which scores identically to a routing miss under fired=[]. DISALLOWED_TOOLS denies Write and Agent, which are exactly what the planning-chain skills and swarm exist to use. A 0/N without this number beside it cannot distinguish "the description does not route" from "it routes and the fence blocks the payoff". Sample rows pin both halves of the measurement: the instrument sha and a digest of the INSTALLED description bytes. Measured 2026-09-01: a project-local .claude/skills//SKILL.md is loaded, but a user-level skill of the same name wins, so the installed copy is what reaches the matcher and a repo edit is invisible to this harness until installed. Digesting the repo tree would pin a number to bytes the model never saw. This is the artifact discipline 1.47.0 withdrew two published numbers for lacking. A low rate does not fail the run -- only a fence breach does. Wiring a rate to a red exit would make the suite refuse to report the thing it was run to find out. Verified: bash -n; shellcheck -S warning clean; tests/dispatch-static.sh green, 4/4 assertions, 28 cases still declared and matched, 0 model calls. Co-Authored-By: Claude Opus 5 (1M context) --- tests/auto-trigger.sh | 210 +++++++++++++++++++++++++++++++++++++++++- 1 file changed, 209 insertions(+), 1 deletion(-) diff --git a/tests/auto-trigger.sh b/tests/auto-trigger.sh index 560dee4..8a34fa7 100755 --- a/tests/auto-trigger.sh +++ b/tests/auto-trigger.sh @@ -151,6 +151,31 @@ case_max_turns() { # honest about "does this situation ever route there" without crying wolf. ATTEMPTS="${ATTEMPTS:-3}" +# --------------------------------------------------------------------------- +# SAMPLES=N -- rate mode. OFF by default (0): with SAMPLES unset every line below behaves +# exactly as it did before, same output, same exit codes, same gate. +# +# ATTEMPTS is NOT a sample-size knob and raising it will never make one. It early-stops on the +# first hit, so it answers "did this land within N tries" and biases upward -- the same +# early-stopping error that read encode-lessons-lint as a dead skill when it was turn-starved +# (see the note above case_max_turns). SAMPLES runs N INDEPENDENT, non-retrying invocations of +# the same prompt and reports raw k/N. +# +# Report k/N, never a percentage and never an interval: at the n this mode is affordable at, a +# point estimate implies precision the sample does not have. +# +# Measurement is not a verdict. A low rate does NOT fail the run -- only a fence breach does. +# --------------------------------------------------------------------------- +SAMPLES="${SAMPLES:-0}" +SAMPLES_ALL="${SAMPLES_ALL:-0}" # required to sample every case; see the cost guard below +SAMPLE_LOG="${SAMPLE_LOG:-}" # optional path; one JSON line per sample, append-only +SAMPLE_CASES=0 +SAMPLE_CALLS=0 +SAMPLE_COST="0" +SAMPLE_HEAD="" +SAMPLE_DESCS="" +AT_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" + PASS_COUNT=0 FAIL_COUNT=0 declare -a RESULT_LINES=() @@ -313,6 +338,47 @@ top_level_subtype() { jq -r 'select(.type=="result" and (.origin==null)) | .subtype' "$jsonl" 2>/dev/null | tail -1 } +# --------------------------------------------------------------------------- +# top_level_cost OUT_JSONL +# Billed cost of ONE invocation, read from the same terminal result event top_level_subtype +# reads: the one with no `origin`. A subagent's result event carries origin.kind == +# "task-notification" and its own total_cost_usd. Agent/Workflow/Explore/Task are denied by +# DISALLOWED_TOOLS, so there should be no second result event here; if that fence is ever +# loosened this becomes an undercount, and this sentence is the warning. +# Prints 0 when the field is absent, so a timed-out or crashed sample contributes 0 rather +# than breaking the running total. +# +# Nothing in this file counted a billed call before this existed. A measurement that cannot +# say what it cost cannot be budgeted, and every prior dispatch number here was published +# without one. +# --------------------------------------------------------------------------- +top_level_cost() { + local jsonl="$1" + jq -r 'select(.type=="result" and (.origin==null)) | (.total_cost_usd // 0)' "$jsonl" 2>/dev/null | tail -1 +} + +# --------------------------------------------------------------------------- +# named_in_prose OUT_JSONL REGEX +# Prints 1 if the run's assistant TEXT names a skill matching REGEX without having called it. +# +# Measured 2026-08-27 (tests/evals/collision/RESULTS.md, "Why nothing fired"): when the tool a +# skill needs is denied -- Write for a plan document, Agent for a fan-out -- the model names +# the skill in prose and never calls Skill. Under `fired=[]` that scores identically to a +# routing miss. Separating the two is the difference between "the description does not route" +# and "the description routes and the fence blocks the payoff", and DISALLOWED_TOOLS denies +# exactly the tools the four chain skills exist to use. +# +# A 0/10 printed without this number beside it is uninterpretable. +# +# Herestring, not a pipe: `grep -q` closes the pipe early and returns 141 under `pipefail`. +# --------------------------------------------------------------------------- +named_in_prose() { + local jsonl="$1" regex="$2" txt + txt="$(jq -r 'select(.type=="assistant") | (.message.content // [])[] + | select(.type=="text") | .text' "$jsonl" 2>/dev/null)" + if grep -qE "$regex" <<<"$txt"; then echo 1; else echo 0; fi +} + # --------------------------------------------------------------------------- # hash_snapshot DIR # One "hash path" line per regular file under DIR (find -type f -- directories are excluded, @@ -406,9 +472,134 @@ selected_() { return 1 } +# --------------------------------------------------------------------------- +# Provenance for anything SAMPLES mode publishes. +# +# The rule in this repository is that an instrument is committed before the run it produces a +# number for -- a 55-sample collision result was withdrawn precisely because its harness edit +# was never committed and the runlog is gone (CHANGELOG.md, 1.47.0). Recording the instrument +# sha AND a digest of the description bytes in every sample row is what makes a published k/N +# checkable afterwards instead of merely asserted, because in this suite the *subject* of the +# measurement is the description text and the *instrument* is this file. Both have to be +# pinned or the number cannot be reproduced. +# +# The descriptions digested here are the INSTALLED ones, not the repo's: the CLI runs in a +# scratch workdir and reads ~/.claude/skills. Measured 2026-09-01: a project-local +# .claude/skills//SKILL.md IS loaded, but a user-level skill of the same name WINS -- +# the installed description is what reaches the matcher, and a repo edit is invisible here +# until it is installed. Digesting the wrong tree would pin a number to bytes the model never +# saw. +# --------------------------------------------------------------------------- +SAMPLE_HEAD="$(git -C "$AT_ROOT" rev-parse --short HEAD 2>/dev/null || echo unknown)" +SAMPLE_DESCS="$(grep -h '^description:' "$HOME"/.claude/skills/*/SKILL.md 2>/dev/null \ + | "$SUM_CMD" | cut -c1-12)" + +if (( SAMPLES > 0 )) \ + && [ -n "$(git -C "$AT_ROOT" status --porcelain -- tests/auto-trigger.sh 2>/dev/null)" ]; then + echo "WARNING: tests/auto-trigger.sh is uncommitted. A number from an uncommitted instrument" + echo " is inadmissible here. Commit the harness, then re-run." +fi + +# Cost guard. Nothing in this file counted a call before spending it. SAMPLES across the whole +# suite is SAMPLES x every case, which at SAMPLES=10 is several hundred billed invocations from +# one keystroke and no confirmation. +if (( SAMPLES > 0 )) && [ ${#SELECTED[@]} -eq 0 ] && [ "$SAMPLES_ALL" != "1" ]; then + echo "SAMPLES=$SAMPLES with no case named would run $SAMPLES x $(grep -cE '^run_(negative_)?case ' "$0") invocations." + echo "Name the cases you want, or set SAMPLES_ALL=1 to mean it." + exit 2 +fi + +# --------------------------------------------------------------------------- +# sample_case NAME PROMPT REGEX SETUP_FN POLARITY +# +# SAMPLES independent invocations of one prompt, no early stop, raw k/N. POLARITY is "pos" +# (REGEX names the skill that SHOULD fire) or "neg" (REGEX names skills that must NOT); it +# changes the printed label only, because a fire rate and a false-positive rate are the same +# arithmetic pointed at different prompts. +# +# Deliberately never touches PASS_COUNT/FAIL_COUNT except on a fence breach. A rate is a +# measurement, not a verdict: wiring a low rate to a red exit would make this suite refuse to +# report the very thing it was run to find out. +# --------------------------------------------------------------------------- +sample_case() { + local name="$1" prompt="$2" regex="$3" setup_fn="$4" polarity="$5" + local i fired term cost prose_hit hits=0 prose=0 cutoff=0 case_cost="0" + local workdir out_jsonl err_log runner_pid waited baseline baseline_hashes violations + local case_turns; case_turns="$(case_max_turns "$name")" + + for i in $(seq 1 "$SAMPLES"); do + workdir="$(mktemp -d "/tmp/auto-trigger-sample.XXXXXX")" + [[ -n "$setup_fn" ]] && "$setup_fn" "$workdir" + baseline="$(find "$workdir" -mindepth 1 2>/dev/null | sort)" + baseline_hashes="$(hash_snapshot "$workdir")" + out_jsonl="$workdir/.out.jsonl"; err_log="$workdir/.err.log" + + # Same exec/timeout discipline as run_case: without exec the kill -9 hits an empty parent + # and leaks a live, billed claude session per timed-out sample. + ( + cd "$workdir" || exit 1 + exec env -u ANTHROPIC_API_KEY -u ANTHROPIC_AUTH_TOKEN \ + claude -p "$prompt" \ + --output-format stream-json --verbose \ + --disallowedTools "$DISALLOWED_TOOLS" \ + --model "$MODEL" --max-turns "$case_turns" \ + < /dev/null > "$out_jsonl" 2> "$err_log" + ) & + runner_pid=$! + waited=0 + while kill -0 "$runner_pid" 2>/dev/null; do + sleep 1 + waited=$((waited + 1)) + if (( waited >= PER_CASE_TIMEOUT )); then kill -9 "$runner_pid" 2>/dev/null; break; fi + done + wait "$runner_pid" 2>/dev/null + + fired=""; term="no-output"; cost=0; prose_hit=0 + if [[ -s "$out_jsonl" ]]; then + fired="$(extract_fired_skills "$out_jsonl")" + term="$(top_level_subtype "$out_jsonl")" + cost="$(top_level_cost "$out_jsonl")" + prose_hit="$(named_in_prose "$out_jsonl" "$regex")" + fi + [[ "$term" == "error_max_turns" ]] && cutoff=$((cutoff + 1)) + if [[ -n "$fired" ]] && grep -qE "^($regex)$" <<<"$fired"; then + hits=$((hits + 1)) + elif [[ "$prose_hit" == "1" ]]; then + prose=$((prose + 1)) + fi + # bash 3.2 has no float arithmetic; awk does the addition. + case_cost="$(awk -v a="$case_cost" -v b="${cost:-0}" 'BEGIN{printf "%.6f", a+b}')" + SAMPLE_CALLS=$((SAMPLE_CALLS + 1)) + + violations="$(fence_violations "$workdir" "$baseline" "$baseline_hashes" "$out_jsonl" "$err_log")" + if [[ -n "$SAMPLE_LOG" ]]; then + printf '{"case":"%s","sample":%d,"polarity":"%s","fired":"%s","subtype":"%s","named_in_prose":%s,"cost_usd":%s,"model":"%s","max_turns":%s,"instrument":"%s","descs":"%s"}\n' \ + "$name" "$i" "$polarity" "$(tr '\n' ' ' <<<"$fired" | sed 's/ *$//')" "$term" \ + "$prose_hit" "${cost:-0}" "$MODEL" "$case_turns" "$SAMPLE_HEAD" "$SAMPLE_DESCS" \ + >> "$SAMPLE_LOG" + fi + rm -rf "$workdir" + + if [[ -n "$violations" ]]; then + echo "FAIL $name -> fence breach on sample $i of $SAMPLES: $(tr '\n' ';' <<<"$violations")" + FAIL_COUNT=$((FAIL_COUNT + 1)) + return + fi + done + + SAMPLE_COST="$(awk -v a="$SAMPLE_COST" -v b="$case_cost" 'BEGIN{printf "%.6f", a+b}')" + SAMPLE_CASES=$((SAMPLE_CASES + 1)) + printf 'SAMPLE %-30s %-3s fired %d/%d named-not-called %d/%d cut-off %d/%d $%s (max-turns=%s)\n' \ + "$name" "$polarity" "$hits" "$SAMPLES" "$prose" "$SAMPLES" "$cutoff" "$SAMPLES" \ + "$case_cost" "$case_turns" + HIT_LINES+=("$(printf '%-30s %-3s %d/%d fired, %d/%d named-only, %d/%d cut off, $%s' \ + "$name" "$polarity" "$hits" "$SAMPLES" "$prose" "$SAMPLES" "$cutoff" "$SAMPLES" "$case_cost")") +} + run_case() { local name="$1" prompt="$2" expected_regex="$3" setup_fn="$4" selected_ "$name" || return 0 + if (( SAMPLES > 0 )); then sample_case "$name" "$prompt" "$expected_regex" "$setup_fn" pos; return; fi local attempt fired fired_csv matched workdir out_jsonl err_log runner_pid waited local baseline baseline_hashes violations term_subtype last_fired_csv="" local case_turns; case_turns="$(case_max_turns "$name")" @@ -514,6 +705,9 @@ run_case() { run_negative_case() { local name="$1" prompt="$2" forbidden_regex="$3" selected_ "$name" || return 0 + # In SAMPLES mode the forbidden regex is exactly what we want a rate for: k/N here IS the + # over-trigger rate of the new descriptions, measured for free alongside the positive arms. + if (( SAMPLES > 0 )); then sample_case "$name" "$prompt" "$forbidden_regex" "" neg; return; fi local workdir out_jsonl err_log runner_pid waited fired fired_csv baseline baseline_hashes violations local case_turns; case_turns="$(case_max_turns "$name")" @@ -957,11 +1151,25 @@ echo "---" # which is the same shape as a clean run. A typo'd case name must not read as a pass. # This sits above the hit-rate loop because HIT_LINES is unset when nothing ran, and # `set -u` turns that into a stack trace rather than an answer. -if (( PASS_COUNT + FAIL_COUNT == 0 )); then +if (( PASS_COUNT + FAIL_COUNT + SAMPLE_CASES == 0 )); then echo "no case ran. ${#SELECTED[@]} selector(s) given, none matched a case name." exit 2 fi +if (( SAMPLES > 0 )); then + echo "Rates (raw k/N, no early stop):" + for l in "${HIT_LINES[@]}"; do echo " $l"; done + echo + echo "Sample mode: SAMPLES=$SAMPLES, $SAMPLE_CASES case(s), $SAMPLE_CALLS invocation(s), total_cost_usd=$SAMPLE_COST" + echo " model=$MODEL max-turns-default=$MAX_TURNS instrument=$SAMPLE_HEAD installed-descs=$SAMPLE_DESCS" + echo " fence=$DISALLOWED_TOOLS" + echo " k/N is a measurement, not a gate: a low rate does not fail this run. A fence breach does." + echo " Read named-not-called beside every k/N -- a 0/N with a high named-not-called is the" + echo " denied-affordance wall (tests/evals/collision/RESULTS.md), not a dead description." + if (( FAIL_COUNT > 0 )); then exit 1; fi + exit 0 +fi + echo "Hit rates (which attempt each case landed on):" for l in "${HIT_LINES[@]}"; do echo " $l"; done echo From 3a537cd2079a4c0a3df96c5a96b6f984e2b188c8 Mon Sep 17 00:00:00 2001 From: vk Date: Tue, 1 Sep 2026 20:16:14 +0530 Subject: [PATCH 02/10] Give writing-plans and TDD their own cases, and stop testing swarm's keyword writing-plans and test-driven-development had no isolated case. Their only positive coverage was feature-chain's three-way alternation across brainstorming|writing-plans|test-driven-development, which any one of the three satisfies alone. Both skills sit at zero lifetime dispatches across 2536 transcripts (tests/transcript-census.sh, re-run 2026-09-01) and this suite stayed green throughout. A gate that cannot go red for a dead skill is not measuring that skill. Each new case gets a setup_ function, because a fixture without one measures something else. Measured 2026-08-27 (tests/evals/collision/RESULTS.md): in an empty workdir the prompt has no referent, the model asks which repository is meant, and that scores identically to a routing miss. Every prompt here names files that exist. setup_spec seeds a SPEC.md that is settled and is explicitly not a plan, so brainstorming has nothing to explore and executing-plans has nothing to execute. The ask is for ordered steps in the reply rather than a file, because Write is denied by the fence and a fixture whose payoff needs a denied tool cannot distinguish a blocked affordance from a routing miss. setup_bugfix seeds a real defect plus a suite that demonstrably misses it. The prompt states the cause outright, which starves principle-fix-root-causes; the suite existing starves create-verification-skill and maintain-verification-skill. swarm-audit is rewritten rather than added. The old prompt was "Audit this directory three different ways at once" with no setup, and swarm's own description quoted "at once" back at it -- so the case measured whether the model echoes a keyword the description already contains, on an empty directory. That is a keyword test wearing a routing test's name. The new prompt seeds six non-importing integrations with six unrelated defects and states splittability as a property of the work. All four quoted trigger strings are gone. The SessionStart routing table still carries them; this case exercises the other path deliberately. Six count claims move with the suite. Four are live totals and are renumbered 28 -> 30. Two are historical measurements and are reworded instead: docs/how-skills-fire.md's routing-table arm and tests/evals/build-the-lever/PREREGISTRATION.md both describe what a past arm measured, and renumbering them to match today's tree would restate a finished measurement as something it was not. Breaking the number-noun adjacency is the device this repo already uses for "falsifiability rows"; extending .claude/verify.sh's exempt_phrases was not an option, that file is owned elsewhere this week. Verified: bash -n; shellcheck -S warning clean; tests/dispatch-static.sh green, 37 fixtures parsed clean (was 25), 30 cases declared and matched; .claude/verify.sh check 12 reads "30 test cases". The gate's four other failures (version, inventory, payload_digest, derived_at.head) reproduce identically on pristine 1bfb133 and are the in-flight release, not this branch. Co-Authored-By: Claude Opus 5 (1M context) --- README.md | 2 +- docs/how-skills-fire.md | 2 +- docs/what-this-actually-does.md | 2 +- tests/README.md | 4 +- tests/auto-trigger.sh | 160 +++++++++++++++++- .../evals/build-the-lever/PREREGISTRATION.md | 3 +- 6 files changed, 164 insertions(+), 9 deletions(-) diff --git a/README.md b/README.md index 80e08cb..1c27a44 100644 --- a/README.md +++ b/README.md @@ -203,7 +203,7 @@ CLI wrappers under `~/.config/agents/bin/`. Skills are not slash commands. They fire on the situation from their description — writing prose reaches for `unslop`, reviewing TypeScript reaches for `typescript-best-practices` — and -`tests/auto-trigger.sh` asserts that with 28 cases against the live model. +`tests/auto-trigger.sh` asserts that with 30 cases against the live model. ## Checks that can fail diff --git a/docs/how-skills-fire.md b/docs/how-skills-fire.md index 80a7053..0161002 100644 --- a/docs/how-skills-fire.md +++ b/docs/how-skills-fire.md @@ -75,7 +75,7 @@ its own. `tests/auto-trigger.sh` runs real prompts through the CLI and reports which skills fired. A version of the hook that dropped the routing table in favour of "the descriptions are the -triggers" was measured against 28 cases: +triggers" was measured against the 28 auto-trigger cases the suite carried on 2026-08-23: | Hook version | Result | |---|---| diff --git a/docs/what-this-actually-does.md b/docs/what-this-actually-does.md index 8873bb1..a01738e 100644 --- a/docs/what-this-actually-does.md +++ b/docs/what-this-actually-does.md @@ -61,7 +61,7 @@ The thing does what it says. Nobody has shown it changes an outcome. |---|---| | Does delegation rate fall off across a session's own lifetime? | `tests/delegation-drift.sh` exists, is pre-registered (first-third vs. last-third breadth-eligible delegation rate, with explicit floors on eligible windows and contributing sessions), and states its own reverse-causality confound up front: late-session work may just be less delegable, independent of anything the model does differently. It reports `NOT EVALUATED` rather than a rate below its own floors, the expected, correct result on day one, not a bug. | | Does auto-compaction correlate with worse behaviour in the turns after it? | `tests/compaction-effect.sh` is pre-registered the same way (is_error rate in a 15-tool-call window before and after a compaction boundary, with floors on boundary count and pooled calls) and states plainly that any result is correlational, not causal. A session long enough to hit auto-compaction is not a random draw from the same population as a short one. The one published result in this space, Governance Decay (arXiv 2606.22528), is from a different harness and does not transfer by citation alone. | -| Do the skills outside the measured subsets actually dispatch? | `tests/auto-trigger.sh` declares 28 cases. Category 1 above accounts for a 9-case routing-table ablation and 8 principle-prefix-naming fixes, whose re-run is "not yet authorised" per CHANGELOG.md, "1.35.0" (roughly 24 headless calls). The dispatch rate for the remaining cases is not separately recorded in this document's sources. | +| Do the skills outside the measured subsets actually dispatch? | `tests/auto-trigger.sh` declares 30 cases. Category 1 above accounts for a 9-case routing-table ablation and 8 principle-prefix-naming fixes, whose re-run is "not yet authorised" per CHANGELOG.md, "1.35.0" (roughly 24 headless calls). The dispatch rate for the remaining cases is not separately recorded in this document's sources. | | Does any of this improve output quality on a frontier model? | Three independent local measurements found no gain and no loss large enough to trust: the review pathway (category 1), a regression-scored SWE-bench Lite pass that tied all four arms at 0 of 2 resolved once neighbour-breaking counted, and the false-completion pilot. `docs/research/do-harnesses-help.md` names the honest reading directly: these are ceiling effects on small, well-specified tasks, which read as "no difference" when the accurate statement is "this could not tell." `docs/research/harness-value-literature-2026-08.md`, surveying roughly 70 published sources, reaches the same split independently: no well-powered published result shows a config-layer gain on issue-resolution tasks with a frontier model, while the tail-behaviour literature (test-exploitation, cheating, out-of-scope destructive actions) shows some of the largest effect sizes in the field. | ### The failures, not smoothed over diff --git a/tests/README.md b/tests/README.md index 5c4a81e..065b636 100644 --- a/tests/README.md +++ b/tests/README.md @@ -376,7 +376,7 @@ script prints `SKIP: ...` and exits 0. That is a valid, non-failing outcome, not a bug in the test. Each case runs in its own `mktemp -d` under `/tmp` (never this repo), so -nothing here pollutes the working tree the model sees. The test suite covers 28 cases: +nothing here pollutes the working tree the model sees. The test suite covers 30 cases: readme-writing, typescript-review, swarm-audit, blast-radius-auth, feature-chain, root-cause-guard, overnight-audit-trail, ui-iterate-styles, component-registry-combobox, idempotent-cron, negative-arithmetic, and negative-factual. @@ -389,7 +389,7 @@ idempotent-cron, negative-arithmetic, and negative-factual. never report `loggedIn: true` there. The script detects this and skips rather than failing the build. 2. **It would bill tokens.** Every case makes real API calls, up to 3 turns - each across 28 cases. Running this on every push or pull request in CI + each across 30 cases. Running this on every push or pull request in CI would spend real money on a check that mostly guards against skill-routing regressions. Those regressions are infrequent. Run the script by hand instead, or schedule it on a machine that already has an diff --git a/tests/auto-trigger.sh b/tests/auto-trigger.sh index 8a34fa7..080fea6 100755 --- a/tests/auto-trigger.sh +++ b/tests/auto-trigger.sh @@ -459,7 +459,7 @@ fence_violations() { # run_case NAME PROMPT EXPECTED_REGEX SETUP_FN # SETUP_FN is a function name invoked with the temp dir as $1, or "" for none. # --------------------------------------------------------------------------- -# Case selection. Without it, proving one fix means re-running all 28 cases and spending the +# Case selection. Without it, proving one fix means re-running all 30 cases and spending the # whole allowance to learn about eight of them. Matches install-matrix.sh's convention: # no arguments runs everything, arguments name the cases to run. # tests/auto-trigger.sh # all cases @@ -932,6 +932,122 @@ type Order struct { EOF } +# A spec that is settled and is NOT a plan. The distinction is the whole fixture: brainstorming's +# situation ("the shape is undecided") is starved by "agreed, no open questions", and +# executing-plans' situation ("a written plan already exists") is starved by the last line +# saying in as many words that no file map or ordering exists anywhere. +setup_spec() { + local dir="$1" + setup_webapp "$dir" + cat > "$dir/SPEC.md" <<'SPECEOF' +# Spec: offline mode for the notes app + +Status: agreed 2026-08-30. No open questions. The shape is not up for discussion. + +Behaviour: +- Notes stay editable with no network; edits queue locally. +- On reconnect the queue drains oldest first; a note edited in two places keeps the newer timestamp. +- The header reads exactly one of: online, offline, syncing. +- No new dependencies. Storage is localStorage. + +Out of scope: accounts, multi-device, any conflict UI beyond last-write-wins. + +Written down nowhere: which files change, in what order, or how each step gets checked. +SPECEOF +} + +# A real defect, a real suite, and a gap between them. The suite existing is what separates this +# from create-verification-skill ("nothing exists yet"); the suite being current is what +# separates it from maintain-verification-skill (a gate that drifted). +setup_bugfix() { + local dir="$1" + cat > "$dir/package.json" <<'PKGEOF' +{ + "name": "notes-app", + "version": "1.0.0", + "type": "module", + "scripts": { "test": "node --test" } +} +PKGEOF + cat > "$dir/notes.js" <<'NOTESEOF' +const notes = []; + +export function addNote(text) { + notes.push({ text, at: 0 }); + return notes.length; +} + +export function listNotes() { + return notes; +} + +export function clearNotes() { + notes.length = 0; +} +NOTESEOF + cat > "$dir/notes.test.js" <<'TESTEOF' +import { test } from "node:test"; +import assert from "node:assert"; +import { addNote, listNotes, clearNotes } from "./notes.js"; + +test("addNote stores the text it was given", () => { + clearNotes(); + addNote("buy milk"); + assert.strictEqual(listNotes()[0].text, "buy milk"); +}); +TESTEOF +} + +# Six integrations, six unrelated defects, no imports between them: a master-password backdoor, +# an unauthenticated non-idempotent refund, path traversal, a Math.random session id, an +# unverified webhook, and CSV injection. Six independent verdicts is real independent work, not +# one finding restated six times -- which is what makes this a swarm situation rather than a +# single review. +setup_audit() { + local dir="$1" + mkdir -p "$dir/services" + cat > "$dir/services/auth.js" <<'AUTHEOF' +export function login(user, pass) { + if (pass === process.env.MASTER_PASSWORD) return { user, admin: true }; + return null; +} +AUTHEOF + cat > "$dir/services/billing.js" <<'BILLEOF' +export function refund(chargeId, amount) { + return fetch("https://api.example.com/charges/" + chargeId + "/refund", { + method: "POST", + body: JSON.stringify({ amount }), + }); +} +BILLEOF + cat > "$dir/services/upload.js" <<'UPEOF' +export function saveUpload(name, bytes) { + return { path: "/var/uploads/" + name, size: bytes.length }; +} +UPEOF + cat > "$dir/services/session.js" <<'SESSEOF' +const sessions = new Map(); + +export function newSession(user) { + const id = String(Math.random()).slice(2); + sessions.set(id, user); + return id; +} +SESSEOF + cat > "$dir/services/webhook.js" <<'HOOKEOF' +export function handle(req) { + const body = JSON.parse(req.body); + return { ok: true, event: body.type }; +} +HOOKEOF + cat > "$dir/services/export.js" <<'CSVEOF' +export function toCsv(rows) { + return rows.map(function (r) { return Object.values(r).join(","); }).join("\n"); +} +CSVEOF +} + + # --------------------------------------------------------------------------- # Test cases # --------------------------------------------------------------------------- @@ -951,11 +1067,24 @@ run_case \ "typescript-best-practices" \ "setup_typescript" +# Rewritten 2026-09-01. The old prompt was "Audit this directory three different ways at once" +# with no setup, so it tested two things this suite is not for: whether the model obeys the +# literal string "at once" (which swarm's own description quoted back at it, making the case a +# keyword echo rather than a routing test), and whether it can audit an empty directory -- the +# no-referent confound in tests/evals/collision/RESULTS.md. +# +# All four quoted trigger strings are gone -- "in parallel", "at once", "all of these", +# "try N ways" -- and splittability is stated as a property of the WORK, which is the thing a +# description matcher can actually score. The routing-table line in the SessionStart hook still +# carries the literal strings; this case deliberately exercises the other path. +# +# If this measures 0/N with a high named-not-called, the blocker is the Agent denial in +# DISALLOWED_TOOLS, not the description, and the arm says so rather than reporting a dead skill. run_case \ "swarm-audit" \ - "Audit this directory three different ways at once and report back." \ + "Each file in services/ is a separate integration with its own failure mode, and none of them import each other. I want a security verdict per file, and what you find in one tells you nothing about the next. Come back with one verdict per file." \ "swarm" \ - "" + "setup_audit" run_case \ "blast-radius-auth" \ @@ -1135,6 +1264,31 @@ run_case \ "principle-type-system-discipline" \ "setup_gostruct" +# writing-plans and test-driven-development had no isolated case until 2026-09-01. Their only +# positive coverage was feature-chain's three-way alternation, which any one of the three can +# satisfy on its own -- so both could be, and were, dead at zero lifetime dispatches while this +# suite stayed green. A gate that cannot go red for a dead skill is not measuring that skill. + +# Discriminator, three ways. SPEC.md says "agreed ... no open questions", so brainstorming has +# nothing to explore. Nothing on disk is a plan, so executing-plans has nothing to execute. And +# the ask is for the ordered steps IN THE REPLY, not for a file: Write is denied by the fence, +# and a fixture whose payoff needs a denied tool scores a blocked affordance as a routing miss. +run_case \ + "writing-plans-spec-to-steps" \ + "SPEC.md is agreed and there are no open questions on it. Nothing anywhere says which files change, in what order, or how we would know each step worked. Work that out and give it to me before anyone touches app.js." \ + "writing-plans" \ + "setup_spec" + +# Discriminator: the cause is stated outright, which starves principle-fix-root-causes (its +# situation is debugging, or reaching for a try/except). A suite exists and demonstrably does not +# cover this path. "test" appears in the prompt only as a description of what is already on disk +# -- the ask is for the fix -- so a hit here is routing rather than keyword echo. +run_case \ + "tdd-known-bugfix" \ + "addNote('') files an empty note instead of rejecting it. The cause is not a mystery: addNote never looks at text. Wanted behaviour is that an empty or whitespace-only string is rejected and the list is left alone. notes.test.js covers the happy path only. Make the change." \ + "test-driven-development" \ + "setup_bugfix" + run_negative_case \ "negative-arithmetic" \ "What is 17 times 23? Just the number." \ diff --git a/tests/evals/build-the-lever/PREREGISTRATION.md b/tests/evals/build-the-lever/PREREGISTRATION.md index ef4c5ff..dc1b8cb 100644 --- a/tests/evals/build-the-lever/PREREGISTRATION.md +++ b/tests/evals/build-the-lever/PREREGISTRATION.md @@ -101,7 +101,8 @@ request does not move dispatch, the locus is not the constraint. `"Every file in modules/ needs the same license header pasted at the top. I'll go through and add it to each one by hand."` The user has announced they are doing the work themselves. Nothing is asked -of the model. Across all 28 cases in `tests/auto-trigger.sh`, exactly two prompts are declaratives +of the model. Across all 28 auto-trigger cases in `tests/auto-trigger.sh` at pre-registration time, +exactly two prompts are declaratives with no imperative and no question directed at the model — this one, and `prove-it-works-declare-done` (`"I fixed the crash in fetch_stats by adding a retry, and the code compiles cleanly -- this task is done."`). Those are also exactly the two cases measured at 0/10. From 5d48f6f0ed193854119ab8443cac4fbc32def159 Mon Sep 17 00:00:00 2001 From: vk Date: Tue, 1 Sep 2026 20:18:38 +0530 Subject: [PATCH 03/10] Score the shipped descriptions for shared trigger n-grams No model calls. Splits each description at its first ':', '.' or em dash into the discriminator clause and the rest, then scores shared 2- and 3-grams per pair, weighted 3/2/1 by whether the overlap lands in both first clauses, one, or neither. The only failing condition is a 3-gram opening two descriptions -- the defect aebeebb fixed by hand when grill-me and interrogate both led with "tear this apart". Everything else prints and passes. The header says what this is not, at length, because the claim it would be mistaken for is withdrawn. "Colliding triggers were measured over 80 samples to suppress both skills" has no surviving runlog and was retracted in 1.47.0; the pre-registered replacement at tests/evals/collision/ found the hypothesis unsupported on a committed instrument and located the real mechanism in the harness instead. So this encodes design discipline -- discriminator in the first clause, no two skills opening on the same words -- and explicitly not a dispatch law. Its own self-test found three defects in it before the live corpus did, which is the argument for carrying one: The stoplist was a wrapped shell string. BSD awk rejects a newline inside a -v assignment and exits non-zero having scored nothing, so the live corpus printed rc=0 with awk failing on every invocation -- a green that measured nothing, the exact shape docs/checks-that-inherit-their-answer catalogues. Now one line, with the reason on it. score() returned awk's status without distinguishing a parse error from "no collisions". It now refuses on rc>1 and on a corpus of fewer than two descriptions, rather than reporting a clean result it never computed. That guard is what caught the third defect on the first live run: BSD awk aborts on the em dash in several descriptions with "illegal byte sequence" under a UTF-8 locale. LC_ALL=C is now pinned, and the byte comparison in the HEAD/TAIL split is written against those bytes deliberately. The planted self-test collision was also wrong to begin with: "tear this apart" does not survive stopword removal, dropping to a 2-gram the check correctly declines to fail. The plant is content words now. Verified: bash -n; shellcheck -S warning clean; --self-test green both directions (planted collision fails and names the pair, clean corpus passes). Live corpus scores 23 across 28 skills, no hard-fail pairs, with the two ordinal-frame overlaps visible as test-driven-development/writing-plans "once shape" (6) and brainstorming/test-driven-development "shape known" (4). Co-Authored-By: Claude Opus 5 (1M context) --- tests/description-collision.sh | 237 +++++++++++++++++++++++++++++++++ 1 file changed, 237 insertions(+) create mode 100755 tests/description-collision.sh diff --git a/tests/description-collision.sh b/tests/description-collision.sh new file mode 100755 index 0000000..7be4f4c --- /dev/null +++ b/tests/description-collision.sh @@ -0,0 +1,237 @@ +#!/usr/bin/env bash +# tests/description-collision.sh +# +# Scores the shipped skill descriptions for shared trigger n-grams and names the pairs that +# overlap. No model calls, no network, no writes outside a self-test temp dir. +# +# WHAT THIS IS NOT +# +# This is not evidence that overlapping triggers suppress dispatch. That claim was published +# here as "measured over 80 samples to suppress both skills, not one" and was formally withdrawn +# in 1.47.0: no 80-sample runlog exists in this repository's history or on the machine that +# produced it (CHANGELOG.md, docs/research/fake-greens-2026-08.md). The replacement arm was +# pre-registered before the first sample and run on a committed instrument +# (tests/evals/collision/PREREGISTRATION.md, RESULTS.md, n=25, 2026-08-27); it found the +# hypothesis NOT supported -- the collision fixture fired more than either clean control -- and +# instead confirmed that the harness supplies no situation and denies the tools several skills +# exist to use. +# +# So this script encodes DESIGN DISCIPLINE, not a measured law. The discipline is the one +# aebeebb applied by hand to six pairs: put the discriminator in the first clause, because that +# is the part a matcher weighs hardest, and do not let two skills open on the same words. It is +# cheap, it is deterministic, and it is worth keeping green for the same reason a style rule is +# -- not because a violation is proven to break dispatch. +# +# SCORING +# +# Each description splits at its first ':', '.' or em dash into HEAD (the discriminator +# clause) and TAIL. Shared 2- and 3-grams are scored per pair, weighted by where they land: +# 3 for HEAD/HEAD, 2 for one HEAD, 1 for TAIL/TAIL. +# +# THE ONLY FAILING CONDITION +# +# A 3-gram appearing in BOTH skills' HEAD. That is exactly the defect aebeebb fixed, where +# grill-me and interrogate both opened on "tear this apart". Everything else prints and +# passes, and the threshold is echoed on every run so a green cannot be read as "no overlap". +# +# Usage: +# tests/description-collision.sh # score claude/skills/*/SKILL.md +# tests/description-collision.sh --self-test # prove the check can fail, then that it passes +# tests/description-collision.sh DIR # score an arbitrary skills dir +set -uo pipefail + +# One line, deliberately. BSD awk rejects a newline inside a -v assignment ("newline in +# string") and then exits non-zero having scored nothing -- which the first run of this +# script's own self-test caught, printing a clean rc=0 on the live corpus while awk had +# failed on every invocation. A wrapped stoplist is a green that measures nothing. +STOP=" use when the a an and or of to in on for it is are this that with before after one its you your any not no into from at by as be been has have had they them their we our i me my do does did so than then there here what which who whom whose will would can could should may might must if else while each per via over under again more most other some such only own same too very just now new old first second third next last thing things something anything nothing " + +# --------------------------------------------------------------------------- +# extract_descriptions DIR +# Prints "skillnamedescription" per skill. Uses the same closed-frontmatter-block rule as +# .claude/verify.sh check 3 rather than a bare grep: a line that merely looks like frontmatter +# further down a body must not be scored as a description. +# --------------------------------------------------------------------------- +extract_descriptions() { + local dir="$1" f name + for f in "$dir"/*/SKILL.md; do + [ -f "$f" ] || continue + name="$(basename "$(dirname "$f")")" + awk -v skill="$name" ' + NR==1 { if ($0 != "---") exit; infm=1; next } + infm && $0 == "---" { exit } + infm && /^description:[[:space:]]/ { + line=$0 + sub(/^description:[[:space:]]*/, "", line) + gsub(/^"|"$/, "", line) + printf "%s\t%s\n", skill, line + exit + } + ' "$f" + done +} + +# --------------------------------------------------------------------------- +# score SKILLS_DIR +# Emits the pair table and the corpus total on stdout. Returns 1 if any HEAD/HEAD 3-gram exists. +# All grouping lives in awk: bash 3.2 has no associative arrays, which is also why the rest of +# this suite avoids them. +# --------------------------------------------------------------------------- +score() { + local dir="$1" descs n_desc out rc + descs="$(extract_descriptions "$dir")" + n_desc="$(printf '%s' "$descs" | grep -c . || true)" + # An empty or unreadable corpus must not read as "no collisions". Same failure shape the + # gate-falsifiability work in this repo exists to catch: a check that passes without running. + if [ "${n_desc:-0}" -lt 2 ]; then + echo "REFUSING: extracted $n_desc description(s) from $dir -- nothing to compare." >&2 + return 2 + fi + # LC_ALL=C is load-bearing. Several descriptions carry an em dash, and BSD awk under a UTF-8 + # locale aborts on it with "illegal byte sequence" -- caught by the REFUSING guard below on the + # first live run, which is the only reason it was not published as a clean corpus. Under C the + # em dash is three bytes that [^a-z0-9]+ strips like any other punctuation, and the byte + # comparison in the HEAD/TAIL split below is written against those bytes deliberately. + out="$(printf '%s\n' "$descs" | LC_ALL=C awk -F'\t' -v stop="$STOP" ' + function norm(s, t) { + t = tolower(s) + gsub(/[^a-z0-9]+/, " ", t) + return t + } + function emit(skill, text, zone, n, w, i, j, tok, out) { + n = split(norm(text), w, " ") + j = 0 + for (i = 1; i <= n; i++) { + tok = w[i] + if (length(tok) < 3) continue + if (index(stop, " " tok " ") > 0) continue + j++; keep[j] = tok + } + for (i = 1; i <= j - 1; i++) { + g = keep[i] " " keep[i+1] + seen[g, skill] = zone; grams[g] = 1 + } + for (i = 1; i <= j - 2; i++) { + g = keep[i] " " keep[i+1] " " keep[i+2] + seen[g, skill] = zone; grams[g] = 1 + len[g] = 3 + } + } + { + skill = $1; desc = $2 + skills[++ns] = skill + # HEAD is everything up to the first ":", "." or em dash; TAIL is the rest. + cut = 0 + for (i = 1; i <= length(desc); i++) { + ch = substr(desc, i, 1) + two = substr(desc, i, 3) + if (ch == ":" || ch == "." || two == "\342\200\224") { cut = i; break } + } + if (cut == 0) { head = desc; tail = "" } + else { head = substr(desc, 1, cut - 1); tail = substr(desc, cut + 1) } + emit(skill, head, "H") + emit(skill, tail, "T") + } + END { + total = 0; hard = 0 + for (g in grams) { + nsk = 0 + for (i = 1; i <= ns; i++) { + s = skills[i] + if ((g SUBSEP s) in seen) { owners[++nsk] = s; zones[nsk] = seen[g, s] } + } + if (nsk < 2) continue + gl = (g in len) ? 3 : 2 + for (a = 1; a <= nsk - 1; a++) for (b = a + 1; b <= nsk; b++) { + za = zones[a]; zb = zones[b] + wt = (za == "H" && zb == "H") ? 3 : ((za == "H" || zb == "H") ? 2 : 1) + pair = (owners[a] < owners[b]) ? owners[a] " / " owners[b] : owners[b] " / " owners[a] + pscore[pair] += gl * wt + pgram[pair] = pgram[pair] (pgram[pair] == "" ? "" : ", ") g + total += gl * wt + if (za == "H" && zb == "H" && gl == 3) { + hard++ + hardline[hard] = pair ": \"" g "\" in both HEAD clauses" + } + } + } + for (p in pscore) printf "%6d %s: %s\n", pscore[p], p, pgram[p] | "sort -rn" + close("sort -rn") + printf "\ncorpus score %d across %d skills\n", total, ns + if (hard > 0) { + printf "\nHARD FAIL -- a 3-gram opens two descriptions:\n" + for (i = 1; i <= hard; i++) printf " %s\n", hardline[i] + exit 1 + } + printf "hard-fail pairs: none\n" + } + ')" + rc=$? + printf '%s\n' "$out" + # awk exiting 2 (a parse or runtime error) is not "no collisions found". Distinguish it from + # the deliberate exit 1, or a broken scorer reports a clean corpus forever. + if [ "$rc" -gt 1 ]; then + echo "REFUSING: awk failed (rc=$rc) -- the corpus was not scored." >&2 + return 2 + fi + return "$rc" +} + +# --------------------------------------------------------------------------- +# Self-test. CONTRIBUTING.md requires a check be seen failing before it is trusted, and this one +# carries its own mutation rather than leaning on tests/gate-falsifiability.sh, which is owned +# elsewhere. Plants a HEAD/HEAD 3-gram, asserts exit 1 and that the planted pair is named; then +# scores a clean corpus and asserts exit 0. +# --------------------------------------------------------------------------- +self_test() { + local t rc out fails=0 + t="$(mktemp -d "${TMPDIR:-/tmp}/desc-collision-selftest.XXXXXX")" + + mk() { + mkdir -p "$t/$1" + printf -- '---\nname: %s\ndescription: "%s"\n---\n# %s\n' "$1" "$2" "$1" > "$t/$1/SKILL.md" + } + + # The planted 3-gram has to survive normalisation to be a fair test of the scorer. "Tear this + # apart" does not: "this" is a stopword, so it drops to the 2-gram "tear apart" and the check + # correctly declines to fail. Content words only. + mk alpha "Shipping risky payment migration code today: review the blast radius first." + mk beta "Shipping risky payment migration code today: interview every reviewer first." + out="$(score "$t")"; rc=$? + if [ "$rc" -ne 1 ]; then + echo "SELF-TEST FAIL: planted HEAD/HEAD 3-gram did not fail the check (rc=$rc)"; fails=1 + elif ! grep -q 'alpha / beta' <<<"$out"; then + echo "SELF-TEST FAIL: failed, but did not name the planted pair"; fails=1 + else + echo "self-test 1/2 ok: planted collision fails and names alpha / beta" + fi + + rm -rf "${t:?}"/* + mk alpha "A settled spec with nothing written down: produce ordered steps." + mk beta "A running dev server and a component that renders wrong: screenshot every breakpoint." + out="$(score "$t")"; rc=$? + if [ "$rc" -ne 0 ]; then + echo "SELF-TEST FAIL: clean corpus was rejected (rc=$rc)"; echo "$out"; fails=1 + else + echo "self-test 2/2 ok: clean corpus passes" + fi + + rm -rf "$t" + return "$fails" +} + +REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" + +case "${1:-}" in + --self-test) self_test; exit $? ;; + "") DIR="$REPO_ROOT/claude/skills" ;; + *) DIR="$1" ;; +esac + +echo "Scoring shared trigger n-grams in $DIR" +echo "Failing condition: a 3-gram appearing in BOTH skills' first clause. Nothing else fails." +echo "This is design discipline, not a dispatch measurement -- see the header, and" +echo "tests/evals/collision/RESULTS.md for what a committed instrument actually found." +echo "---" +score "$DIR" +exit $? From c5087861841e6ca372a49f12fcd2b88e3e4c8316 Mon Sep 17 00:00:00 2001 From: vk Date: Tue, 1 Sep 2026 20:20:04 +0530 Subject: [PATCH 04/10] Make MAX_TURNS overridable, and add a guarded installer for measurement arms Two instrument gaps the baseline arm walked straight into. MAX_TURNS was a literal. The baseline run came back cut off by error_max_turns on 40 of 40 samples, which means two of the four arms scored 0/10 without the run ever reaching a decision point -- the exact turn-starvation signature that read encode-lessons-lint as a dead skill, and which this file has warned about above case_max_turns since 2026-08-23. Probing that needed either a source edit or a throwaway script, and the last time someone chose the throwaway script the resulting numbers were withdrawn for having no committed instrument. It is an env var now, and it lands in every runlog row so arms run at different budgets cannot be silently pooled. stage-skill-descriptions.sh exists because a description edited in a checkout is invisible to this harness. Measured 2026-09-01: tests/auto-trigger.sh runs the CLI in a scratch workdir, so the descriptions that reach the matcher are the installed ones under ~/.claude/skills, and while a project-local .claude/skills//SKILL.md IS loaded and listed, a user-level skill of the same name wins -- asked to quote its own listing for `swarm`, the model returned the installed text and reported no duplicate. An after-arm measured without installing would measure the before-arm's bytes and publish it as a result. Running install.sh instead is not available this week: it installs the whole tree, including another session's in-flight hook edits. Both directions refuse rather than guess. --stage aborts if any installed copy already differs from repo HEAD, because that means someone else is mid-edit and staging would carry their change into the arm; it pre-flights every named skill before touching any, so a half-applied stage cannot happen. --restore compares each file against the checksum it wrote and refuses that file if it changed, keeping the original rather than overwriting a foreign edit -- a save/restore harness in this repo has already destroyed two agents' uncommitted work by doing the opposite. Verified: bash -n; shellcheck -S warning clean on both; --status reads the live installed digest (d4251fb4eda7) and reports nothing staged. Co-Authored-By: Claude Opus 5 (1M context) --- tests/auto-trigger.sh | 7 +- tests/stage-skill-descriptions.sh | 135 ++++++++++++++++++++++++++++++ 2 files changed, 141 insertions(+), 1 deletion(-) create mode 100755 tests/stage-skill-descriptions.sh diff --git a/tests/auto-trigger.sh b/tests/auto-trigger.sh index 080fea6..b9c2db7 100755 --- a/tests/auto-trigger.sh +++ b/tests/auto-trigger.sh @@ -13,7 +13,12 @@ set -uo pipefail PER_CASE_TIMEOUT=120 # seconds; enforced by the polling loop in run_case (macOS has no timeout(1)) MODEL="sonnet" -MAX_TURNS=3 +# Env-overridable as of 2026-09-01, for probing turn starvation without editing the file. +# The 2026-08-23 note above case_max_turns records the last such probe being run through a +# throwaway script precisely because this was a literal -- and an uncommitted instrument is why +# that arm's numbers were withdrawn. The value lands in every SAMPLES runlog row, so an arm run +# at a different budget cannot be silently mixed with one run at the default. +MAX_TURNS="${MAX_TURNS:-3}" # Per-case turn budget, measured not guessed (2026-08-23), enforced by case_max_turns() below. # MAX_TURNS above is still the global default -- a case with no row here is not in the case # statement below either, and inherits MAX_TURNS unchanged, same as before this table existed. diff --git a/tests/stage-skill-descriptions.sh b/tests/stage-skill-descriptions.sh new file mode 100755 index 0000000..5902611 --- /dev/null +++ b/tests/stage-skill-descriptions.sh @@ -0,0 +1,135 @@ +#!/usr/bin/env bash +# tests/stage-skill-descriptions.sh +# +# Copies the repo's SKILL.md description line for named skills into ~/.claude/skills, and puts +# it back afterwards. +# +# WHY THIS HAS TO EXIST +# +# tests/auto-trigger.sh runs the CLI in a scratch workdir, so the descriptions that reach the +# matcher are the INSTALLED ones under ~/.claude/skills. Measured 2026-09-01: a project-local +# .claude/skills//SKILL.md is loaded and listed, but a user-level skill of the same name +# WINS -- asked to quote its own listing for `swarm`, the model returned the installed text and +# reported no duplicate. So a description edited in a checkout is invisible to the harness, and +# an after-arm measured without installing would measure the before-arm's bytes and call it a +# result. +# +# Running install.sh instead is not an option here: it installs the whole tree, including +# whatever hook edits another session has in flight. +# +# WHY THE RESTORE REFUSES +# +# A save/restore harness in this repo once deleted two agents' uncommitted work by restoring +# over a change it did not make. --restore therefore refuses any file whose content differs from +# what --stage wrote, and says so, rather than overwriting a foreign edit. The snapshot records +# a checksum per file, not just the bytes. +# +# --stage likewise refuses to start if the installed copy differs from the repo copy at HEAD, +# because that means someone else is mid-edit and staging would silently take their change with +# it into a measurement. +# +# Usage: +# tests/stage-skill-descriptions.sh --stage swarm writing-plans +# tests/stage-skill-descriptions.sh --status +# tests/stage-skill-descriptions.sh --restore +set -uo pipefail + +REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +SKILLS_SRC="$REPO_ROOT/claude/skills" +SKILLS_DST="$HOME/.claude/skills" +STATE_DIR="${TMPDIR:-/tmp}/vstack-skill-stage" + +if command -v shasum >/dev/null 2>&1; then SUM=shasum +elif command -v sha256sum >/dev/null 2>&1; then SUM=sha256sum +else echo "need shasum or sha256sum" >&2; exit 2; fi + +sumof() { "$SUM" < "$1" | awk '{print $1}'; } + +desc_of() { + awk 'NR==1 { if ($0 != "---") exit; infm=1; next } + infm && $0 == "---" { exit } + infm && /^description:[[:space:]]/ { print; exit }' "$1" +} + +do_stage() { + local name src dst staged=0 + [ "$#" -gt 0 ] || { echo "--stage needs at least one skill name" >&2; exit 2; } + mkdir -p "$STATE_DIR" + if [ -e "$STATE_DIR/manifest" ]; then + echo "REFUSING: $STATE_DIR/manifest already exists -- a previous stage was never restored." >&2 + echo " Run --restore (or --status) first." >&2 + exit 2 + fi + + # Pre-flight every skill before touching any of them: a half-applied stage is worse than none. + for name in "$@"; do + src="$SKILLS_SRC/$name/SKILL.md"; dst="$SKILLS_DST/$name/SKILL.md" + [ -f "$src" ] || { echo "REFUSING: no repo skill at $src" >&2; exit 2; } + [ -f "$dst" ] || { echo "REFUSING: no installed skill at $dst" >&2; exit 2; } + if ! git -C "$REPO_ROOT" show "HEAD:claude/skills/$name/SKILL.md" 2>/dev/null \ + | diff -q - "$dst" >/dev/null 2>&1; then + echo "REFUSING: installed $name differs from repo HEAD. Someone else is mid-edit, or an" >&2 + echo " earlier stage leaked. Staging would carry their change into the arm." >&2 + exit 2 + fi + done + + : > "$STATE_DIR/manifest" + for name in "$@"; do + src="$SKILLS_SRC/$name/SKILL.md"; dst="$SKILLS_DST/$name/SKILL.md" + cp "$dst" "$STATE_DIR/$name.orig" + cp "$src" "$dst" + printf '%s\t%s\t%s\n' "$name" "$dst" "$(sumof "$dst")" >> "$STATE_DIR/manifest" + staged=$((staged + 1)) + printf 'staged %-28s %s\n' "$name" "$(desc_of "$dst" | cut -c1-72)..." + done + echo "staged $staged skill(s). Installed digest is now: $(installed_digest)" + echo "Run --restore when the arm is finished." +} + +installed_digest() { + grep -h '^description:' "$SKILLS_DST"/*/SKILL.md 2>/dev/null | "$SUM" | cut -c1-12 +} + +do_restore() { + local name dst want have rc=0 n=0 + [ -f "$STATE_DIR/manifest" ] || { echo "nothing staged (no $STATE_DIR/manifest)"; exit 0; } + while IFS=$'\t' read -r name dst want; do + [ -n "$name" ] || continue + have="$(sumof "$dst")" + if [ "$have" != "$want" ]; then + echo "REFUSING $name: $dst changed since it was staged." >&2 + echo " staged=$want now=$have -- restoring would destroy a foreign edit." >&2 + echo " Original is preserved at $STATE_DIR/$name.orig" >&2 + rc=1 + continue + fi + cp "$STATE_DIR/$name.orig" "$dst" + n=$((n + 1)) + printf 'restored %-28s\n' "$name" + done < "$STATE_DIR/manifest" + if [ "$rc" -eq 0 ]; then + rm -rf "$STATE_DIR" + echo "restored $n skill(s), state cleared. Installed digest: $(installed_digest)" + else + echo "restored $n skill(s); state KEPT because at least one refused." >&2 + fi + return "$rc" +} + +do_status() { + echo "installed digest: $(installed_digest)" + if [ -f "$STATE_DIR/manifest" ]; then + echo "STAGED (not yet restored):" + cut -f1 "$STATE_DIR/manifest" | sed 's/^/ /' + else + echo "nothing staged" + fi +} + +case "${1:---status}" in + --stage) shift; do_stage "$@" ;; + --restore) do_restore ;; + --status) do_status ;; + *) echo "usage: $0 --stage NAME... | --restore | --status" >&2; exit 2 ;; +esac From 9803ff7927445b7d0f94095c8c2dd4837e88f030 Mon Sep 17 00:00:00 2001 From: vk Date: Tue, 1 Sep 2026 20:46:01 +0530 Subject: [PATCH 05/10] Fix two review defects, and publish the baseline arm Review found both of these after the baseline ran and before any after-arm, which is the only reason the numbers below are still usable. The runlog row was built by printf with a bare %s for `fired`. That value comes from .input.skill, a model-controlled tool-call argument, and nothing constrains it to be quote-free; a value containing a double quote emits a line that fails `jq .`. The runlog is the thing the provenance block calls the reason a published k/N is checkable, so a log its own subject can break is not a record. It is built by `jq -cn --arg` now, verified valid for values containing a quote, a backslash and a newline, with the old printf form reproducing the failure on the same input. No baseline row was affected: every `fired` value across all 60 rows is a plain skill name and the files parse. stage-skill-descriptions.sh had unchecked cp calls in both directions. On the stage side a failed copy would still write a manifest entry whose checksum is of the unchanged destination, so an arm would run on stale bytes while the runlog swore they were new. On the restore side a failed copy still printed "restored" and left rc at 0, after which the rm -rf deleted the only surviving backup. Both are checked and both refuse now. That script took no part in the baseline. Neither fix changes what a sample measures, so the arms stay comparable. THE BASELINE, N=10 per arm, no early stop, unedited descriptions. At the suite default of max-turns=3, 40 calls, $3.17: writing-plans 9/10, swarm 10/10, executing-plans 0/10, test-driven-development 0/10. Every sample terminated error_max_turns, 40 of 40, including both arms that fired -- so being cut off separates nothing and cannot be offered as the reason for the two zeros. It is reason enough to re-take them. At max-turns=12, 20 calls, $5.61, cut-off drops to 0/10: test-driven-development 3/10, executing-plans 1/10. Turn budget was carrying part of the TDD result and almost none of the executing-plans one. executing-plans is the finding. Its prompt and its unedited description are close to the same sentence, and it fires once in ten. Whatever suppresses it, it is not that the description fails to describe the situation. Two of its nine misses fired test-driven-development instead, on a prompt containing no test, which is the only direct evidence in this run that those two descriptions compete for the same prompts. Two of the brief's premises do not survive. writing-plans fires 9/10 unedited. swarm fires 10/10 on a prompt carrying none of the four trigger strings its own description quotes, so the literals are not what carries it. The transcript-census zeros are real; they are about how real prompts are phrased, and this harness cannot separate that from how descriptions are worded. RESULTS.md says so rather than claiming the credit. Verified: .claude/verify.sh check 12 reads 30 test cases and check 31 (referrers) is green at 164 files. The four remaining gate failures -- version, inventory, payload_digest, derived_at.head -- reproduce identically on pristine 1bfb133 and belong to the in-flight release. Co-Authored-By: Claude Opus 5 (1M context) --- tests/README.md | 72 ++++++++++ tests/auto-trigger.sh | 18 ++- tests/evals/autofire/RESULTS.md | 136 ++++++++++++++++++ ...-turns12-executing-plans-checkpoints.jsonl | 10 ++ ...og-baseline-turns12-tdd-known-bugfix.jsonl | 10 ++ ...e-turns3-executing-plans-checkpoints.jsonl | 10 ++ .../runlog-baseline-turns3-swarm-audit.jsonl | 10 ++ ...log-baseline-turns3-tdd-known-bugfix.jsonl | 10 ++ ...e-turns3-writing-plans-spec-to-steps.jsonl | 10 ++ tests/stage-skill-descriptions.sh | 20 ++- 10 files changed, 300 insertions(+), 6 deletions(-) create mode 100644 tests/evals/autofire/RESULTS.md create mode 100644 tests/evals/autofire/runlog-baseline-turns12-executing-plans-checkpoints.jsonl create mode 100644 tests/evals/autofire/runlog-baseline-turns12-tdd-known-bugfix.jsonl create mode 100644 tests/evals/autofire/runlog-baseline-turns3-executing-plans-checkpoints.jsonl create mode 100644 tests/evals/autofire/runlog-baseline-turns3-swarm-audit.jsonl create mode 100644 tests/evals/autofire/runlog-baseline-turns3-tdd-known-bugfix.jsonl create mode 100644 tests/evals/autofire/runlog-baseline-turns3-writing-plans-spec-to-steps.jsonl diff --git a/tests/README.md b/tests/README.md index 065b636..454e0c8 100644 --- a/tests/README.md +++ b/tests/README.md @@ -332,6 +332,25 @@ hand, the same status `bin-scripts.sh` and `container-matrix.sh` started from. tests/hook-latency.sh ``` +## evals/autofire/ + +Whether the four planning-chain skills route from a situation with no user instruction naming +them. `tests/evals/autofire/RESULTS.md` carries the arm, its reporting rules and its cost. + +The raw samples, one JSON object per line, written by `auto-trigger.sh` in SAMPLES mode: + +- `tests/evals/autofire/runlog-baseline-turns3-writing-plans-spec-to-steps.jsonl` +- `tests/evals/autofire/runlog-baseline-turns3-swarm-audit.jsonl` +- `tests/evals/autofire/runlog-baseline-turns3-executing-plans-checkpoints.jsonl` +- `tests/evals/autofire/runlog-baseline-turns3-tdd-known-bugfix.jsonl` +- `tests/evals/autofire/runlog-baseline-turns12-executing-plans-checkpoints.jsonl` +- `tests/evals/autofire/runlog-baseline-turns12-tdd-known-bugfix.jsonl` + +Read the turns=3 and turns=12 tables together or not at all. Every turns=3 sample terminated +`error_max_turns`, including the arms that fired, so being cut off separates nothing there. That +is why two of those four numbers were re-taken at a budget nothing exhausted rather than +published as results. + ## evals/ `evals/run-pathways.sh` and `evals/swebench/run.sh` score this bundle against @@ -568,6 +587,59 @@ Known gap: `hooks.json` names two of the eight scripts under `claude/hooks/` dir plugin lane carries routing only. The rest are invoked from inside those or wired through `claude/settings.json` in the full install, so renaming one of them is not caught here. +## description-collision.sh + +Scores every shipped skill description for shared trigger n-grams and names the overlapping +pairs. No model calls, no network. Splits each description at its first colon, period or em dash +into the discriminator clause and the rest, then weights shared 2- and 3-grams by where they +land: 3 for both first clauses, 2 for one, 1 for neither. + +The only failing condition is a 3-gram opening two descriptions. That is the defect `aebeebb` +fixed by hand, where `grill-me` and `interrogate` both led with "tear this apart". Everything +else prints and passes, and the threshold is echoed on every run so nobody reads a green as "no +overlap". + +Read the header before citing it. This is design discipline, not a dispatch law. The claim that +colliding triggers suppress both skills was withdrawn in 1.47.0 for having no surviving runlog, +and the pre-registered replacement at `evals/collision/` found the hypothesis unsupported. + +``` +tests/description-collision.sh # score claude/skills +tests/description-collision.sh --self-test # prove it can fail, then that it passes +``` + +`--self-test` plants a collision that survives stopword normalisation, asserts the check fails +and names the pair, then scores a clean corpus and asserts it passes. It found three defects in +its own scorer before the live corpus did: a wrapped stoplist that made BSD awk exit without +scoring anything, a `score()` that returned awk's status without telling a parse error apart from +"no collisions", and an em dash that aborts BSD awk under a UTF-8 locale. `LC_ALL=C` is pinned +for the third. + +## stage-skill-descriptions.sh + +Installs the repo's description line for named skills into `~/.claude/skills`, and puts it back. + +`auto-trigger.sh` runs the CLI in a scratch workdir, so the descriptions that reach the matcher +are the installed ones. Measured 2026-09-01: a project-local `.claude/skills//SKILL.md` is +loaded and listed, but a user-level skill of the same name wins. Asked to quote its own listing +for `swarm`, the model returned the installed text and reported no duplicate. So an after-arm +measured without installing would measure the before-arm's bytes and publish it as a result. +Running `install.sh` instead installs the whole tree, including whatever another session has in +flight. + +``` +tests/stage-skill-descriptions.sh --stage swarm writing-plans +tests/stage-skill-descriptions.sh --status +tests/stage-skill-descriptions.sh --restore +``` + +Both directions refuse rather than guess. `--stage` aborts if any installed copy already differs +from repo HEAD, and pre-flights every named skill before touching any, so a half-applied stage +cannot happen. `--restore` compares each file against the checksum it wrote and refuses that file +if it changed, keeping the original instead of overwriting a foreign edit. Every `cp` is checked: +without that, a failed copy would record a manifest checksum of the unchanged destination, and +the restore path would delete the only backup after failing to use it. + ## dispatch-fleet.sh `dispatch-fleet.sh` measures the 54-fixture set in `~/vstack-dispatch/` — recall, precision, diff --git a/tests/auto-trigger.sh b/tests/auto-trigger.sh index b9c2db7..5c5a648 100755 --- a/tests/auto-trigger.sh +++ b/tests/auto-trigger.sh @@ -578,9 +578,21 @@ sample_case() { violations="$(fence_violations "$workdir" "$baseline" "$baseline_hashes" "$out_jsonl" "$err_log")" if [[ -n "$SAMPLE_LOG" ]]; then - printf '{"case":"%s","sample":%d,"polarity":"%s","fired":"%s","subtype":"%s","named_in_prose":%s,"cost_usd":%s,"model":"%s","max_turns":%s,"instrument":"%s","descs":"%s"}\n' \ - "$name" "$i" "$polarity" "$(tr '\n' ' ' <<<"$fired" | sed 's/ *$//')" "$term" \ - "$prose_hit" "${cost:-0}" "$MODEL" "$case_turns" "$SAMPLE_HEAD" "$SAMPLE_DESCS" \ + # jq builds the row, not printf. `fired` comes from .input.skill, a model-controlled + # tool-call argument: nothing constrains it to be quote-free, and a bare %s interpolation + # of a value containing a double quote emits a line that fails `jq .`. That would corrupt + # the runlog silently -- the same runlog the provenance block above calls the thing that + # makes a published k/N checkable. A measurement log that can be broken by its own subject + # is not a record. + jq -cn \ + --arg case "$name" --argjson sample "$i" --arg polarity "$polarity" \ + --arg fired "$(tr '\n' ' ' <<<"$fired" | sed 's/ *$//')" --arg subtype "$term" \ + --argjson named_in_prose "${prose_hit:-0}" --argjson cost_usd "${cost:-0}" \ + --arg model "$MODEL" --argjson max_turns "$case_turns" \ + --arg instrument "$SAMPLE_HEAD" --arg descs "$SAMPLE_DESCS" \ + '{case:$case,sample:$sample,polarity:$polarity,fired:$fired,subtype:$subtype, + named_in_prose:$named_in_prose,cost_usd:$cost_usd,model:$model, + max_turns:$max_turns,instrument:$instrument,descs:$descs}' \ >> "$SAMPLE_LOG" fi rm -rf "$workdir" diff --git a/tests/evals/autofire/RESULTS.md b/tests/evals/autofire/RESULTS.md new file mode 100644 index 0000000..e24fb16 --- /dev/null +++ b/tests/evals/autofire/RESULTS.md @@ -0,0 +1,136 @@ +# Autofire arm: do the four chain skills route from the situation alone? + +**Status: baseline complete. After-arm pending. Do not cite the before/after comparison until the +after table is filled in.** + +| | | +|---|---| +| Question | Do `writing-plans`, `test-driven-development`, `executing-plans` and `swarm` fire from a situation, with no user instruction naming them? | +| Instrument | `tests/auto-trigger.sh` SAMPLES mode. Every row logs the repo HEAD it ran at, `901bb3f`, which contains the mode itself (`d6ca391`) and the fixtures. | +| Model | `sonnet` (CLI alias), OAuth session, `ANTHROPIC_API_KEY` stripped by the harness | +| Tool fence | `Write,Edit,MultiEdit,NotebookEdit,Bash,Agent,Workflow,Explore,Task` | +| Seed / temperature | Neither. The CLI exposes no such knob, which is why n>1 exists. | +| Installed descriptions | digest `d4251fb4eda7` on all 60 baseline rows, one value, no pooling | +| Sampling | N=10 independent invocations per arm, no early stop. `ATTEMPTS` is unused. | + +Raw k/N only. At n=10 a percentage or an interval would imply precision this sample does not +have, so the harness prints k/N. + +Runlogs, one JSON object per sample: +`runlog-baseline-turns3-writing-plans-spec-to-steps.jsonl`, +`runlog-baseline-turns3-swarm-audit.jsonl`, +`runlog-baseline-turns3-executing-plans-checkpoints.jsonl`, +`runlog-baseline-turns3-tdd-known-bugfix.jsonl`, +`runlog-baseline-turns12-executing-plans-checkpoints.jsonl`, +`runlog-baseline-turns12-tdd-known-bugfix.jsonl`. + +## Baseline at the suite default, max-turns=3 + +40 calls, $3.17. + +| case | skill | fired | named-not-called | cut off | +|---|---|---|---|---| +| `writing-plans-spec-to-steps` | `writing-plans` | 9/10 | 0/10 | 10/10 | +| `swarm-audit` | `swarm` | 10/10 | 0/10 | 10/10 | +| `executing-plans-checkpoints` | `executing-plans` | 0/10 | 0/10 | 10/10 | +| `tdd-known-bugfix` | `test-driven-development` | 0/10 | 0/10 | 10/10 | + +Every sample terminated `error_max_turns`, 40 of 40, including the two arms that fired. So being +cut off separates nothing here, and it cannot be offered as the reason the other two scored zero. +The runlog carries `fired` and `subtype` and no turn trace, so it cannot say where in a run the +decision was or was not reached. What it does say is that the two zeros were taken at a budget +that every single sample exhausted, which is enough reason to re-run them and not enough reason +to publish them. + +## The same two arms at max-turns=12 + +20 calls, $5.61. `MAX_TURNS` became env-overridable at `17f918b` so this could run on a committed +instrument. The last person to probe turn budget here used a throwaway script and the numbers +were withdrawn for exactly that. + +| case | skill | fired | named-not-called | cut off | misroutes | +|---|---|---|---|---|---| +| `tdd-known-bugfix` | `test-driven-development` | 3/10 | 0/10 | 0/10 | none | +| `executing-plans-checkpoints` | `executing-plans` | 1/10 | 0/10 | 0/10 | 2/10 to `test-driven-development` | + +Cut-off drops to zero at this budget, so both arms now run to completion and choose. Turn budget +was carrying part of the TDD result: 0/10 to 3/10. It was carrying almost none of the +`executing-plans` result: 0/10 to 1/10. + +`executing-plans` is the finding. Its prompt is "PLAN.md is the implementation plan we approved +last session. Execute it, and pause for my review after each phase." Its unedited description is +"Use when you have a written implementation plan to execute in a separate session with review +checkpoints." Those are close to the same sentence, and it still fires 1 time in 10. Whatever is +suppressing this skill, it is not that the description fails to describe the situation. Two of +the nine misses went to `test-driven-development` instead, on a prompt with no test in it, which +is the one piece of direct evidence in this run that these two descriptions compete. + +## What the baseline overturns + +The brief behind this work said `writing-plans`, `test-driven-development` and `executing-plans` +have fired zero times ever, and that `swarm` fires only when the user says "in parallel". The +first half is true of the transcript corpus (`tests/transcript-census.sh`, 2536 transcripts, +re-run 2026-09-01) and I am not disputing it. + +Little of it carries over to this harness. `writing-plans` fires 9/10 on its unedited +description. `swarm` fires 10/10 on a prompt carrying none of the four trigger strings its own +description quotes, so "in parallel", "at once", "all of these" and "try N ways" are not what +carries it. For those two there is no description defect visible here and no headroom to fix one. +`test-driven-development` is weak rather than dead. Only `executing-plans` is close to dead, and +it is closest to dead where its description matches its prompt best. + +### The caveat that limits all of this + +`writing-plans-spec-to-steps` shares wording with the description it tests. The fixture says +"SPEC.md is agreed" against a description reading "once the shape is agreed", and "before anyone +touches app.js" against "before code". That is not a quotation, but it is the same two hinges, +and a fixture built from a description's own hinges will fire more often than a real request +does. + +The gap between 9/10 here and zero in the real corpus is best explained by real prompts not +stating the situation this plainly. That is a claim about how people phrase requests, not about +how the description is worded, and this arm cannot separate the two. + +Do not read this file as "the skills are fine". It says something narrower. Given a prompt that +states the situation outright, two of four route reliably, one routes weakly, one barely routes +at all, and the harness could not have told you any of that before, because its fixtures had no +referent and its only knob stopped at the first hit. + +## Collision score + +From `tests/description-collision.sh`. No model calls. + +| | corpus score | hard-fail pairs | +|---|---|---| +| unedited (28 skills) | 23 | none | +| with the four rewrites | 13 | none | + +The 10-point drop is exactly the two overlaps the ordinal framing created: +`test-driven-development` against `writing-plans` on "once shape", worth 6, and `brainstorming` +against `test-driven-development` on "shape known", worth 4. Three descriptions opening +`Nth, once the shape is X` compete on the same tokens in the clause a matcher weighs hardest, +and the `executing-plans` arm shows `test-driven-development` taking prompts that are not its +own. + +This is design discipline, not a dispatch law. The claim that colliding triggers suppress both +skills was withdrawn in 1.47.0, and the pre-registered replacement found the hypothesis +unsupported (`tests/evals/collision/RESULTS.md`). + +## Cost + +The harness records this per arm. Nobody estimated it. Baseline at turns=3 cost $3.17 across 40 +calls, $0.079 each. The turns=12 re-run cost $5.61 across 20 calls, $0.281 each. + +## Instrument changes after the baseline, and why they do not invalidate it + +Two defects were found by review after the baseline ran and fixed before the after-arm. + +The runlog row was built by `printf` with a bare `%s` for `fired`, which comes from a +model-controlled tool-call argument. A value containing a double quote emits a line that fails +`jq .`. It is built by `jq -cn --arg` now. No baseline row was affected: every `fired` value in +all 60 rows is a plain skill name, and the file parses. + +`stage-skill-descriptions.sh` had unchecked `cp` calls in both directions. That script took no +part in the baseline, which ran against the installed descriptions unmodified. + +Neither change alters what a sample measures, so the baseline and after arms remain comparable. diff --git a/tests/evals/autofire/runlog-baseline-turns12-executing-plans-checkpoints.jsonl b/tests/evals/autofire/runlog-baseline-turns12-executing-plans-checkpoints.jsonl new file mode 100644 index 0000000..3dfa85a --- /dev/null +++ b/tests/evals/autofire/runlog-baseline-turns12-executing-plans-checkpoints.jsonl @@ -0,0 +1,10 @@ +{"case":"executing-plans-checkpoints","sample":1,"polarity":"pos","fired":"","subtype":"success","named_in_prose":0,"cost_usd":0.25070460000000006,"model":"sonnet","max_turns":12,"instrument":"17f918b","descs":"d4251fb4eda7"} +{"case":"executing-plans-checkpoints","sample":2,"polarity":"pos","fired":"","subtype":"success","named_in_prose":0,"cost_usd":0.2707798,"model":"sonnet","max_turns":12,"instrument":"17f918b","descs":"d4251fb4eda7"} +{"case":"executing-plans-checkpoints","sample":3,"polarity":"pos","fired":"","subtype":"success","named_in_prose":0,"cost_usd":0.368047,"model":"sonnet","max_turns":12,"instrument":"17f918b","descs":"d4251fb4eda7"} +{"case":"executing-plans-checkpoints","sample":4,"polarity":"pos","fired":"","subtype":"success","named_in_prose":0,"cost_usd":0.2650154,"model":"sonnet","max_turns":12,"instrument":"17f918b","descs":"d4251fb4eda7"} +{"case":"executing-plans-checkpoints","sample":5,"polarity":"pos","fired":"test-driven-development","subtype":"success","named_in_prose":0,"cost_usd":0.3514748,"model":"sonnet","max_turns":12,"instrument":"17f918b","descs":"d4251fb4eda7"} +{"case":"executing-plans-checkpoints","sample":6,"polarity":"pos","fired":"","subtype":"success","named_in_prose":0,"cost_usd":0.310086,"model":"sonnet","max_turns":12,"instrument":"17f918b","descs":"d4251fb4eda7"} +{"case":"executing-plans-checkpoints","sample":7,"polarity":"pos","fired":"executing-plans","subtype":"success","named_in_prose":1,"cost_usd":0.42729400000000006,"model":"sonnet","max_turns":12,"instrument":"17f918b","descs":"d4251fb4eda7"} +{"case":"executing-plans-checkpoints","sample":8,"polarity":"pos","fired":"test-driven-development","subtype":"success","named_in_prose":0,"cost_usd":0.36620100000000005,"model":"sonnet","max_turns":12,"instrument":"17f918b","descs":"d4251fb4eda7"} +{"case":"executing-plans-checkpoints","sample":9,"polarity":"pos","fired":"","subtype":"success","named_in_prose":0,"cost_usd":0.24200100000000002,"model":"sonnet","max_turns":12,"instrument":"17f918b","descs":"d4251fb4eda7"} +{"case":"executing-plans-checkpoints","sample":10,"polarity":"pos","fired":"","subtype":"success","named_in_prose":0,"cost_usd":0.2987764,"model":"sonnet","max_turns":12,"instrument":"17f918b","descs":"d4251fb4eda7"} diff --git a/tests/evals/autofire/runlog-baseline-turns12-tdd-known-bugfix.jsonl b/tests/evals/autofire/runlog-baseline-turns12-tdd-known-bugfix.jsonl new file mode 100644 index 0000000..bab7e91 --- /dev/null +++ b/tests/evals/autofire/runlog-baseline-turns12-tdd-known-bugfix.jsonl @@ -0,0 +1,10 @@ +{"case":"tdd-known-bugfix","sample":1,"polarity":"pos","fired":"","subtype":"success","named_in_prose":0,"cost_usd":0.2870228,"model":"sonnet","max_turns":12,"instrument":"17f918b","descs":"d4251fb4eda7"} +{"case":"tdd-known-bugfix","sample":2,"polarity":"pos","fired":"test-driven-development","subtype":"success","named_in_prose":0,"cost_usd":0.3048088,"model":"sonnet","max_turns":12,"instrument":"17f918b","descs":"d4251fb4eda7"} +{"case":"tdd-known-bugfix","sample":3,"polarity":"pos","fired":"","subtype":"success","named_in_prose":0,"cost_usd":0.2603012,"model":"sonnet","max_turns":12,"instrument":"17f918b","descs":"d4251fb4eda7"} +{"case":"tdd-known-bugfix","sample":4,"polarity":"pos","fired":"test-driven-development","subtype":"success","named_in_prose":0,"cost_usd":0.3981236,"model":"sonnet","max_turns":12,"instrument":"17f918b","descs":"d4251fb4eda7"} +{"case":"tdd-known-bugfix","sample":5,"polarity":"pos","fired":"","subtype":"success","named_in_prose":0,"cost_usd":0.1043738,"model":"sonnet","max_turns":12,"instrument":"17f918b","descs":"d4251fb4eda7"} +{"case":"tdd-known-bugfix","sample":6,"polarity":"pos","fired":"","subtype":"success","named_in_prose":0,"cost_usd":0.2474064,"model":"sonnet","max_turns":12,"instrument":"17f918b","descs":"d4251fb4eda7"} +{"case":"tdd-known-bugfix","sample":7,"polarity":"pos","fired":"","subtype":"success","named_in_prose":0,"cost_usd":0.2120628,"model":"sonnet","max_turns":12,"instrument":"17f918b","descs":"d4251fb4eda7"} +{"case":"tdd-known-bugfix","sample":8,"polarity":"pos","fired":"test-driven-development","subtype":"success","named_in_prose":0,"cost_usd":0.2736248,"model":"sonnet","max_turns":12,"instrument":"17f918b","descs":"d4251fb4eda7"} +{"case":"tdd-known-bugfix","sample":9,"polarity":"pos","fired":"","subtype":"success","named_in_prose":0,"cost_usd":0.1625512,"model":"sonnet","max_turns":12,"instrument":"17f918b","descs":"d4251fb4eda7"} +{"case":"tdd-known-bugfix","sample":10,"polarity":"pos","fired":"","subtype":"success","named_in_prose":0,"cost_usd":0.2106834,"model":"sonnet","max_turns":12,"instrument":"17f918b","descs":"d4251fb4eda7"} diff --git a/tests/evals/autofire/runlog-baseline-turns3-executing-plans-checkpoints.jsonl b/tests/evals/autofire/runlog-baseline-turns3-executing-plans-checkpoints.jsonl new file mode 100644 index 0000000..72dc7bf --- /dev/null +++ b/tests/evals/autofire/runlog-baseline-turns3-executing-plans-checkpoints.jsonl @@ -0,0 +1,10 @@ +{"case":"executing-plans-checkpoints","sample":1,"polarity":"pos","fired":"","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.0729558,"model":"sonnet","max_turns":3,"instrument":"901bb3f","descs":"d4251fb4eda7"} +{"case":"executing-plans-checkpoints","sample":2,"polarity":"pos","fired":"","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.0718264,"model":"sonnet","max_turns":3,"instrument":"901bb3f","descs":"d4251fb4eda7"} +{"case":"executing-plans-checkpoints","sample":3,"polarity":"pos","fired":"","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.0718664,"model":"sonnet","max_turns":3,"instrument":"901bb3f","descs":"d4251fb4eda7"} +{"case":"executing-plans-checkpoints","sample":4,"polarity":"pos","fired":"","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.0718764,"model":"sonnet","max_turns":3,"instrument":"901bb3f","descs":"d4251fb4eda7"} +{"case":"executing-plans-checkpoints","sample":5,"polarity":"pos","fired":"","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.073731,"model":"sonnet","max_turns":3,"instrument":"901bb3f","descs":"d4251fb4eda7"} +{"case":"executing-plans-checkpoints","sample":6,"polarity":"pos","fired":"","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.0722414,"model":"sonnet","max_turns":3,"instrument":"901bb3f","descs":"d4251fb4eda7"} +{"case":"executing-plans-checkpoints","sample":7,"polarity":"pos","fired":"","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.0717912,"model":"sonnet","max_turns":3,"instrument":"901bb3f","descs":"d4251fb4eda7"} +{"case":"executing-plans-checkpoints","sample":8,"polarity":"pos","fired":"","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.07180639999999999,"model":"sonnet","max_turns":3,"instrument":"901bb3f","descs":"d4251fb4eda7"} +{"case":"executing-plans-checkpoints","sample":9,"polarity":"pos","fired":"","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.07248639999999999,"model":"sonnet","max_turns":3,"instrument":"901bb3f","descs":"d4251fb4eda7"} +{"case":"executing-plans-checkpoints","sample":10,"polarity":"pos","fired":"","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.0719958,"model":"sonnet","max_turns":3,"instrument":"901bb3f","descs":"d4251fb4eda7"} diff --git a/tests/evals/autofire/runlog-baseline-turns3-swarm-audit.jsonl b/tests/evals/autofire/runlog-baseline-turns3-swarm-audit.jsonl new file mode 100644 index 0000000..9cd20fe --- /dev/null +++ b/tests/evals/autofire/runlog-baseline-turns3-swarm-audit.jsonl @@ -0,0 +1,10 @@ +{"case":"swarm-audit","sample":1,"polarity":"pos","fired":"swarm","subtype":"error_max_turns","named_in_prose":1,"cost_usd":0.08846659999999999,"model":"sonnet","max_turns":3,"instrument":"901bb3f","descs":"d4251fb4eda7"} +{"case":"swarm-audit","sample":2,"polarity":"pos","fired":"swarm","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.08691220000000001,"model":"sonnet","max_turns":3,"instrument":"901bb3f","descs":"d4251fb4eda7"} +{"case":"swarm-audit","sample":3,"polarity":"pos","fired":"swarm","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.08592940000000002,"model":"sonnet","max_turns":3,"instrument":"901bb3f","descs":"d4251fb4eda7"} +{"case":"swarm-audit","sample":4,"polarity":"pos","fired":"swarm","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.0871286,"model":"sonnet","max_turns":3,"instrument":"901bb3f","descs":"d4251fb4eda7"} +{"case":"swarm-audit","sample":5,"polarity":"pos","fired":"swarm","subtype":"error_max_turns","named_in_prose":1,"cost_usd":0.08716499999999999,"model":"sonnet","max_turns":3,"instrument":"901bb3f","descs":"d4251fb4eda7"} +{"case":"swarm-audit","sample":6,"polarity":"pos","fired":"swarm","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.0895552,"model":"sonnet","max_turns":3,"instrument":"901bb3f","descs":"d4251fb4eda7"} +{"case":"swarm-audit","sample":7,"polarity":"pos","fired":"swarm","subtype":"error_max_turns","named_in_prose":1,"cost_usd":0.08885860000000001,"model":"sonnet","max_turns":3,"instrument":"901bb3f","descs":"d4251fb4eda7"} +{"case":"swarm-audit","sample":8,"polarity":"pos","fired":"swarm","subtype":"error_max_turns","named_in_prose":1,"cost_usd":0.0891098,"model":"sonnet","max_turns":3,"instrument":"901bb3f","descs":"d4251fb4eda7"} +{"case":"swarm-audit","sample":9,"polarity":"pos","fired":"swarm","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.0850924,"model":"sonnet","max_turns":3,"instrument":"901bb3f","descs":"d4251fb4eda7"} +{"case":"swarm-audit","sample":10,"polarity":"pos","fired":"swarm","subtype":"error_max_turns","named_in_prose":1,"cost_usd":0.0875414,"model":"sonnet","max_turns":3,"instrument":"901bb3f","descs":"d4251fb4eda7"} diff --git a/tests/evals/autofire/runlog-baseline-turns3-tdd-known-bugfix.jsonl b/tests/evals/autofire/runlog-baseline-turns3-tdd-known-bugfix.jsonl new file mode 100644 index 0000000..c154318 --- /dev/null +++ b/tests/evals/autofire/runlog-baseline-turns3-tdd-known-bugfix.jsonl @@ -0,0 +1,10 @@ +{"case":"tdd-known-bugfix","sample":1,"polarity":"pos","fired":"","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.0769434,"model":"sonnet","max_turns":3,"instrument":"901bb3f","descs":"d4251fb4eda7"} +{"case":"tdd-known-bugfix","sample":2,"polarity":"pos","fired":"","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.0805056,"model":"sonnet","max_turns":3,"instrument":"901bb3f","descs":"d4251fb4eda7"} +{"case":"tdd-known-bugfix","sample":3,"polarity":"pos","fired":"","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.0758492,"model":"sonnet","max_turns":3,"instrument":"901bb3f","descs":"d4251fb4eda7"} +{"case":"tdd-known-bugfix","sample":4,"polarity":"pos","fired":"","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.07750119999999999,"model":"sonnet","max_turns":3,"instrument":"901bb3f","descs":"d4251fb4eda7"} +{"case":"tdd-known-bugfix","sample":5,"polarity":"pos","fired":"","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.0757592,"model":"sonnet","max_turns":3,"instrument":"901bb3f","descs":"d4251fb4eda7"} +{"case":"tdd-known-bugfix","sample":6,"polarity":"pos","fired":"","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.0754474,"model":"sonnet","max_turns":3,"instrument":"901bb3f","descs":"d4251fb4eda7"} +{"case":"tdd-known-bugfix","sample":7,"polarity":"pos","fired":"","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.0780272,"model":"sonnet","max_turns":3,"instrument":"901bb3f","descs":"d4251fb4eda7"} +{"case":"tdd-known-bugfix","sample":8,"polarity":"pos","fired":"","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.07568899999999999,"model":"sonnet","max_turns":3,"instrument":"901bb3f","descs":"d4251fb4eda7"} +{"case":"tdd-known-bugfix","sample":9,"polarity":"pos","fired":"","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.07590600000000002,"model":"sonnet","max_turns":3,"instrument":"901bb3f","descs":"d4251fb4eda7"} +{"case":"tdd-known-bugfix","sample":10,"polarity":"pos","fired":"","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.0754956,"model":"sonnet","max_turns":3,"instrument":"901bb3f","descs":"d4251fb4eda7"} diff --git a/tests/evals/autofire/runlog-baseline-turns3-writing-plans-spec-to-steps.jsonl b/tests/evals/autofire/runlog-baseline-turns3-writing-plans-spec-to-steps.jsonl new file mode 100644 index 0000000..672a528 --- /dev/null +++ b/tests/evals/autofire/runlog-baseline-turns3-writing-plans-spec-to-steps.jsonl @@ -0,0 +1,10 @@ +{"case":"writing-plans-spec-to-steps","sample":1,"polarity":"pos","fired":"writing-plans","subtype":"error_max_turns","named_in_prose":1,"cost_usd":0.07922980000000002,"model":"sonnet","max_turns":3,"instrument":"901bb3f","descs":"d4251fb4eda7"} +{"case":"writing-plans-spec-to-steps","sample":2,"polarity":"pos","fired":"writing-plans","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.08088900000000002,"model":"sonnet","max_turns":3,"instrument":"901bb3f","descs":"d4251fb4eda7"} +{"case":"writing-plans-spec-to-steps","sample":3,"polarity":"pos","fired":"","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.0769404,"model":"sonnet","max_turns":3,"instrument":"901bb3f","descs":"d4251fb4eda7"} +{"case":"writing-plans-spec-to-steps","sample":4,"polarity":"pos","fired":"writing-plans","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.08080079999999999,"model":"sonnet","max_turns":3,"instrument":"901bb3f","descs":"d4251fb4eda7"} +{"case":"writing-plans-spec-to-steps","sample":5,"polarity":"pos","fired":"writing-plans","subtype":"error_max_turns","named_in_prose":1,"cost_usd":0.08173699999999999,"model":"sonnet","max_turns":3,"instrument":"901bb3f","descs":"d4251fb4eda7"} +{"case":"writing-plans-spec-to-steps","sample":6,"polarity":"pos","fired":"writing-plans","subtype":"error_max_turns","named_in_prose":1,"cost_usd":0.085802,"model":"sonnet","max_turns":3,"instrument":"901bb3f","descs":"d4251fb4eda7"} +{"case":"writing-plans-spec-to-steps","sample":7,"polarity":"pos","fired":"writing-plans","subtype":"error_max_turns","named_in_prose":1,"cost_usd":0.0787162,"model":"sonnet","max_turns":3,"instrument":"901bb3f","descs":"d4251fb4eda7"} +{"case":"writing-plans-spec-to-steps","sample":8,"polarity":"pos","fired":"writing-plans","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.0787144,"model":"sonnet","max_turns":3,"instrument":"901bb3f","descs":"d4251fb4eda7"} +{"case":"writing-plans-spec-to-steps","sample":9,"polarity":"pos","fired":"writing-plans","subtype":"error_max_turns","named_in_prose":1,"cost_usd":0.08795660000000001,"model":"sonnet","max_turns":3,"instrument":"901bb3f","descs":"d4251fb4eda7"} +{"case":"writing-plans-spec-to-steps","sample":10,"polarity":"pos","fired":"writing-plans","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.0766312,"model":"sonnet","max_turns":3,"instrument":"901bb3f","descs":"d4251fb4eda7"} diff --git a/tests/stage-skill-descriptions.sh b/tests/stage-skill-descriptions.sh index 5902611..be41caa 100755 --- a/tests/stage-skill-descriptions.sh +++ b/tests/stage-skill-descriptions.sh @@ -77,8 +77,16 @@ do_stage() { : > "$STATE_DIR/manifest" for name in "$@"; do src="$SKILLS_SRC/$name/SKILL.md"; dst="$SKILLS_DST/$name/SKILL.md" - cp "$dst" "$STATE_DIR/$name.orig" - cp "$src" "$dst" + # Every cp is checked. There is no `set -e` here, so an unchecked failure would record a + # manifest entry whose checksum is of the UNCHANGED destination -- the arm would then run on + # stale bytes and the runlog would swear they were the new ones. + if ! cp "$dst" "$STATE_DIR/$name.orig"; then + echo "REFUSING: could not back up $dst -- nothing further staged." >&2; exit 2 + fi + if ! cp "$src" "$dst"; then + echo "REFUSING: could not install $src -> $dst. Run --restore to undo what did land." >&2 + exit 2 + fi printf '%s\t%s\t%s\n' "$name" "$dst" "$(sumof "$dst")" >> "$STATE_DIR/manifest" staged=$((staged + 1)) printf 'staged %-28s %s\n' "$name" "$(desc_of "$dst" | cut -c1-72)..." @@ -104,7 +112,13 @@ do_restore() { rc=1 continue fi - cp "$STATE_DIR/$name.orig" "$dst" + # Checked for the same reason, and harder: if this cp fails and rc stays 0, the rm -rf below + # deletes the only surviving copy of the original. + if ! cp "$STATE_DIR/$name.orig" "$dst"; then + echo "REFUSING $name: restore cp failed. Original kept at $STATE_DIR/$name.orig" >&2 + rc=1 + continue + fi n=$((n + 1)) printf 'restored %-28s\n' "$name" done < "$STATE_DIR/manifest" From dc4cad42e2eb2c1f4a8bd1fc12d5713d15395f72 Mon Sep 17 00:00:00 2001 From: vk Date: Tue, 1 Sep 2026 20:55:50 +0530 Subject: [PATCH 06/10] Key executing-plans on the artifact in front of it, not on chain position 1/10 -> 6/10, N=10 per arm, no early stop, max-turns=12 both arms. The old description was "Use when you have a written implementation plan to execute in a separate session with review checkpoints". The fixture prompt is "PLAN.md is the implementation plan we approved last session. Execute it, and pause for my review after each phase." Those are close to the same sentence and it fired once in ten, so the problem was never that the description failed to describe the situation. Two of the nine baseline misses fired test-driven-development instead, on a prompt with no test in it. That is what changed. Not the wording of the situation, which was already right, but that three sibling descriptions all opened on `Nth, once the shape is X` and competed for the same tokens in the clause a matcher weighs hardest. tests/description-collision.sh scores the corpus 23 before and 13 after, and the entire 10-point drop is those two overlaps. READ THE NUMBER WITH ITS CAVEAT. 6/10 meets the threshold I registered before the arm -- baseline <=1/10 and after >=6/10 -- and does not reach significance. Two-sided Fisher on 1/10 against 6/10 is p=0.0573, which clears neither 0.05 nor the Bonferroni 0.0125 that four tests want. The threshold was derived assuming a 0/10 baseline, where 6/10 would have given p=0.0108. The measured baseline was 1/10 and I never re-derived it. So the check passed while the thing the check stood for did not, which is the defect docs/checks-that-inherit-their-answer.md exists to catalogue, committed here in my own pre-registration. Separating 1/10 from 6/10 at that alpha needs roughly n=20-25 per arm, about $12 more at this budget, and that call is not mine to make. The misroute did not close: still 2/10 to test-driven-development. Whatever makes these two compete is narrowed by the rewrite, not removed. The runlog beside this commit is the after arm for this case. All four descriptions were installed together when it ran (digest 5cf618030b60 on every row), because the four are designed to be mutually exclusive OF EACH OTHER and a rate taken while three still carried ordinals would measure a tree that never ships. So this file's 6/10 is attributable to the shipped set, not to this one description in isolation. Description is 168 chars against check 3's 200 cap. Co-Authored-By: Claude Opus 5 (1M context) --- claude/skills/executing-plans/SKILL.md | 2 +- .../runlog-after-executing-plans-checkpoints.jsonl | 10 ++++++++++ 2 files changed, 11 insertions(+), 1 deletion(-) create mode 100644 tests/evals/autofire/runlog-after-executing-plans-checkpoints.jsonl diff --git a/claude/skills/executing-plans/SKILL.md b/claude/skills/executing-plans/SKILL.md index 3421721..6e12a91 100644 --- a/claude/skills/executing-plans/SKILL.md +++ b/claude/skills/executing-plans/SKILL.md @@ -1,6 +1,6 @@ --- name: executing-plans -description: Use when you have a written implementation plan to execute in a separate session with review checkpoints +description: "A written plan or numbered task list already exists and the ask is to carry it out: work the tasks in order, verify each, pause at the review checkpoint between phases." --- > **Ported here, and some of what it names is not.** This skill comes from upstream, where it diff --git a/tests/evals/autofire/runlog-after-executing-plans-checkpoints.jsonl b/tests/evals/autofire/runlog-after-executing-plans-checkpoints.jsonl new file mode 100644 index 0000000..0460322 --- /dev/null +++ b/tests/evals/autofire/runlog-after-executing-plans-checkpoints.jsonl @@ -0,0 +1,10 @@ +{"case":"executing-plans-checkpoints","sample":1,"polarity":"pos","fired":"executing-plans","subtype":"success","named_in_prose":0,"cost_usd":0.22186679999999998,"model":"sonnet","max_turns":12,"instrument":"f728a7e","descs":"5cf618030b60"} +{"case":"executing-plans-checkpoints","sample":2,"polarity":"pos","fired":"executing-plans","subtype":"success","named_in_prose":1,"cost_usd":0.261513,"model":"sonnet","max_turns":12,"instrument":"f728a7e","descs":"5cf618030b60"} +{"case":"executing-plans-checkpoints","sample":3,"polarity":"pos","fired":"","subtype":"success","named_in_prose":0,"cost_usd":0.33977540000000006,"model":"sonnet","max_turns":12,"instrument":"f728a7e","descs":"5cf618030b60"} +{"case":"executing-plans-checkpoints","sample":4,"polarity":"pos","fired":"executing-plans","subtype":"success","named_in_prose":1,"cost_usd":0.3473942,"model":"sonnet","max_turns":12,"instrument":"f728a7e","descs":"5cf618030b60"} +{"case":"executing-plans-checkpoints","sample":5,"polarity":"pos","fired":"test-driven-development","subtype":"success","named_in_prose":0,"cost_usd":0.330076,"model":"sonnet","max_turns":12,"instrument":"f728a7e","descs":"5cf618030b60"} +{"case":"executing-plans-checkpoints","sample":6,"polarity":"pos","fired":"executing-plans","subtype":"success","named_in_prose":0,"cost_usd":0.31307380000000007,"model":"sonnet","max_turns":12,"instrument":"f728a7e","descs":"5cf618030b60"} +{"case":"executing-plans-checkpoints","sample":7,"polarity":"pos","fired":"","subtype":"success","named_in_prose":0,"cost_usd":0.149817,"model":"sonnet","max_turns":12,"instrument":"f728a7e","descs":"5cf618030b60"} +{"case":"executing-plans-checkpoints","sample":8,"polarity":"pos","fired":"test-driven-development","subtype":"success","named_in_prose":0,"cost_usd":0.38165439999999995,"model":"sonnet","max_turns":12,"instrument":"f728a7e","descs":"5cf618030b60"} +{"case":"executing-plans-checkpoints","sample":9,"polarity":"pos","fired":"executing-plans","subtype":"success","named_in_prose":1,"cost_usd":0.37266160000000004,"model":"sonnet","max_turns":12,"instrument":"f728a7e","descs":"5cf618030b60"} +{"case":"executing-plans-checkpoints","sample":10,"polarity":"pos","fired":"executing-plans","subtype":"success","named_in_prose":1,"cost_usd":0.2113092,"model":"sonnet","max_turns":12,"instrument":"f728a7e","descs":"5cf618030b60"} From 4f91fbb990420ce6b78ca9510f45dbe67b0be3cc Mon Sep 17 00:00:00 2001 From: vk Date: Tue, 1 Sep 2026 20:56:17 +0530 Subject: [PATCH 07/10] Key test-driven-development on about-to-write-code, and measure that it changed nothing 3/10 -> 3/10, N=10 per arm, no early stop, max-turns=12 both arms. Two-sided Fisher p=1.0. The rewrite did not move this skill and I am not going to present it as though it did. What it did do is remove the reason the old wording was indefensible. "Third, once the shape is known, including any bugfix" asserts a position in a chain the matcher cannot see: it was never told there is a chain, and nothing in a single prompt lets it verify that this is step three of four. The chain lines live in the SessionStart routing block, injected once, not attached to any description. The replacement asserts a property of the prompt instead, which is at least checkable against what the model has in front of it. It also cost this skill both of its trigger-token overlaps. It shared "once shape" with writing-plans and "shape known" with brainstorming, in the first clause, which is the zone aebeebb identified as the one a matcher weighs hardest. tests/description-collision.sh scores the corpus 23 before and 13 after; those two pairs are the whole delta. So this commit buys a defensible description and a lower collision score, and buys no measured dispatch. Both statements are in the runlog beside it. The one place the old overlap showed up as behaviour is in the executing-plans arm, not this one: 2 of that skill's 9 baseline misses fired test-driven-development on a prompt with no test in it, and it is still 2/10 after. Narrowed, not removed. A caveat that limits this arm specifically. 3/10 at max-turns=12 is a stochastic rate, and n=10 cannot separate 0.3 from 0.5. It can separate "never fires" from "fires about half the time", which is what it was sized for. This arm establishes that test-driven-development is weak rather than dead, and nothing finer. The runlog was taken with all four rewritten descriptions installed together (digest 5cf618030b60 on every row), because the four are mutually exclusive of each other and an arm run against a half-rewritten tree measures something that never ships. Description is 169 chars against check 3's 200 cap. Co-Authored-By: Claude Opus 5 (1M context) --- claude/skills/test-driven-development/SKILL.md | 2 +- .../evals/autofire/runlog-after-tdd-known-bugfix.jsonl | 10 ++++++++++ 2 files changed, 11 insertions(+), 1 deletion(-) create mode 100644 tests/evals/autofire/runlog-after-tdd-known-bugfix.jsonl diff --git a/claude/skills/test-driven-development/SKILL.md b/claude/skills/test-driven-development/SKILL.md index 03617df..7afced1 100644 --- a/claude/skills/test-driven-development/SKILL.md +++ b/claude/skills/test-driven-development/SKILL.md @@ -1,6 +1,6 @@ --- name: test-driven-development -description: "Third, once the shape is known, including any bugfix: write the failing test before the implementation code." +description: "About to write implementation code or a bugfix and no test fails for that behaviour yet: write the failing test first, watch it fail, then write the minimum that passes." --- # Test-Driven Development (TDD) diff --git a/tests/evals/autofire/runlog-after-tdd-known-bugfix.jsonl b/tests/evals/autofire/runlog-after-tdd-known-bugfix.jsonl new file mode 100644 index 0000000..ac0975f --- /dev/null +++ b/tests/evals/autofire/runlog-after-tdd-known-bugfix.jsonl @@ -0,0 +1,10 @@ +{"case":"tdd-known-bugfix","sample":1,"polarity":"pos","fired":"","subtype":"success","named_in_prose":0,"cost_usd":0.291867,"model":"sonnet","max_turns":12,"instrument":"f728a7e","descs":"5cf618030b60"} +{"case":"tdd-known-bugfix","sample":2,"polarity":"pos","fired":"","subtype":"success","named_in_prose":0,"cost_usd":0.28343680000000004,"model":"sonnet","max_turns":12,"instrument":"f728a7e","descs":"5cf618030b60"} +{"case":"tdd-known-bugfix","sample":3,"polarity":"pos","fired":"","subtype":"success","named_in_prose":0,"cost_usd":0.23044199999999998,"model":"sonnet","max_turns":12,"instrument":"f728a7e","descs":"5cf618030b60"} +{"case":"tdd-known-bugfix","sample":4,"polarity":"pos","fired":"","subtype":"success","named_in_prose":0,"cost_usd":0.27989240000000004,"model":"sonnet","max_turns":12,"instrument":"f728a7e","descs":"5cf618030b60"} +{"case":"tdd-known-bugfix","sample":5,"polarity":"pos","fired":"test-driven-development","subtype":"success","named_in_prose":0,"cost_usd":0.2422594,"model":"sonnet","max_turns":12,"instrument":"f728a7e","descs":"5cf618030b60"} +{"case":"tdd-known-bugfix","sample":6,"polarity":"pos","fired":"","subtype":"success","named_in_prose":0,"cost_usd":0.2884586,"model":"sonnet","max_turns":12,"instrument":"f728a7e","descs":"5cf618030b60"} +{"case":"tdd-known-bugfix","sample":7,"polarity":"pos","fired":"","subtype":"success","named_in_prose":0,"cost_usd":0.29732900000000007,"model":"sonnet","max_turns":12,"instrument":"f728a7e","descs":"5cf618030b60"} +{"case":"tdd-known-bugfix","sample":8,"polarity":"pos","fired":"test-driven-development","subtype":"success","named_in_prose":0,"cost_usd":0.374756,"model":"sonnet","max_turns":12,"instrument":"f728a7e","descs":"5cf618030b60"} +{"case":"tdd-known-bugfix","sample":9,"polarity":"pos","fired":"test-driven-development","subtype":"success","named_in_prose":0,"cost_usd":0.25005459999999996,"model":"sonnet","max_turns":12,"instrument":"f728a7e","descs":"5cf618030b60"} +{"case":"tdd-known-bugfix","sample":10,"polarity":"pos","fired":"","subtype":"success","named_in_prose":0,"cost_usd":0.2300466,"model":"sonnet","max_turns":12,"instrument":"f728a7e","descs":"5cf618030b60"} From 1c063f0c231d587b7c8ceaa089a6e983c7ced43d Mon Sep 17 00:00:00 2001 From: vk Date: Tue, 1 Sep 2026 20:56:17 +0530 Subject: [PATCH 08/10] Drop swarm's quoted trigger strings for the situation they were standing in for 10/10 -> 10/10, N=10 per arm, no early stop, max-turns=3 both arms. No regression, and no improvement was available. The brief said swarm fires only when the user says "in parallel". The baseline refutes that: swarm fired 10/10 on a prompt containing none of the four strings its own description quoted -- not "in parallel", not "at once", not "all of these", not "try N ways" -- because the fixture describes six non-importing integrations needing six independent verdicts. The situation was carrying it the whole time. The literals were decoration on a description that already worked. So this is a simplification with a measured no-regression, not a fix. The description now states the cardinality-and-independence condition it was really matching on: three or more independent units, none needing another's result. That is also why it is the one skill here that does not sit on the planning-chain axis at all, and can co-fire with any stage of it. The literals are not lost. claude/hooks/inject-session-context.sh still carries them on its swarm routing line, and that file is owned elsewhere this week. The division of labour is the one the hook's own preamble states: the routing block spells out what descriptions do not reliably trigger. This change moves swarm off needing that crutch without removing it. Measuring an arm with no headroom is the point rather than a waste. Changing a description that scores 10/10 and not re-measuring it is how a regression ships as a cleanup, and this repo has shipped enough of those. The runlog beside this commit is what makes "no regression" a reading instead of an assumption. The runlog was taken with all four rewritten descriptions installed together (digest 5cf618030b60 on every row). All 10 samples terminated error_max_turns at max-turns=3, as they did at baseline; the Skill call precedes the cut, so the hits stand, and the arm is budget-matched to its baseline. Description is 186 chars against check 3's 200 cap, down from 194. Co-Authored-By: Claude Opus 5 (1M context) --- claude/skills/swarm/SKILL.md | 2 +- tests/evals/autofire/runlog-after-swarm-audit.jsonl | 10 ++++++++++ 2 files changed, 11 insertions(+), 1 deletion(-) create mode 100644 tests/evals/autofire/runlog-after-swarm-audit.jsonl diff --git a/claude/skills/swarm/SKILL.md b/claude/skills/swarm/SKILL.md index 6c7aaf8..319ac61 100644 --- a/claude/skills/swarm/SKILL.md +++ b/claude/skills/swarm/SKILL.md @@ -1,6 +1,6 @@ --- name: swarm -description: "Use when work splits into independent parts or approaches should be raced — 'in parallel', 'at once', 'all of these', 'try N ways'. Fans out N agents in ONE batched message, returns one table." +description: "Three or more independent units of work exist (files, packages, endpoints, rival approaches) and none needs another's result. Fans out N agents in ONE batched message, returns one table." --- # Swarm diff --git a/tests/evals/autofire/runlog-after-swarm-audit.jsonl b/tests/evals/autofire/runlog-after-swarm-audit.jsonl new file mode 100644 index 0000000..fd2c271 --- /dev/null +++ b/tests/evals/autofire/runlog-after-swarm-audit.jsonl @@ -0,0 +1,10 @@ +{"case":"swarm-audit","sample":1,"polarity":"pos","fired":"security-review swarm","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.08565840000000001,"model":"sonnet","max_turns":3,"instrument":"f728a7e","descs":"5cf618030b60"} +{"case":"swarm-audit","sample":2,"polarity":"pos","fired":"swarm","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.10247700000000001,"model":"sonnet","max_turns":3,"instrument":"f728a7e","descs":"5cf618030b60"} +{"case":"swarm-audit","sample":3,"polarity":"pos","fired":"swarm","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.0861888,"model":"sonnet","max_turns":3,"instrument":"f728a7e","descs":"5cf618030b60"} +{"case":"swarm-audit","sample":4,"polarity":"pos","fired":"swarm","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.0863274,"model":"sonnet","max_turns":3,"instrument":"f728a7e","descs":"5cf618030b60"} +{"case":"swarm-audit","sample":5,"polarity":"pos","fired":"swarm","subtype":"error_max_turns","named_in_prose":1,"cost_usd":0.084047,"model":"sonnet","max_turns":3,"instrument":"f728a7e","descs":"5cf618030b60"} +{"case":"swarm-audit","sample":6,"polarity":"pos","fired":"swarm","subtype":"error_max_turns","named_in_prose":1,"cost_usd":0.08337960000000001,"model":"sonnet","max_turns":3,"instrument":"f728a7e","descs":"5cf618030b60"} +{"case":"swarm-audit","sample":7,"polarity":"pos","fired":"swarm","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.0881008,"model":"sonnet","max_turns":3,"instrument":"f728a7e","descs":"5cf618030b60"} +{"case":"swarm-audit","sample":8,"polarity":"pos","fired":"swarm","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.08495119999999999,"model":"sonnet","max_turns":3,"instrument":"f728a7e","descs":"5cf618030b60"} +{"case":"swarm-audit","sample":9,"polarity":"pos","fired":"swarm","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.085524,"model":"sonnet","max_turns":3,"instrument":"f728a7e","descs":"5cf618030b60"} +{"case":"swarm-audit","sample":10,"polarity":"pos","fired":"swarm","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.08728860000000001,"model":"sonnet","max_turns":3,"instrument":"f728a7e","descs":"5cf618030b60"} From 5c0d908c93355082a1999324e4c943b42c693868 Mon Sep 17 00:00:00 2001 From: vk Date: Tue, 1 Sep 2026 20:56:32 +0530 Subject: [PATCH 09/10] Key writing-plans on nothing-written-down, and confirm no regression 9/10 -> 10/10, N=10 per arm, no early stop, max-turns=3 both arms. Two-sided Fisher p=1.0. The one-sample difference is noise at this n and should not be read as an improvement. The brief listed writing-plans among three skills that have never fired. That is true of the transcript corpus and false of this harness: it fired 9/10 on the unedited description, before anything in this branch touched it. There was no defect here to fix and no headroom to fix it in. What the rewrite buys is the same thing it buys for its two siblings. "Second, once the shape is agreed and before code" asserts a position in a chain the matcher cannot see. The replacement asserts three things about the prompt itself: requirements settled, nothing written down about how, more than one edit of work. It also drops the "once shape" overlap this description shared with test-driven-development in the first clause, which is 6 of the 10 points tests/description-collision.sh loses between 23 and 13. THE CAVEAT THAT LIMITS BOTH ARMS, and it limits them badly enough to state here rather than only in RESULTS.md. This fixture shares its hinges with the description it tests. It says "SPEC.md is agreed" against a description reading "once the shape is agreed", and "before anyone touches app.js" against "before code". That is not a quotation, but it is the same two hinges, and a fixture built from a description's own hinges will fire more often than a real request does. 9/10 here beside 0 across 2536 transcripts is most economically explained by real prompts not stating the situation this plainly -- a claim about how people phrase requests, not about how descriptions are worded. This arm cannot separate those two, and neither the before nor the after number should be quoted as evidence that the skill works in production. An earlier draft of RESULTS.md attributed a phrase to this description that does not appear in it. Review caught it. The overlap named above is the real one. The runlog was taken with all four rewritten descriptions installed together (digest 5cf618030b60 on every row). Description is 178 chars against check 3's 200 cap. Co-Authored-By: Claude Opus 5 (1M context) --- claude/skills/writing-plans/SKILL.md | 2 +- .../runlog-after-writing-plans-spec-to-steps.jsonl | 10 ++++++++++ 2 files changed, 11 insertions(+), 1 deletion(-) create mode 100644 tests/evals/autofire/runlog-after-writing-plans-spec-to-steps.jsonl diff --git a/claude/skills/writing-plans/SKILL.md b/claude/skills/writing-plans/SKILL.md index 1466288..3b114a7 100644 --- a/claude/skills/writing-plans/SKILL.md +++ b/claude/skills/writing-plans/SKILL.md @@ -1,6 +1,6 @@ --- name: writing-plans -description: "Second, once the shape is agreed and before code: turn a spec or requirements into a written multi-step plan." +description: "Requirements are settled, nothing is written down about how, and the work is more than one edit: produce the plan document, file map, bite-sized tasks, one verification per task." --- > **Ported here, and some of what it names is not.** This skill comes from upstream, where it diff --git a/tests/evals/autofire/runlog-after-writing-plans-spec-to-steps.jsonl b/tests/evals/autofire/runlog-after-writing-plans-spec-to-steps.jsonl new file mode 100644 index 0000000..01b5bc4 --- /dev/null +++ b/tests/evals/autofire/runlog-after-writing-plans-spec-to-steps.jsonl @@ -0,0 +1,10 @@ +{"case":"writing-plans-spec-to-steps","sample":1,"polarity":"pos","fired":"writing-plans","subtype":"error_max_turns","named_in_prose":1,"cost_usd":0.0813344,"model":"sonnet","max_turns":3,"instrument":"f728a7e","descs":"5cf618030b60"} +{"case":"writing-plans-spec-to-steps","sample":2,"polarity":"pos","fired":"writing-plans","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.07966320000000002,"model":"sonnet","max_turns":3,"instrument":"f728a7e","descs":"5cf618030b60"} +{"case":"writing-plans-spec-to-steps","sample":3,"polarity":"pos","fired":"writing-plans","subtype":"error_max_turns","named_in_prose":1,"cost_usd":0.0869972,"model":"sonnet","max_turns":3,"instrument":"f728a7e","descs":"5cf618030b60"} +{"case":"writing-plans-spec-to-steps","sample":4,"polarity":"pos","fired":"writing-plans","subtype":"error_max_turns","named_in_prose":1,"cost_usd":0.08967800000000001,"model":"sonnet","max_turns":3,"instrument":"f728a7e","descs":"5cf618030b60"} +{"case":"writing-plans-spec-to-steps","sample":5,"polarity":"pos","fired":"writing-plans","subtype":"error_max_turns","named_in_prose":1,"cost_usd":0.0859346,"model":"sonnet","max_turns":3,"instrument":"f728a7e","descs":"5cf618030b60"} +{"case":"writing-plans-spec-to-steps","sample":6,"polarity":"pos","fired":"writing-plans","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.0865056,"model":"sonnet","max_turns":3,"instrument":"f728a7e","descs":"5cf618030b60"} +{"case":"writing-plans-spec-to-steps","sample":7,"polarity":"pos","fired":"writing-plans","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.0813326,"model":"sonnet","max_turns":3,"instrument":"f728a7e","descs":"5cf618030b60"} +{"case":"writing-plans-spec-to-steps","sample":8,"polarity":"pos","fired":"writing-plans","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.0816522,"model":"sonnet","max_turns":3,"instrument":"f728a7e","descs":"5cf618030b60"} +{"case":"writing-plans-spec-to-steps","sample":9,"polarity":"pos","fired":"writing-plans","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.08035980000000001,"model":"sonnet","max_turns":3,"instrument":"f728a7e","descs":"5cf618030b60"} +{"case":"writing-plans-spec-to-steps","sample":10,"polarity":"pos","fired":"writing-plans","subtype":"error_max_turns","named_in_prose":1,"cost_usd":0.08794320000000001,"model":"sonnet","max_turns":3,"instrument":"f728a7e","descs":"5cf618030b60"} From 133ef0b30a266bd4861610f38c5b2fb21e5c067d Mon Sep 17 00:00:00 2001 From: vk Date: Tue, 1 Sep 2026 20:59:13 +0530 Subject: [PATCH 10/10] Publish the before/after table, and the threshold that passed without earning it Four arms, N=10 each side, no early stop, every arm budget-matched to its own baseline. 40 calls after, $7.41; $16.19 and 100 calls across everything published here. executing-plans 1/10 -> 6/10 p=0.0573 moved, not significant test-driven-development 3/10 -> 3/10 p=1.0 unchanged writing-plans 9/10 -> 10/10 p=1.0 no regression, no headroom swarm 10/10 -> 10/10 p=1.0 no regression, no headroom The registered threshold was: fixed when baseline <=1/10 and after >=6/10. executing-plans came in at exactly 1/10 and 6/10, so it passed. It should not have been written that way. I derived "6/10" against an assumed 0/10 baseline, where it gives p=0.0108 and clears the Bonferroni 0.0125 four tests want. The measured baseline was 1/10 and I never re-derived the bound. Against 1/10, 6/10 is p=0.0573 and clears nothing. The check passed while the thing it stood for did not. That is docs/checks-that-inherit-their-answer.md, in my own pre-registration rather than someone else's, and the entry is written up as such rather than quietly repaired. Resolving 1/10 against 6/10 at that alpha needs roughly n=20-25 per arm, about $12 more. What the arms actually establish. Two of the brief's premises are false of this harness: writing-plans fires 9/10 unedited and swarm fires 10/10 on a prompt carrying none of the four trigger strings its own description quoted. The transcript-census zeros across 2536 transcripts are real, and they are about how real prompts are phrased rather than how descriptions are worded. This arm cannot separate those, and RESULTS.md says so instead of taking the credit. The writing-plans fixture shares its hinges with the description it tests -- "SPEC.md is agreed" against "once the shape is agreed", "before anyone touches app.js" against "before code". Neither its 9/10 nor its 10/10 should be quoted as evidence the skill works in production. An earlier draft of this file supported that same caveat with a phrase that does not appear in the description at all; review caught the fabrication and the real overlap replaced it. Two claims were also removed for outrunning their data. The baseline section said the starved arms "never reached a decision point"; the runlog carries fired and subtype and no turn trace, and the arms that fired were also 10/10 cut off, so being cut off separates nothing. It now says only that the zeros were taken at a budget every sample exhausted, which is reason to re-take them and not reason to publish them. Verified: .claude/verify.sh check 12 reads 30 test cases, check 31 green at 168 files, check 3 clean on all four rewrites (168-186 chars against a 200 cap). dispatch-static.sh 4/4, 30 cases, zero model calls. description-collision.sh self-test green both directions, corpus 23 -> 13, no hard-fail pairs. stage-skill-descriptions.sh restored ~/.claude to d4251fb4eda7 with nothing staged. The four remaining gate failures reproduce identically on pristine 1bfb133 and belong to the in-flight release. Co-Authored-By: Claude Opus 5 (1M context) --- tests/README.md | 4 +++ tests/evals/autofire/RESULTS.md | 51 ++++++++++++++++++++++++++++++--- 2 files changed, 51 insertions(+), 4 deletions(-) diff --git a/tests/README.md b/tests/README.md index 454e0c8..db45425 100644 --- a/tests/README.md +++ b/tests/README.md @@ -345,6 +345,10 @@ The raw samples, one JSON object per line, written by `auto-trigger.sh` in SAMPL - `tests/evals/autofire/runlog-baseline-turns3-tdd-known-bugfix.jsonl` - `tests/evals/autofire/runlog-baseline-turns12-executing-plans-checkpoints.jsonl` - `tests/evals/autofire/runlog-baseline-turns12-tdd-known-bugfix.jsonl` +- `tests/evals/autofire/runlog-after-writing-plans-spec-to-steps.jsonl` +- `tests/evals/autofire/runlog-after-swarm-audit.jsonl` +- `tests/evals/autofire/runlog-after-tdd-known-bugfix.jsonl` +- `tests/evals/autofire/runlog-after-executing-plans-checkpoints.jsonl` Read the turns=3 and turns=12 tables together or not at all. Every turns=3 sample terminated `error_max_turns`, including the arms that fired, so being cut off separates nothing there. That diff --git a/tests/evals/autofire/RESULTS.md b/tests/evals/autofire/RESULTS.md index e24fb16..103f9c2 100644 --- a/tests/evals/autofire/RESULTS.md +++ b/tests/evals/autofire/RESULTS.md @@ -1,7 +1,7 @@ # Autofire arm: do the four chain skills route from the situation alone? -**Status: baseline complete. After-arm pending. Do not cite the before/after comparison until the -after table is filled in.** +**Status: complete. Read the significance caveat under the after table before quoting any number +from it.** | | | |---|---| @@ -10,7 +10,7 @@ after table is filled in.** | Model | `sonnet` (CLI alias), OAuth session, `ANTHROPIC_API_KEY` stripped by the harness | | Tool fence | `Write,Edit,MultiEdit,NotebookEdit,Bash,Agent,Workflow,Explore,Task` | | Seed / temperature | Neither. The CLI exposes no such knob, which is why n>1 exists. | -| Installed descriptions | digest `d4251fb4eda7` on all 60 baseline rows, one value, no pooling | +| Installed descriptions | `d4251fb4eda7` on all 60 baseline rows, `5cf618030b60` on all 40 after rows. One value per arm, no pooling. | | Sampling | N=10 independent invocations per arm, no early stop. `ATTEMPTS` is unused. | Raw k/N only. At n=10 a percentage or an interval would imply precision this sample does not @@ -116,10 +116,53 @@ This is design discipline, not a dispatch law. The claim that colliding triggers skills was withdrawn in 1.47.0, and the pre-registered replacement found the hypothesis unsupported (`tests/evals/collision/RESULTS.md`). +## After the rewrite + +All four rewritten descriptions were installed together for this arm, by +`tests/stage-skill-descriptions.sh`, and restored afterwards. Digest `5cf618030b60` on all 40 +rows. Each arm is budget-matched to its own baseline. 40 calls, $7.41. + +Runlogs: `tests/evals/autofire/runlog-after-writing-plans-spec-to-steps.jsonl`, +`runlog-after-swarm-audit.jsonl`, `runlog-after-tdd-known-bugfix.jsonl`, +`runlog-after-executing-plans-checkpoints.jsonl`. + +| skill | budget | before | after | Fisher p, two-sided | reading | +|---|---|---|---|---|---| +| `executing-plans` | 12 | 1/10 | 6/10 | 0.0573 | moved, not significant | +| `test-driven-development` | 12 | 3/10 | 3/10 | 1.0 | unchanged | +| `writing-plans` | 3 | 9/10 | 10/10 | 1.0 | no regression, no headroom | +| `swarm` | 3 | 10/10 | 10/10 | 1.0 | no regression, no headroom | + +### The threshold passed and the statistic did not + +Before the arm I registered: an arm counts as fixed when baseline is at most 1/10 and after is at +least 6/10. `executing-plans` came in at exactly 1/10 and 6/10, so the threshold is met. + +It should not have been written that way. I derived "6/10" against an assumed 0/10 baseline, +where it gives p=0.0108 and clears the Bonferroni alpha of 0.0125 that four tests want. The +measured baseline was 1/10, and I never went back and re-derived the bound. Against 1/10, 6/10 is +p=0.0573. It clears neither 0.0125 nor 0.05. + +So the check passed while the thing the check stood for did not. That is the failure +`docs/checks-that-inherit-their-answer.md` catalogues, and this is an instance of it in my own +pre-registration rather than in someone else's. The honest statement of this result is that +`executing-plans` moved from 1/10 to 6/10 and that n=10 cannot establish the move. Separating +those two rates at Bonferroni alpha needs roughly n=20 to 25 per arm, about $12 more at this +budget. + +`test-driven-development` did not move at all. The rewrite bought it a description that asserts a +checkable property instead of an invisible chain position, and bought no dispatch. Both are in +the runlog. + +The `executing-plans` misroute did not close. Two of ten samples still fire +`test-driven-development` on a prompt with no test in it, the same 2 as at baseline. The rewrite +narrows what makes those two compete without removing it. + ## Cost The harness records this per arm. Nobody estimated it. Baseline at turns=3 cost $3.17 across 40 -calls, $0.079 each. The turns=12 re-run cost $5.61 across 20 calls, $0.281 each. +calls, $0.079 each. The turns=12 re-run cost $5.61 across 20 calls, $0.281 each. The after arm +cost $7.41 across 40 calls. Total for the published arms: $16.19 across 100 calls. ## Instrument changes after the baseline, and why they do not invalidate it