diff --git a/README.md b/README.md index 80e08cb..1c27a44 100644 --- a/README.md +++ b/README.md @@ -203,7 +203,7 @@ CLI wrappers under `~/.config/agents/bin/`. Skills are not slash commands. They fire on the situation from their description — writing prose reaches for `unslop`, reviewing TypeScript reaches for `typescript-best-practices` — and -`tests/auto-trigger.sh` asserts that with 28 cases against the live model. +`tests/auto-trigger.sh` asserts that with 30 cases against the live model. ## Checks that can fail diff --git a/claude/skills/executing-plans/SKILL.md b/claude/skills/executing-plans/SKILL.md index 3421721..6e12a91 100644 --- a/claude/skills/executing-plans/SKILL.md +++ b/claude/skills/executing-plans/SKILL.md @@ -1,6 +1,6 @@ --- name: executing-plans -description: Use when you have a written implementation plan to execute in a separate session with review checkpoints +description: "A written plan or numbered task list already exists and the ask is to carry it out: work the tasks in order, verify each, pause at the review checkpoint between phases." --- > **Ported here, and some of what it names is not.** This skill comes from upstream, where it diff --git a/claude/skills/swarm/SKILL.md b/claude/skills/swarm/SKILL.md index 6c7aaf8..319ac61 100644 --- a/claude/skills/swarm/SKILL.md +++ b/claude/skills/swarm/SKILL.md @@ -1,6 +1,6 @@ --- name: swarm -description: "Use when work splits into independent parts or approaches should be raced — 'in parallel', 'at once', 'all of these', 'try N ways'. Fans out N agents in ONE batched message, returns one table." +description: "Three or more independent units of work exist (files, packages, endpoints, rival approaches) and none needs another's result. Fans out N agents in ONE batched message, returns one table." --- # Swarm diff --git a/claude/skills/test-driven-development/SKILL.md b/claude/skills/test-driven-development/SKILL.md index 03617df..7afced1 100644 --- a/claude/skills/test-driven-development/SKILL.md +++ b/claude/skills/test-driven-development/SKILL.md @@ -1,6 +1,6 @@ --- name: test-driven-development -description: "Third, once the shape is known, including any bugfix: write the failing test before the implementation code." +description: "About to write implementation code or a bugfix and no test fails for that behaviour yet: write the failing test first, watch it fail, then write the minimum that passes." --- # Test-Driven Development (TDD) diff --git a/claude/skills/writing-plans/SKILL.md b/claude/skills/writing-plans/SKILL.md index 1466288..3b114a7 100644 --- a/claude/skills/writing-plans/SKILL.md +++ b/claude/skills/writing-plans/SKILL.md @@ -1,6 +1,6 @@ --- name: writing-plans -description: "Second, once the shape is agreed and before code: turn a spec or requirements into a written multi-step plan." +description: "Requirements are settled, nothing is written down about how, and the work is more than one edit: produce the plan document, file map, bite-sized tasks, one verification per task." --- > **Ported here, and some of what it names is not.** This skill comes from upstream, where it diff --git a/docs/how-skills-fire.md b/docs/how-skills-fire.md index 80a7053..0161002 100644 --- a/docs/how-skills-fire.md +++ b/docs/how-skills-fire.md @@ -75,7 +75,7 @@ its own. `tests/auto-trigger.sh` runs real prompts through the CLI and reports which skills fired. A version of the hook that dropped the routing table in favour of "the descriptions are the -triggers" was measured against 28 cases: +triggers" was measured against the 28 auto-trigger cases the suite carried on 2026-08-23: | Hook version | Result | |---|---| diff --git a/docs/what-this-actually-does.md b/docs/what-this-actually-does.md index 8873bb1..a01738e 100644 --- a/docs/what-this-actually-does.md +++ b/docs/what-this-actually-does.md @@ -61,7 +61,7 @@ The thing does what it says. Nobody has shown it changes an outcome. |---|---| | Does delegation rate fall off across a session's own lifetime? | `tests/delegation-drift.sh` exists, is pre-registered (first-third vs. last-third breadth-eligible delegation rate, with explicit floors on eligible windows and contributing sessions), and states its own reverse-causality confound up front: late-session work may just be less delegable, independent of anything the model does differently. It reports `NOT EVALUATED` rather than a rate below its own floors, the expected, correct result on day one, not a bug. | | Does auto-compaction correlate with worse behaviour in the turns after it? | `tests/compaction-effect.sh` is pre-registered the same way (is_error rate in a 15-tool-call window before and after a compaction boundary, with floors on boundary count and pooled calls) and states plainly that any result is correlational, not causal. A session long enough to hit auto-compaction is not a random draw from the same population as a short one. The one published result in this space, Governance Decay (arXiv 2606.22528), is from a different harness and does not transfer by citation alone. | -| Do the skills outside the measured subsets actually dispatch? | `tests/auto-trigger.sh` declares 28 cases. Category 1 above accounts for a 9-case routing-table ablation and 8 principle-prefix-naming fixes, whose re-run is "not yet authorised" per CHANGELOG.md, "1.35.0" (roughly 24 headless calls). The dispatch rate for the remaining cases is not separately recorded in this document's sources. | +| Do the skills outside the measured subsets actually dispatch? | `tests/auto-trigger.sh` declares 30 cases. Category 1 above accounts for a 9-case routing-table ablation and 8 principle-prefix-naming fixes, whose re-run is "not yet authorised" per CHANGELOG.md, "1.35.0" (roughly 24 headless calls). The dispatch rate for the remaining cases is not separately recorded in this document's sources. | | Does any of this improve output quality on a frontier model? | Three independent local measurements found no gain and no loss large enough to trust: the review pathway (category 1), a regression-scored SWE-bench Lite pass that tied all four arms at 0 of 2 resolved once neighbour-breaking counted, and the false-completion pilot. `docs/research/do-harnesses-help.md` names the honest reading directly: these are ceiling effects on small, well-specified tasks, which read as "no difference" when the accurate statement is "this could not tell." `docs/research/harness-value-literature-2026-08.md`, surveying roughly 70 published sources, reaches the same split independently: no well-powered published result shows a config-layer gain on issue-resolution tasks with a frontier model, while the tail-behaviour literature (test-exploitation, cheating, out-of-scope destructive actions) shows some of the largest effect sizes in the field. | ### The failures, not smoothed over diff --git a/tests/README.md b/tests/README.md index 5c4a81e..db45425 100644 --- a/tests/README.md +++ b/tests/README.md @@ -332,6 +332,29 @@ hand, the same status `bin-scripts.sh` and `container-matrix.sh` started from. tests/hook-latency.sh ``` +## evals/autofire/ + +Whether the four planning-chain skills route from a situation with no user instruction naming +them. `tests/evals/autofire/RESULTS.md` carries the arm, its reporting rules and its cost. + +The raw samples, one JSON object per line, written by `auto-trigger.sh` in SAMPLES mode: + +- `tests/evals/autofire/runlog-baseline-turns3-writing-plans-spec-to-steps.jsonl` +- `tests/evals/autofire/runlog-baseline-turns3-swarm-audit.jsonl` +- `tests/evals/autofire/runlog-baseline-turns3-executing-plans-checkpoints.jsonl` +- `tests/evals/autofire/runlog-baseline-turns3-tdd-known-bugfix.jsonl` +- `tests/evals/autofire/runlog-baseline-turns12-executing-plans-checkpoints.jsonl` +- `tests/evals/autofire/runlog-baseline-turns12-tdd-known-bugfix.jsonl` +- `tests/evals/autofire/runlog-after-writing-plans-spec-to-steps.jsonl` +- `tests/evals/autofire/runlog-after-swarm-audit.jsonl` +- `tests/evals/autofire/runlog-after-tdd-known-bugfix.jsonl` +- `tests/evals/autofire/runlog-after-executing-plans-checkpoints.jsonl` + +Read the turns=3 and turns=12 tables together or not at all. Every turns=3 sample terminated +`error_max_turns`, including the arms that fired, so being cut off separates nothing there. That +is why two of those four numbers were re-taken at a budget nothing exhausted rather than +published as results. + ## evals/ `evals/run-pathways.sh` and `evals/swebench/run.sh` score this bundle against @@ -376,7 +399,7 @@ script prints `SKIP: ...` and exits 0. That is a valid, non-failing outcome, not a bug in the test. Each case runs in its own `mktemp -d` under `/tmp` (never this repo), so -nothing here pollutes the working tree the model sees. The test suite covers 28 cases: +nothing here pollutes the working tree the model sees. The test suite covers 30 cases: readme-writing, typescript-review, swarm-audit, blast-radius-auth, feature-chain, root-cause-guard, overnight-audit-trail, ui-iterate-styles, component-registry-combobox, idempotent-cron, negative-arithmetic, and negative-factual. @@ -389,7 +412,7 @@ idempotent-cron, negative-arithmetic, and negative-factual. never report `loggedIn: true` there. The script detects this and skips rather than failing the build. 2. **It would bill tokens.** Every case makes real API calls, up to 3 turns - each across 28 cases. Running this on every push or pull request in CI + each across 30 cases. Running this on every push or pull request in CI would spend real money on a check that mostly guards against skill-routing regressions. Those regressions are infrequent. Run the script by hand instead, or schedule it on a machine that already has an @@ -568,6 +591,59 @@ Known gap: `hooks.json` names two of the eight scripts under `claude/hooks/` dir plugin lane carries routing only. The rest are invoked from inside those or wired through `claude/settings.json` in the full install, so renaming one of them is not caught here. +## description-collision.sh + +Scores every shipped skill description for shared trigger n-grams and names the overlapping +pairs. No model calls, no network. Splits each description at its first colon, period or em dash +into the discriminator clause and the rest, then weights shared 2- and 3-grams by where they +land: 3 for both first clauses, 2 for one, 1 for neither. + +The only failing condition is a 3-gram opening two descriptions. That is the defect `aebeebb` +fixed by hand, where `grill-me` and `interrogate` both led with "tear this apart". Everything +else prints and passes, and the threshold is echoed on every run so nobody reads a green as "no +overlap". + +Read the header before citing it. This is design discipline, not a dispatch law. The claim that +colliding triggers suppress both skills was withdrawn in 1.47.0 for having no surviving runlog, +and the pre-registered replacement at `evals/collision/` found the hypothesis unsupported. + +``` +tests/description-collision.sh # score claude/skills +tests/description-collision.sh --self-test # prove it can fail, then that it passes +``` + +`--self-test` plants a collision that survives stopword normalisation, asserts the check fails +and names the pair, then scores a clean corpus and asserts it passes. It found three defects in +its own scorer before the live corpus did: a wrapped stoplist that made BSD awk exit without +scoring anything, a `score()` that returned awk's status without telling a parse error apart from +"no collisions", and an em dash that aborts BSD awk under a UTF-8 locale. `LC_ALL=C` is pinned +for the third. + +## stage-skill-descriptions.sh + +Installs the repo's description line for named skills into `~/.claude/skills`, and puts it back. + +`auto-trigger.sh` runs the CLI in a scratch workdir, so the descriptions that reach the matcher +are the installed ones. Measured 2026-09-01: a project-local `.claude/skills//SKILL.md` is +loaded and listed, but a user-level skill of the same name wins. Asked to quote its own listing +for `swarm`, the model returned the installed text and reported no duplicate. So an after-arm +measured without installing would measure the before-arm's bytes and publish it as a result. +Running `install.sh` instead installs the whole tree, including whatever another session has in +flight. + +``` +tests/stage-skill-descriptions.sh --stage swarm writing-plans +tests/stage-skill-descriptions.sh --status +tests/stage-skill-descriptions.sh --restore +``` + +Both directions refuse rather than guess. `--stage` aborts if any installed copy already differs +from repo HEAD, and pre-flights every named skill before touching any, so a half-applied stage +cannot happen. `--restore` compares each file against the checksum it wrote and refuses that file +if it changed, keeping the original instead of overwriting a foreign edit. Every `cp` is checked: +without that, a failed copy would record a manifest checksum of the unchanged destination, and +the restore path would delete the only backup after failing to use it. + ## dispatch-fleet.sh `dispatch-fleet.sh` measures the 54-fixture set in `~/vstack-dispatch/` — recall, precision, diff --git a/tests/auto-trigger.sh b/tests/auto-trigger.sh index 560dee4..5c5a648 100755 --- a/tests/auto-trigger.sh +++ b/tests/auto-trigger.sh @@ -13,7 +13,12 @@ set -uo pipefail PER_CASE_TIMEOUT=120 # seconds; enforced by the polling loop in run_case (macOS has no timeout(1)) MODEL="sonnet" -MAX_TURNS=3 +# Env-overridable as of 2026-09-01, for probing turn starvation without editing the file. +# The 2026-08-23 note above case_max_turns records the last such probe being run through a +# throwaway script precisely because this was a literal -- and an uncommitted instrument is why +# that arm's numbers were withdrawn. The value lands in every SAMPLES runlog row, so an arm run +# at a different budget cannot be silently mixed with one run at the default. +MAX_TURNS="${MAX_TURNS:-3}" # Per-case turn budget, measured not guessed (2026-08-23), enforced by case_max_turns() below. # MAX_TURNS above is still the global default -- a case with no row here is not in the case # statement below either, and inherits MAX_TURNS unchanged, same as before this table existed. @@ -151,6 +156,31 @@ case_max_turns() { # honest about "does this situation ever route there" without crying wolf. ATTEMPTS="${ATTEMPTS:-3}" +# --------------------------------------------------------------------------- +# SAMPLES=N -- rate mode. OFF by default (0): with SAMPLES unset every line below behaves +# exactly as it did before, same output, same exit codes, same gate. +# +# ATTEMPTS is NOT a sample-size knob and raising it will never make one. It early-stops on the +# first hit, so it answers "did this land within N tries" and biases upward -- the same +# early-stopping error that read encode-lessons-lint as a dead skill when it was turn-starved +# (see the note above case_max_turns). SAMPLES runs N INDEPENDENT, non-retrying invocations of +# the same prompt and reports raw k/N. +# +# Report k/N, never a percentage and never an interval: at the n this mode is affordable at, a +# point estimate implies precision the sample does not have. +# +# Measurement is not a verdict. A low rate does NOT fail the run -- only a fence breach does. +# --------------------------------------------------------------------------- +SAMPLES="${SAMPLES:-0}" +SAMPLES_ALL="${SAMPLES_ALL:-0}" # required to sample every case; see the cost guard below +SAMPLE_LOG="${SAMPLE_LOG:-}" # optional path; one JSON line per sample, append-only +SAMPLE_CASES=0 +SAMPLE_CALLS=0 +SAMPLE_COST="0" +SAMPLE_HEAD="" +SAMPLE_DESCS="" +AT_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" + PASS_COUNT=0 FAIL_COUNT=0 declare -a RESULT_LINES=() @@ -313,6 +343,47 @@ top_level_subtype() { jq -r 'select(.type=="result" and (.origin==null)) | .subtype' "$jsonl" 2>/dev/null | tail -1 } +# --------------------------------------------------------------------------- +# top_level_cost OUT_JSONL +# Billed cost of ONE invocation, read from the same terminal result event top_level_subtype +# reads: the one with no `origin`. A subagent's result event carries origin.kind == +# "task-notification" and its own total_cost_usd. Agent/Workflow/Explore/Task are denied by +# DISALLOWED_TOOLS, so there should be no second result event here; if that fence is ever +# loosened this becomes an undercount, and this sentence is the warning. +# Prints 0 when the field is absent, so a timed-out or crashed sample contributes 0 rather +# than breaking the running total. +# +# Nothing in this file counted a billed call before this existed. A measurement that cannot +# say what it cost cannot be budgeted, and every prior dispatch number here was published +# without one. +# --------------------------------------------------------------------------- +top_level_cost() { + local jsonl="$1" + jq -r 'select(.type=="result" and (.origin==null)) | (.total_cost_usd // 0)' "$jsonl" 2>/dev/null | tail -1 +} + +# --------------------------------------------------------------------------- +# named_in_prose OUT_JSONL REGEX +# Prints 1 if the run's assistant TEXT names a skill matching REGEX without having called it. +# +# Measured 2026-08-27 (tests/evals/collision/RESULTS.md, "Why nothing fired"): when the tool a +# skill needs is denied -- Write for a plan document, Agent for a fan-out -- the model names +# the skill in prose and never calls Skill. Under `fired=[]` that scores identically to a +# routing miss. Separating the two is the difference between "the description does not route" +# and "the description routes and the fence blocks the payoff", and DISALLOWED_TOOLS denies +# exactly the tools the four chain skills exist to use. +# +# A 0/10 printed without this number beside it is uninterpretable. +# +# Herestring, not a pipe: `grep -q` closes the pipe early and returns 141 under `pipefail`. +# --------------------------------------------------------------------------- +named_in_prose() { + local jsonl="$1" regex="$2" txt + txt="$(jq -r 'select(.type=="assistant") | (.message.content // [])[] + | select(.type=="text") | .text' "$jsonl" 2>/dev/null)" + if grep -qE "$regex" <<<"$txt"; then echo 1; else echo 0; fi +} + # --------------------------------------------------------------------------- # hash_snapshot DIR # One "hash path" line per regular file under DIR (find -type f -- directories are excluded, @@ -393,7 +464,7 @@ fence_violations() { # run_case NAME PROMPT EXPECTED_REGEX SETUP_FN # SETUP_FN is a function name invoked with the temp dir as $1, or "" for none. # --------------------------------------------------------------------------- -# Case selection. Without it, proving one fix means re-running all 28 cases and spending the +# Case selection. Without it, proving one fix means re-running all 30 cases and spending the # whole allowance to learn about eight of them. Matches install-matrix.sh's convention: # no arguments runs everything, arguments name the cases to run. # tests/auto-trigger.sh # all cases @@ -406,9 +477,146 @@ selected_() { return 1 } +# --------------------------------------------------------------------------- +# Provenance for anything SAMPLES mode publishes. +# +# The rule in this repository is that an instrument is committed before the run it produces a +# number for -- a 55-sample collision result was withdrawn precisely because its harness edit +# was never committed and the runlog is gone (CHANGELOG.md, 1.47.0). Recording the instrument +# sha AND a digest of the description bytes in every sample row is what makes a published k/N +# checkable afterwards instead of merely asserted, because in this suite the *subject* of the +# measurement is the description text and the *instrument* is this file. Both have to be +# pinned or the number cannot be reproduced. +# +# The descriptions digested here are the INSTALLED ones, not the repo's: the CLI runs in a +# scratch workdir and reads ~/.claude/skills. Measured 2026-09-01: a project-local +# .claude/skills//SKILL.md IS loaded, but a user-level skill of the same name WINS -- +# the installed description is what reaches the matcher, and a repo edit is invisible here +# until it is installed. Digesting the wrong tree would pin a number to bytes the model never +# saw. +# --------------------------------------------------------------------------- +SAMPLE_HEAD="$(git -C "$AT_ROOT" rev-parse --short HEAD 2>/dev/null || echo unknown)" +SAMPLE_DESCS="$(grep -h '^description:' "$HOME"/.claude/skills/*/SKILL.md 2>/dev/null \ + | "$SUM_CMD" | cut -c1-12)" + +if (( SAMPLES > 0 )) \ + && [ -n "$(git -C "$AT_ROOT" status --porcelain -- tests/auto-trigger.sh 2>/dev/null)" ]; then + echo "WARNING: tests/auto-trigger.sh is uncommitted. A number from an uncommitted instrument" + echo " is inadmissible here. Commit the harness, then re-run." +fi + +# Cost guard. Nothing in this file counted a call before spending it. SAMPLES across the whole +# suite is SAMPLES x every case, which at SAMPLES=10 is several hundred billed invocations from +# one keystroke and no confirmation. +if (( SAMPLES > 0 )) && [ ${#SELECTED[@]} -eq 0 ] && [ "$SAMPLES_ALL" != "1" ]; then + echo "SAMPLES=$SAMPLES with no case named would run $SAMPLES x $(grep -cE '^run_(negative_)?case ' "$0") invocations." + echo "Name the cases you want, or set SAMPLES_ALL=1 to mean it." + exit 2 +fi + +# --------------------------------------------------------------------------- +# sample_case NAME PROMPT REGEX SETUP_FN POLARITY +# +# SAMPLES independent invocations of one prompt, no early stop, raw k/N. POLARITY is "pos" +# (REGEX names the skill that SHOULD fire) or "neg" (REGEX names skills that must NOT); it +# changes the printed label only, because a fire rate and a false-positive rate are the same +# arithmetic pointed at different prompts. +# +# Deliberately never touches PASS_COUNT/FAIL_COUNT except on a fence breach. A rate is a +# measurement, not a verdict: wiring a low rate to a red exit would make this suite refuse to +# report the very thing it was run to find out. +# --------------------------------------------------------------------------- +sample_case() { + local name="$1" prompt="$2" regex="$3" setup_fn="$4" polarity="$5" + local i fired term cost prose_hit hits=0 prose=0 cutoff=0 case_cost="0" + local workdir out_jsonl err_log runner_pid waited baseline baseline_hashes violations + local case_turns; case_turns="$(case_max_turns "$name")" + + for i in $(seq 1 "$SAMPLES"); do + workdir="$(mktemp -d "/tmp/auto-trigger-sample.XXXXXX")" + [[ -n "$setup_fn" ]] && "$setup_fn" "$workdir" + baseline="$(find "$workdir" -mindepth 1 2>/dev/null | sort)" + baseline_hashes="$(hash_snapshot "$workdir")" + out_jsonl="$workdir/.out.jsonl"; err_log="$workdir/.err.log" + + # Same exec/timeout discipline as run_case: without exec the kill -9 hits an empty parent + # and leaks a live, billed claude session per timed-out sample. + ( + cd "$workdir" || exit 1 + exec env -u ANTHROPIC_API_KEY -u ANTHROPIC_AUTH_TOKEN \ + claude -p "$prompt" \ + --output-format stream-json --verbose \ + --disallowedTools "$DISALLOWED_TOOLS" \ + --model "$MODEL" --max-turns "$case_turns" \ + < /dev/null > "$out_jsonl" 2> "$err_log" + ) & + runner_pid=$! + waited=0 + while kill -0 "$runner_pid" 2>/dev/null; do + sleep 1 + waited=$((waited + 1)) + if (( waited >= PER_CASE_TIMEOUT )); then kill -9 "$runner_pid" 2>/dev/null; break; fi + done + wait "$runner_pid" 2>/dev/null + + fired=""; term="no-output"; cost=0; prose_hit=0 + if [[ -s "$out_jsonl" ]]; then + fired="$(extract_fired_skills "$out_jsonl")" + term="$(top_level_subtype "$out_jsonl")" + cost="$(top_level_cost "$out_jsonl")" + prose_hit="$(named_in_prose "$out_jsonl" "$regex")" + fi + [[ "$term" == "error_max_turns" ]] && cutoff=$((cutoff + 1)) + if [[ -n "$fired" ]] && grep -qE "^($regex)$" <<<"$fired"; then + hits=$((hits + 1)) + elif [[ "$prose_hit" == "1" ]]; then + prose=$((prose + 1)) + fi + # bash 3.2 has no float arithmetic; awk does the addition. + case_cost="$(awk -v a="$case_cost" -v b="${cost:-0}" 'BEGIN{printf "%.6f", a+b}')" + SAMPLE_CALLS=$((SAMPLE_CALLS + 1)) + + violations="$(fence_violations "$workdir" "$baseline" "$baseline_hashes" "$out_jsonl" "$err_log")" + if [[ -n "$SAMPLE_LOG" ]]; then + # jq builds the row, not printf. `fired` comes from .input.skill, a model-controlled + # tool-call argument: nothing constrains it to be quote-free, and a bare %s interpolation + # of a value containing a double quote emits a line that fails `jq .`. That would corrupt + # the runlog silently -- the same runlog the provenance block above calls the thing that + # makes a published k/N checkable. A measurement log that can be broken by its own subject + # is not a record. + jq -cn \ + --arg case "$name" --argjson sample "$i" --arg polarity "$polarity" \ + --arg fired "$(tr '\n' ' ' <<<"$fired" | sed 's/ *$//')" --arg subtype "$term" \ + --argjson named_in_prose "${prose_hit:-0}" --argjson cost_usd "${cost:-0}" \ + --arg model "$MODEL" --argjson max_turns "$case_turns" \ + --arg instrument "$SAMPLE_HEAD" --arg descs "$SAMPLE_DESCS" \ + '{case:$case,sample:$sample,polarity:$polarity,fired:$fired,subtype:$subtype, + named_in_prose:$named_in_prose,cost_usd:$cost_usd,model:$model, + max_turns:$max_turns,instrument:$instrument,descs:$descs}' \ + >> "$SAMPLE_LOG" + fi + rm -rf "$workdir" + + if [[ -n "$violations" ]]; then + echo "FAIL $name -> fence breach on sample $i of $SAMPLES: $(tr '\n' ';' <<<"$violations")" + FAIL_COUNT=$((FAIL_COUNT + 1)) + return + fi + done + + SAMPLE_COST="$(awk -v a="$SAMPLE_COST" -v b="$case_cost" 'BEGIN{printf "%.6f", a+b}')" + SAMPLE_CASES=$((SAMPLE_CASES + 1)) + printf 'SAMPLE %-30s %-3s fired %d/%d named-not-called %d/%d cut-off %d/%d $%s (max-turns=%s)\n' \ + "$name" "$polarity" "$hits" "$SAMPLES" "$prose" "$SAMPLES" "$cutoff" "$SAMPLES" \ + "$case_cost" "$case_turns" + HIT_LINES+=("$(printf '%-30s %-3s %d/%d fired, %d/%d named-only, %d/%d cut off, $%s' \ + "$name" "$polarity" "$hits" "$SAMPLES" "$prose" "$SAMPLES" "$cutoff" "$SAMPLES" "$case_cost")") +} + run_case() { local name="$1" prompt="$2" expected_regex="$3" setup_fn="$4" selected_ "$name" || return 0 + if (( SAMPLES > 0 )); then sample_case "$name" "$prompt" "$expected_regex" "$setup_fn" pos; return; fi local attempt fired fired_csv matched workdir out_jsonl err_log runner_pid waited local baseline baseline_hashes violations term_subtype last_fired_csv="" local case_turns; case_turns="$(case_max_turns "$name")" @@ -514,6 +722,9 @@ run_case() { run_negative_case() { local name="$1" prompt="$2" forbidden_regex="$3" selected_ "$name" || return 0 + # In SAMPLES mode the forbidden regex is exactly what we want a rate for: k/N here IS the + # over-trigger rate of the new descriptions, measured for free alongside the positive arms. + if (( SAMPLES > 0 )); then sample_case "$name" "$prompt" "$forbidden_regex" "" neg; return; fi local workdir out_jsonl err_log runner_pid waited fired fired_csv baseline baseline_hashes violations local case_turns; case_turns="$(case_max_turns "$name")" @@ -738,6 +949,122 @@ type Order struct { EOF } +# A spec that is settled and is NOT a plan. The distinction is the whole fixture: brainstorming's +# situation ("the shape is undecided") is starved by "agreed, no open questions", and +# executing-plans' situation ("a written plan already exists") is starved by the last line +# saying in as many words that no file map or ordering exists anywhere. +setup_spec() { + local dir="$1" + setup_webapp "$dir" + cat > "$dir/SPEC.md" <<'SPECEOF' +# Spec: offline mode for the notes app + +Status: agreed 2026-08-30. No open questions. The shape is not up for discussion. + +Behaviour: +- Notes stay editable with no network; edits queue locally. +- On reconnect the queue drains oldest first; a note edited in two places keeps the newer timestamp. +- The header reads exactly one of: online, offline, syncing. +- No new dependencies. Storage is localStorage. + +Out of scope: accounts, multi-device, any conflict UI beyond last-write-wins. + +Written down nowhere: which files change, in what order, or how each step gets checked. +SPECEOF +} + +# A real defect, a real suite, and a gap between them. The suite existing is what separates this +# from create-verification-skill ("nothing exists yet"); the suite being current is what +# separates it from maintain-verification-skill (a gate that drifted). +setup_bugfix() { + local dir="$1" + cat > "$dir/package.json" <<'PKGEOF' +{ + "name": "notes-app", + "version": "1.0.0", + "type": "module", + "scripts": { "test": "node --test" } +} +PKGEOF + cat > "$dir/notes.js" <<'NOTESEOF' +const notes = []; + +export function addNote(text) { + notes.push({ text, at: 0 }); + return notes.length; +} + +export function listNotes() { + return notes; +} + +export function clearNotes() { + notes.length = 0; +} +NOTESEOF + cat > "$dir/notes.test.js" <<'TESTEOF' +import { test } from "node:test"; +import assert from "node:assert"; +import { addNote, listNotes, clearNotes } from "./notes.js"; + +test("addNote stores the text it was given", () => { + clearNotes(); + addNote("buy milk"); + assert.strictEqual(listNotes()[0].text, "buy milk"); +}); +TESTEOF +} + +# Six integrations, six unrelated defects, no imports between them: a master-password backdoor, +# an unauthenticated non-idempotent refund, path traversal, a Math.random session id, an +# unverified webhook, and CSV injection. Six independent verdicts is real independent work, not +# one finding restated six times -- which is what makes this a swarm situation rather than a +# single review. +setup_audit() { + local dir="$1" + mkdir -p "$dir/services" + cat > "$dir/services/auth.js" <<'AUTHEOF' +export function login(user, pass) { + if (pass === process.env.MASTER_PASSWORD) return { user, admin: true }; + return null; +} +AUTHEOF + cat > "$dir/services/billing.js" <<'BILLEOF' +export function refund(chargeId, amount) { + return fetch("https://api.example.com/charges/" + chargeId + "/refund", { + method: "POST", + body: JSON.stringify({ amount }), + }); +} +BILLEOF + cat > "$dir/services/upload.js" <<'UPEOF' +export function saveUpload(name, bytes) { + return { path: "/var/uploads/" + name, size: bytes.length }; +} +UPEOF + cat > "$dir/services/session.js" <<'SESSEOF' +const sessions = new Map(); + +export function newSession(user) { + const id = String(Math.random()).slice(2); + sessions.set(id, user); + return id; +} +SESSEOF + cat > "$dir/services/webhook.js" <<'HOOKEOF' +export function handle(req) { + const body = JSON.parse(req.body); + return { ok: true, event: body.type }; +} +HOOKEOF + cat > "$dir/services/export.js" <<'CSVEOF' +export function toCsv(rows) { + return rows.map(function (r) { return Object.values(r).join(","); }).join("\n"); +} +CSVEOF +} + + # --------------------------------------------------------------------------- # Test cases # --------------------------------------------------------------------------- @@ -757,11 +1084,24 @@ run_case \ "typescript-best-practices" \ "setup_typescript" +# Rewritten 2026-09-01. The old prompt was "Audit this directory three different ways at once" +# with no setup, so it tested two things this suite is not for: whether the model obeys the +# literal string "at once" (which swarm's own description quoted back at it, making the case a +# keyword echo rather than a routing test), and whether it can audit an empty directory -- the +# no-referent confound in tests/evals/collision/RESULTS.md. +# +# All four quoted trigger strings are gone -- "in parallel", "at once", "all of these", +# "try N ways" -- and splittability is stated as a property of the WORK, which is the thing a +# description matcher can actually score. The routing-table line in the SessionStart hook still +# carries the literal strings; this case deliberately exercises the other path. +# +# If this measures 0/N with a high named-not-called, the blocker is the Agent denial in +# DISALLOWED_TOOLS, not the description, and the arm says so rather than reporting a dead skill. run_case \ "swarm-audit" \ - "Audit this directory three different ways at once and report back." \ + "Each file in services/ is a separate integration with its own failure mode, and none of them import each other. I want a security verdict per file, and what you find in one tells you nothing about the next. Come back with one verdict per file." \ "swarm" \ - "" + "setup_audit" run_case \ "blast-radius-auth" \ @@ -941,6 +1281,31 @@ run_case \ "principle-type-system-discipline" \ "setup_gostruct" +# writing-plans and test-driven-development had no isolated case until 2026-09-01. Their only +# positive coverage was feature-chain's three-way alternation, which any one of the three can +# satisfy on its own -- so both could be, and were, dead at zero lifetime dispatches while this +# suite stayed green. A gate that cannot go red for a dead skill is not measuring that skill. + +# Discriminator, three ways. SPEC.md says "agreed ... no open questions", so brainstorming has +# nothing to explore. Nothing on disk is a plan, so executing-plans has nothing to execute. And +# the ask is for the ordered steps IN THE REPLY, not for a file: Write is denied by the fence, +# and a fixture whose payoff needs a denied tool scores a blocked affordance as a routing miss. +run_case \ + "writing-plans-spec-to-steps" \ + "SPEC.md is agreed and there are no open questions on it. Nothing anywhere says which files change, in what order, or how we would know each step worked. Work that out and give it to me before anyone touches app.js." \ + "writing-plans" \ + "setup_spec" + +# Discriminator: the cause is stated outright, which starves principle-fix-root-causes (its +# situation is debugging, or reaching for a try/except). A suite exists and demonstrably does not +# cover this path. "test" appears in the prompt only as a description of what is already on disk +# -- the ask is for the fix -- so a hit here is routing rather than keyword echo. +run_case \ + "tdd-known-bugfix" \ + "addNote('') files an empty note instead of rejecting it. The cause is not a mystery: addNote never looks at text. Wanted behaviour is that an empty or whitespace-only string is rejected and the list is left alone. notes.test.js covers the happy path only. Make the change." \ + "test-driven-development" \ + "setup_bugfix" + run_negative_case \ "negative-arithmetic" \ "What is 17 times 23? Just the number." \ @@ -957,11 +1322,25 @@ echo "---" # which is the same shape as a clean run. A typo'd case name must not read as a pass. # This sits above the hit-rate loop because HIT_LINES is unset when nothing ran, and # `set -u` turns that into a stack trace rather than an answer. -if (( PASS_COUNT + FAIL_COUNT == 0 )); then +if (( PASS_COUNT + FAIL_COUNT + SAMPLE_CASES == 0 )); then echo "no case ran. ${#SELECTED[@]} selector(s) given, none matched a case name." exit 2 fi +if (( SAMPLES > 0 )); then + echo "Rates (raw k/N, no early stop):" + for l in "${HIT_LINES[@]}"; do echo " $l"; done + echo + echo "Sample mode: SAMPLES=$SAMPLES, $SAMPLE_CASES case(s), $SAMPLE_CALLS invocation(s), total_cost_usd=$SAMPLE_COST" + echo " model=$MODEL max-turns-default=$MAX_TURNS instrument=$SAMPLE_HEAD installed-descs=$SAMPLE_DESCS" + echo " fence=$DISALLOWED_TOOLS" + echo " k/N is a measurement, not a gate: a low rate does not fail this run. A fence breach does." + echo " Read named-not-called beside every k/N -- a 0/N with a high named-not-called is the" + echo " denied-affordance wall (tests/evals/collision/RESULTS.md), not a dead description." + if (( FAIL_COUNT > 0 )); then exit 1; fi + exit 0 +fi + echo "Hit rates (which attempt each case landed on):" for l in "${HIT_LINES[@]}"; do echo " $l"; done echo diff --git a/tests/description-collision.sh b/tests/description-collision.sh new file mode 100755 index 0000000..7be4f4c --- /dev/null +++ b/tests/description-collision.sh @@ -0,0 +1,237 @@ +#!/usr/bin/env bash +# tests/description-collision.sh +# +# Scores the shipped skill descriptions for shared trigger n-grams and names the pairs that +# overlap. No model calls, no network, no writes outside a self-test temp dir. +# +# WHAT THIS IS NOT +# +# This is not evidence that overlapping triggers suppress dispatch. That claim was published +# here as "measured over 80 samples to suppress both skills, not one" and was formally withdrawn +# in 1.47.0: no 80-sample runlog exists in this repository's history or on the machine that +# produced it (CHANGELOG.md, docs/research/fake-greens-2026-08.md). The replacement arm was +# pre-registered before the first sample and run on a committed instrument +# (tests/evals/collision/PREREGISTRATION.md, RESULTS.md, n=25, 2026-08-27); it found the +# hypothesis NOT supported -- the collision fixture fired more than either clean control -- and +# instead confirmed that the harness supplies no situation and denies the tools several skills +# exist to use. +# +# So this script encodes DESIGN DISCIPLINE, not a measured law. The discipline is the one +# aebeebb applied by hand to six pairs: put the discriminator in the first clause, because that +# is the part a matcher weighs hardest, and do not let two skills open on the same words. It is +# cheap, it is deterministic, and it is worth keeping green for the same reason a style rule is +# -- not because a violation is proven to break dispatch. +# +# SCORING +# +# Each description splits at its first ':', '.' or em dash into HEAD (the discriminator +# clause) and TAIL. Shared 2- and 3-grams are scored per pair, weighted by where they land: +# 3 for HEAD/HEAD, 2 for one HEAD, 1 for TAIL/TAIL. +# +# THE ONLY FAILING CONDITION +# +# A 3-gram appearing in BOTH skills' HEAD. That is exactly the defect aebeebb fixed, where +# grill-me and interrogate both opened on "tear this apart". Everything else prints and +# passes, and the threshold is echoed on every run so a green cannot be read as "no overlap". +# +# Usage: +# tests/description-collision.sh # score claude/skills/*/SKILL.md +# tests/description-collision.sh --self-test # prove the check can fail, then that it passes +# tests/description-collision.sh DIR # score an arbitrary skills dir +set -uo pipefail + +# One line, deliberately. BSD awk rejects a newline inside a -v assignment ("newline in +# string") and then exits non-zero having scored nothing -- which the first run of this +# script's own self-test caught, printing a clean rc=0 on the live corpus while awk had +# failed on every invocation. A wrapped stoplist is a green that measures nothing. +STOP=" use when the a an and or of to in on for it is are this that with before after one its you your any not no into from at by as be been has have had they them their we our i me my do does did so than then there here what which who whom whose will would can could should may might must if else while each per via over under again more most other some such only own same too very just now new old first second third next last thing things something anything nothing " + +# --------------------------------------------------------------------------- +# extract_descriptions DIR +# Prints "skillnamedescription" per skill. Uses the same closed-frontmatter-block rule as +# .claude/verify.sh check 3 rather than a bare grep: a line that merely looks like frontmatter +# further down a body must not be scored as a description. +# --------------------------------------------------------------------------- +extract_descriptions() { + local dir="$1" f name + for f in "$dir"/*/SKILL.md; do + [ -f "$f" ] || continue + name="$(basename "$(dirname "$f")")" + awk -v skill="$name" ' + NR==1 { if ($0 != "---") exit; infm=1; next } + infm && $0 == "---" { exit } + infm && /^description:[[:space:]]/ { + line=$0 + sub(/^description:[[:space:]]*/, "", line) + gsub(/^"|"$/, "", line) + printf "%s\t%s\n", skill, line + exit + } + ' "$f" + done +} + +# --------------------------------------------------------------------------- +# score SKILLS_DIR +# Emits the pair table and the corpus total on stdout. Returns 1 if any HEAD/HEAD 3-gram exists. +# All grouping lives in awk: bash 3.2 has no associative arrays, which is also why the rest of +# this suite avoids them. +# --------------------------------------------------------------------------- +score() { + local dir="$1" descs n_desc out rc + descs="$(extract_descriptions "$dir")" + n_desc="$(printf '%s' "$descs" | grep -c . || true)" + # An empty or unreadable corpus must not read as "no collisions". Same failure shape the + # gate-falsifiability work in this repo exists to catch: a check that passes without running. + if [ "${n_desc:-0}" -lt 2 ]; then + echo "REFUSING: extracted $n_desc description(s) from $dir -- nothing to compare." >&2 + return 2 + fi + # LC_ALL=C is load-bearing. Several descriptions carry an em dash, and BSD awk under a UTF-8 + # locale aborts on it with "illegal byte sequence" -- caught by the REFUSING guard below on the + # first live run, which is the only reason it was not published as a clean corpus. Under C the + # em dash is three bytes that [^a-z0-9]+ strips like any other punctuation, and the byte + # comparison in the HEAD/TAIL split below is written against those bytes deliberately. + out="$(printf '%s\n' "$descs" | LC_ALL=C awk -F'\t' -v stop="$STOP" ' + function norm(s, t) { + t = tolower(s) + gsub(/[^a-z0-9]+/, " ", t) + return t + } + function emit(skill, text, zone, n, w, i, j, tok, out) { + n = split(norm(text), w, " ") + j = 0 + for (i = 1; i <= n; i++) { + tok = w[i] + if (length(tok) < 3) continue + if (index(stop, " " tok " ") > 0) continue + j++; keep[j] = tok + } + for (i = 1; i <= j - 1; i++) { + g = keep[i] " " keep[i+1] + seen[g, skill] = zone; grams[g] = 1 + } + for (i = 1; i <= j - 2; i++) { + g = keep[i] " " keep[i+1] " " keep[i+2] + seen[g, skill] = zone; grams[g] = 1 + len[g] = 3 + } + } + { + skill = $1; desc = $2 + skills[++ns] = skill + # HEAD is everything up to the first ":", "." or em dash; TAIL is the rest. + cut = 0 + for (i = 1; i <= length(desc); i++) { + ch = substr(desc, i, 1) + two = substr(desc, i, 3) + if (ch == ":" || ch == "." || two == "\342\200\224") { cut = i; break } + } + if (cut == 0) { head = desc; tail = "" } + else { head = substr(desc, 1, cut - 1); tail = substr(desc, cut + 1) } + emit(skill, head, "H") + emit(skill, tail, "T") + } + END { + total = 0; hard = 0 + for (g in grams) { + nsk = 0 + for (i = 1; i <= ns; i++) { + s = skills[i] + if ((g SUBSEP s) in seen) { owners[++nsk] = s; zones[nsk] = seen[g, s] } + } + if (nsk < 2) continue + gl = (g in len) ? 3 : 2 + for (a = 1; a <= nsk - 1; a++) for (b = a + 1; b <= nsk; b++) { + za = zones[a]; zb = zones[b] + wt = (za == "H" && zb == "H") ? 3 : ((za == "H" || zb == "H") ? 2 : 1) + pair = (owners[a] < owners[b]) ? owners[a] " / " owners[b] : owners[b] " / " owners[a] + pscore[pair] += gl * wt + pgram[pair] = pgram[pair] (pgram[pair] == "" ? "" : ", ") g + total += gl * wt + if (za == "H" && zb == "H" && gl == 3) { + hard++ + hardline[hard] = pair ": \"" g "\" in both HEAD clauses" + } + } + } + for (p in pscore) printf "%6d %s: %s\n", pscore[p], p, pgram[p] | "sort -rn" + close("sort -rn") + printf "\ncorpus score %d across %d skills\n", total, ns + if (hard > 0) { + printf "\nHARD FAIL -- a 3-gram opens two descriptions:\n" + for (i = 1; i <= hard; i++) printf " %s\n", hardline[i] + exit 1 + } + printf "hard-fail pairs: none\n" + } + ')" + rc=$? + printf '%s\n' "$out" + # awk exiting 2 (a parse or runtime error) is not "no collisions found". Distinguish it from + # the deliberate exit 1, or a broken scorer reports a clean corpus forever. + if [ "$rc" -gt 1 ]; then + echo "REFUSING: awk failed (rc=$rc) -- the corpus was not scored." >&2 + return 2 + fi + return "$rc" +} + +# --------------------------------------------------------------------------- +# Self-test. CONTRIBUTING.md requires a check be seen failing before it is trusted, and this one +# carries its own mutation rather than leaning on tests/gate-falsifiability.sh, which is owned +# elsewhere. Plants a HEAD/HEAD 3-gram, asserts exit 1 and that the planted pair is named; then +# scores a clean corpus and asserts exit 0. +# --------------------------------------------------------------------------- +self_test() { + local t rc out fails=0 + t="$(mktemp -d "${TMPDIR:-/tmp}/desc-collision-selftest.XXXXXX")" + + mk() { + mkdir -p "$t/$1" + printf -- '---\nname: %s\ndescription: "%s"\n---\n# %s\n' "$1" "$2" "$1" > "$t/$1/SKILL.md" + } + + # The planted 3-gram has to survive normalisation to be a fair test of the scorer. "Tear this + # apart" does not: "this" is a stopword, so it drops to the 2-gram "tear apart" and the check + # correctly declines to fail. Content words only. + mk alpha "Shipping risky payment migration code today: review the blast radius first." + mk beta "Shipping risky payment migration code today: interview every reviewer first." + out="$(score "$t")"; rc=$? + if [ "$rc" -ne 1 ]; then + echo "SELF-TEST FAIL: planted HEAD/HEAD 3-gram did not fail the check (rc=$rc)"; fails=1 + elif ! grep -q 'alpha / beta' <<<"$out"; then + echo "SELF-TEST FAIL: failed, but did not name the planted pair"; fails=1 + else + echo "self-test 1/2 ok: planted collision fails and names alpha / beta" + fi + + rm -rf "${t:?}"/* + mk alpha "A settled spec with nothing written down: produce ordered steps." + mk beta "A running dev server and a component that renders wrong: screenshot every breakpoint." + out="$(score "$t")"; rc=$? + if [ "$rc" -ne 0 ]; then + echo "SELF-TEST FAIL: clean corpus was rejected (rc=$rc)"; echo "$out"; fails=1 + else + echo "self-test 2/2 ok: clean corpus passes" + fi + + rm -rf "$t" + return "$fails" +} + +REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" + +case "${1:-}" in + --self-test) self_test; exit $? ;; + "") DIR="$REPO_ROOT/claude/skills" ;; + *) DIR="$1" ;; +esac + +echo "Scoring shared trigger n-grams in $DIR" +echo "Failing condition: a 3-gram appearing in BOTH skills' first clause. Nothing else fails." +echo "This is design discipline, not a dispatch measurement -- see the header, and" +echo "tests/evals/collision/RESULTS.md for what a committed instrument actually found." +echo "---" +score "$DIR" +exit $? diff --git a/tests/evals/autofire/RESULTS.md b/tests/evals/autofire/RESULTS.md new file mode 100644 index 0000000..103f9c2 --- /dev/null +++ b/tests/evals/autofire/RESULTS.md @@ -0,0 +1,179 @@ +# Autofire arm: do the four chain skills route from the situation alone? + +**Status: complete. Read the significance caveat under the after table before quoting any number +from it.** + +| | | +|---|---| +| Question | Do `writing-plans`, `test-driven-development`, `executing-plans` and `swarm` fire from a situation, with no user instruction naming them? | +| Instrument | `tests/auto-trigger.sh` SAMPLES mode. Every row logs the repo HEAD it ran at, `901bb3f`, which contains the mode itself (`d6ca391`) and the fixtures. | +| Model | `sonnet` (CLI alias), OAuth session, `ANTHROPIC_API_KEY` stripped by the harness | +| Tool fence | `Write,Edit,MultiEdit,NotebookEdit,Bash,Agent,Workflow,Explore,Task` | +| Seed / temperature | Neither. The CLI exposes no such knob, which is why n>1 exists. | +| Installed descriptions | `d4251fb4eda7` on all 60 baseline rows, `5cf618030b60` on all 40 after rows. One value per arm, no pooling. | +| Sampling | N=10 independent invocations per arm, no early stop. `ATTEMPTS` is unused. | + +Raw k/N only. At n=10 a percentage or an interval would imply precision this sample does not +have, so the harness prints k/N. + +Runlogs, one JSON object per sample: +`runlog-baseline-turns3-writing-plans-spec-to-steps.jsonl`, +`runlog-baseline-turns3-swarm-audit.jsonl`, +`runlog-baseline-turns3-executing-plans-checkpoints.jsonl`, +`runlog-baseline-turns3-tdd-known-bugfix.jsonl`, +`runlog-baseline-turns12-executing-plans-checkpoints.jsonl`, +`runlog-baseline-turns12-tdd-known-bugfix.jsonl`. + +## Baseline at the suite default, max-turns=3 + +40 calls, $3.17. + +| case | skill | fired | named-not-called | cut off | +|---|---|---|---|---| +| `writing-plans-spec-to-steps` | `writing-plans` | 9/10 | 0/10 | 10/10 | +| `swarm-audit` | `swarm` | 10/10 | 0/10 | 10/10 | +| `executing-plans-checkpoints` | `executing-plans` | 0/10 | 0/10 | 10/10 | +| `tdd-known-bugfix` | `test-driven-development` | 0/10 | 0/10 | 10/10 | + +Every sample terminated `error_max_turns`, 40 of 40, including the two arms that fired. So being +cut off separates nothing here, and it cannot be offered as the reason the other two scored zero. +The runlog carries `fired` and `subtype` and no turn trace, so it cannot say where in a run the +decision was or was not reached. What it does say is that the two zeros were taken at a budget +that every single sample exhausted, which is enough reason to re-run them and not enough reason +to publish them. + +## The same two arms at max-turns=12 + +20 calls, $5.61. `MAX_TURNS` became env-overridable at `17f918b` so this could run on a committed +instrument. The last person to probe turn budget here used a throwaway script and the numbers +were withdrawn for exactly that. + +| case | skill | fired | named-not-called | cut off | misroutes | +|---|---|---|---|---|---| +| `tdd-known-bugfix` | `test-driven-development` | 3/10 | 0/10 | 0/10 | none | +| `executing-plans-checkpoints` | `executing-plans` | 1/10 | 0/10 | 0/10 | 2/10 to `test-driven-development` | + +Cut-off drops to zero at this budget, so both arms now run to completion and choose. Turn budget +was carrying part of the TDD result: 0/10 to 3/10. It was carrying almost none of the +`executing-plans` result: 0/10 to 1/10. + +`executing-plans` is the finding. Its prompt is "PLAN.md is the implementation plan we approved +last session. Execute it, and pause for my review after each phase." Its unedited description is +"Use when you have a written implementation plan to execute in a separate session with review +checkpoints." Those are close to the same sentence, and it still fires 1 time in 10. Whatever is +suppressing this skill, it is not that the description fails to describe the situation. Two of +the nine misses went to `test-driven-development` instead, on a prompt with no test in it, which +is the one piece of direct evidence in this run that these two descriptions compete. + +## What the baseline overturns + +The brief behind this work said `writing-plans`, `test-driven-development` and `executing-plans` +have fired zero times ever, and that `swarm` fires only when the user says "in parallel". The +first half is true of the transcript corpus (`tests/transcript-census.sh`, 2536 transcripts, +re-run 2026-09-01) and I am not disputing it. + +Little of it carries over to this harness. `writing-plans` fires 9/10 on its unedited +description. `swarm` fires 10/10 on a prompt carrying none of the four trigger strings its own +description quotes, so "in parallel", "at once", "all of these" and "try N ways" are not what +carries it. For those two there is no description defect visible here and no headroom to fix one. +`test-driven-development` is weak rather than dead. Only `executing-plans` is close to dead, and +it is closest to dead where its description matches its prompt best. + +### The caveat that limits all of this + +`writing-plans-spec-to-steps` shares wording with the description it tests. The fixture says +"SPEC.md is agreed" against a description reading "once the shape is agreed", and "before anyone +touches app.js" against "before code". That is not a quotation, but it is the same two hinges, +and a fixture built from a description's own hinges will fire more often than a real request +does. + +The gap between 9/10 here and zero in the real corpus is best explained by real prompts not +stating the situation this plainly. That is a claim about how people phrase requests, not about +how the description is worded, and this arm cannot separate the two. + +Do not read this file as "the skills are fine". It says something narrower. Given a prompt that +states the situation outright, two of four route reliably, one routes weakly, one barely routes +at all, and the harness could not have told you any of that before, because its fixtures had no +referent and its only knob stopped at the first hit. + +## Collision score + +From `tests/description-collision.sh`. No model calls. + +| | corpus score | hard-fail pairs | +|---|---|---| +| unedited (28 skills) | 23 | none | +| with the four rewrites | 13 | none | + +The 10-point drop is exactly the two overlaps the ordinal framing created: +`test-driven-development` against `writing-plans` on "once shape", worth 6, and `brainstorming` +against `test-driven-development` on "shape known", worth 4. Three descriptions opening +`Nth, once the shape is X` compete on the same tokens in the clause a matcher weighs hardest, +and the `executing-plans` arm shows `test-driven-development` taking prompts that are not its +own. + +This is design discipline, not a dispatch law. The claim that colliding triggers suppress both +skills was withdrawn in 1.47.0, and the pre-registered replacement found the hypothesis +unsupported (`tests/evals/collision/RESULTS.md`). + +## After the rewrite + +All four rewritten descriptions were installed together for this arm, by +`tests/stage-skill-descriptions.sh`, and restored afterwards. Digest `5cf618030b60` on all 40 +rows. Each arm is budget-matched to its own baseline. 40 calls, $7.41. + +Runlogs: `tests/evals/autofire/runlog-after-writing-plans-spec-to-steps.jsonl`, +`runlog-after-swarm-audit.jsonl`, `runlog-after-tdd-known-bugfix.jsonl`, +`runlog-after-executing-plans-checkpoints.jsonl`. + +| skill | budget | before | after | Fisher p, two-sided | reading | +|---|---|---|---|---|---| +| `executing-plans` | 12 | 1/10 | 6/10 | 0.0573 | moved, not significant | +| `test-driven-development` | 12 | 3/10 | 3/10 | 1.0 | unchanged | +| `writing-plans` | 3 | 9/10 | 10/10 | 1.0 | no regression, no headroom | +| `swarm` | 3 | 10/10 | 10/10 | 1.0 | no regression, no headroom | + +### The threshold passed and the statistic did not + +Before the arm I registered: an arm counts as fixed when baseline is at most 1/10 and after is at +least 6/10. `executing-plans` came in at exactly 1/10 and 6/10, so the threshold is met. + +It should not have been written that way. I derived "6/10" against an assumed 0/10 baseline, +where it gives p=0.0108 and clears the Bonferroni alpha of 0.0125 that four tests want. The +measured baseline was 1/10, and I never went back and re-derived the bound. Against 1/10, 6/10 is +p=0.0573. It clears neither 0.0125 nor 0.05. + +So the check passed while the thing the check stood for did not. That is the failure +`docs/checks-that-inherit-their-answer.md` catalogues, and this is an instance of it in my own +pre-registration rather than in someone else's. The honest statement of this result is that +`executing-plans` moved from 1/10 to 6/10 and that n=10 cannot establish the move. Separating +those two rates at Bonferroni alpha needs roughly n=20 to 25 per arm, about $12 more at this +budget. + +`test-driven-development` did not move at all. The rewrite bought it a description that asserts a +checkable property instead of an invisible chain position, and bought no dispatch. Both are in +the runlog. + +The `executing-plans` misroute did not close. Two of ten samples still fire +`test-driven-development` on a prompt with no test in it, the same 2 as at baseline. The rewrite +narrows what makes those two compete without removing it. + +## Cost + +The harness records this per arm. Nobody estimated it. Baseline at turns=3 cost $3.17 across 40 +calls, $0.079 each. The turns=12 re-run cost $5.61 across 20 calls, $0.281 each. The after arm +cost $7.41 across 40 calls. Total for the published arms: $16.19 across 100 calls. + +## Instrument changes after the baseline, and why they do not invalidate it + +Two defects were found by review after the baseline ran and fixed before the after-arm. + +The runlog row was built by `printf` with a bare `%s` for `fired`, which comes from a +model-controlled tool-call argument. A value containing a double quote emits a line that fails +`jq .`. It is built by `jq -cn --arg` now. No baseline row was affected: every `fired` value in +all 60 rows is a plain skill name, and the file parses. + +`stage-skill-descriptions.sh` had unchecked `cp` calls in both directions. That script took no +part in the baseline, which ran against the installed descriptions unmodified. + +Neither change alters what a sample measures, so the baseline and after arms remain comparable. diff --git a/tests/evals/autofire/runlog-after-executing-plans-checkpoints.jsonl b/tests/evals/autofire/runlog-after-executing-plans-checkpoints.jsonl new file mode 100644 index 0000000..0460322 --- /dev/null +++ b/tests/evals/autofire/runlog-after-executing-plans-checkpoints.jsonl @@ -0,0 +1,10 @@ +{"case":"executing-plans-checkpoints","sample":1,"polarity":"pos","fired":"executing-plans","subtype":"success","named_in_prose":0,"cost_usd":0.22186679999999998,"model":"sonnet","max_turns":12,"instrument":"f728a7e","descs":"5cf618030b60"} +{"case":"executing-plans-checkpoints","sample":2,"polarity":"pos","fired":"executing-plans","subtype":"success","named_in_prose":1,"cost_usd":0.261513,"model":"sonnet","max_turns":12,"instrument":"f728a7e","descs":"5cf618030b60"} +{"case":"executing-plans-checkpoints","sample":3,"polarity":"pos","fired":"","subtype":"success","named_in_prose":0,"cost_usd":0.33977540000000006,"model":"sonnet","max_turns":12,"instrument":"f728a7e","descs":"5cf618030b60"} +{"case":"executing-plans-checkpoints","sample":4,"polarity":"pos","fired":"executing-plans","subtype":"success","named_in_prose":1,"cost_usd":0.3473942,"model":"sonnet","max_turns":12,"instrument":"f728a7e","descs":"5cf618030b60"} +{"case":"executing-plans-checkpoints","sample":5,"polarity":"pos","fired":"test-driven-development","subtype":"success","named_in_prose":0,"cost_usd":0.330076,"model":"sonnet","max_turns":12,"instrument":"f728a7e","descs":"5cf618030b60"} +{"case":"executing-plans-checkpoints","sample":6,"polarity":"pos","fired":"executing-plans","subtype":"success","named_in_prose":0,"cost_usd":0.31307380000000007,"model":"sonnet","max_turns":12,"instrument":"f728a7e","descs":"5cf618030b60"} +{"case":"executing-plans-checkpoints","sample":7,"polarity":"pos","fired":"","subtype":"success","named_in_prose":0,"cost_usd":0.149817,"model":"sonnet","max_turns":12,"instrument":"f728a7e","descs":"5cf618030b60"} +{"case":"executing-plans-checkpoints","sample":8,"polarity":"pos","fired":"test-driven-development","subtype":"success","named_in_prose":0,"cost_usd":0.38165439999999995,"model":"sonnet","max_turns":12,"instrument":"f728a7e","descs":"5cf618030b60"} +{"case":"executing-plans-checkpoints","sample":9,"polarity":"pos","fired":"executing-plans","subtype":"success","named_in_prose":1,"cost_usd":0.37266160000000004,"model":"sonnet","max_turns":12,"instrument":"f728a7e","descs":"5cf618030b60"} +{"case":"executing-plans-checkpoints","sample":10,"polarity":"pos","fired":"executing-plans","subtype":"success","named_in_prose":1,"cost_usd":0.2113092,"model":"sonnet","max_turns":12,"instrument":"f728a7e","descs":"5cf618030b60"} diff --git a/tests/evals/autofire/runlog-after-swarm-audit.jsonl b/tests/evals/autofire/runlog-after-swarm-audit.jsonl new file mode 100644 index 0000000..fd2c271 --- /dev/null +++ b/tests/evals/autofire/runlog-after-swarm-audit.jsonl @@ -0,0 +1,10 @@ +{"case":"swarm-audit","sample":1,"polarity":"pos","fired":"security-review swarm","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.08565840000000001,"model":"sonnet","max_turns":3,"instrument":"f728a7e","descs":"5cf618030b60"} +{"case":"swarm-audit","sample":2,"polarity":"pos","fired":"swarm","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.10247700000000001,"model":"sonnet","max_turns":3,"instrument":"f728a7e","descs":"5cf618030b60"} +{"case":"swarm-audit","sample":3,"polarity":"pos","fired":"swarm","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.0861888,"model":"sonnet","max_turns":3,"instrument":"f728a7e","descs":"5cf618030b60"} +{"case":"swarm-audit","sample":4,"polarity":"pos","fired":"swarm","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.0863274,"model":"sonnet","max_turns":3,"instrument":"f728a7e","descs":"5cf618030b60"} +{"case":"swarm-audit","sample":5,"polarity":"pos","fired":"swarm","subtype":"error_max_turns","named_in_prose":1,"cost_usd":0.084047,"model":"sonnet","max_turns":3,"instrument":"f728a7e","descs":"5cf618030b60"} +{"case":"swarm-audit","sample":6,"polarity":"pos","fired":"swarm","subtype":"error_max_turns","named_in_prose":1,"cost_usd":0.08337960000000001,"model":"sonnet","max_turns":3,"instrument":"f728a7e","descs":"5cf618030b60"} +{"case":"swarm-audit","sample":7,"polarity":"pos","fired":"swarm","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.0881008,"model":"sonnet","max_turns":3,"instrument":"f728a7e","descs":"5cf618030b60"} +{"case":"swarm-audit","sample":8,"polarity":"pos","fired":"swarm","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.08495119999999999,"model":"sonnet","max_turns":3,"instrument":"f728a7e","descs":"5cf618030b60"} +{"case":"swarm-audit","sample":9,"polarity":"pos","fired":"swarm","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.085524,"model":"sonnet","max_turns":3,"instrument":"f728a7e","descs":"5cf618030b60"} +{"case":"swarm-audit","sample":10,"polarity":"pos","fired":"swarm","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.08728860000000001,"model":"sonnet","max_turns":3,"instrument":"f728a7e","descs":"5cf618030b60"} diff --git a/tests/evals/autofire/runlog-after-tdd-known-bugfix.jsonl b/tests/evals/autofire/runlog-after-tdd-known-bugfix.jsonl new file mode 100644 index 0000000..ac0975f --- /dev/null +++ b/tests/evals/autofire/runlog-after-tdd-known-bugfix.jsonl @@ -0,0 +1,10 @@ +{"case":"tdd-known-bugfix","sample":1,"polarity":"pos","fired":"","subtype":"success","named_in_prose":0,"cost_usd":0.291867,"model":"sonnet","max_turns":12,"instrument":"f728a7e","descs":"5cf618030b60"} +{"case":"tdd-known-bugfix","sample":2,"polarity":"pos","fired":"","subtype":"success","named_in_prose":0,"cost_usd":0.28343680000000004,"model":"sonnet","max_turns":12,"instrument":"f728a7e","descs":"5cf618030b60"} +{"case":"tdd-known-bugfix","sample":3,"polarity":"pos","fired":"","subtype":"success","named_in_prose":0,"cost_usd":0.23044199999999998,"model":"sonnet","max_turns":12,"instrument":"f728a7e","descs":"5cf618030b60"} +{"case":"tdd-known-bugfix","sample":4,"polarity":"pos","fired":"","subtype":"success","named_in_prose":0,"cost_usd":0.27989240000000004,"model":"sonnet","max_turns":12,"instrument":"f728a7e","descs":"5cf618030b60"} +{"case":"tdd-known-bugfix","sample":5,"polarity":"pos","fired":"test-driven-development","subtype":"success","named_in_prose":0,"cost_usd":0.2422594,"model":"sonnet","max_turns":12,"instrument":"f728a7e","descs":"5cf618030b60"} +{"case":"tdd-known-bugfix","sample":6,"polarity":"pos","fired":"","subtype":"success","named_in_prose":0,"cost_usd":0.2884586,"model":"sonnet","max_turns":12,"instrument":"f728a7e","descs":"5cf618030b60"} +{"case":"tdd-known-bugfix","sample":7,"polarity":"pos","fired":"","subtype":"success","named_in_prose":0,"cost_usd":0.29732900000000007,"model":"sonnet","max_turns":12,"instrument":"f728a7e","descs":"5cf618030b60"} +{"case":"tdd-known-bugfix","sample":8,"polarity":"pos","fired":"test-driven-development","subtype":"success","named_in_prose":0,"cost_usd":0.374756,"model":"sonnet","max_turns":12,"instrument":"f728a7e","descs":"5cf618030b60"} +{"case":"tdd-known-bugfix","sample":9,"polarity":"pos","fired":"test-driven-development","subtype":"success","named_in_prose":0,"cost_usd":0.25005459999999996,"model":"sonnet","max_turns":12,"instrument":"f728a7e","descs":"5cf618030b60"} +{"case":"tdd-known-bugfix","sample":10,"polarity":"pos","fired":"","subtype":"success","named_in_prose":0,"cost_usd":0.2300466,"model":"sonnet","max_turns":12,"instrument":"f728a7e","descs":"5cf618030b60"} diff --git a/tests/evals/autofire/runlog-after-writing-plans-spec-to-steps.jsonl b/tests/evals/autofire/runlog-after-writing-plans-spec-to-steps.jsonl new file mode 100644 index 0000000..01b5bc4 --- /dev/null +++ b/tests/evals/autofire/runlog-after-writing-plans-spec-to-steps.jsonl @@ -0,0 +1,10 @@ +{"case":"writing-plans-spec-to-steps","sample":1,"polarity":"pos","fired":"writing-plans","subtype":"error_max_turns","named_in_prose":1,"cost_usd":0.0813344,"model":"sonnet","max_turns":3,"instrument":"f728a7e","descs":"5cf618030b60"} +{"case":"writing-plans-spec-to-steps","sample":2,"polarity":"pos","fired":"writing-plans","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.07966320000000002,"model":"sonnet","max_turns":3,"instrument":"f728a7e","descs":"5cf618030b60"} +{"case":"writing-plans-spec-to-steps","sample":3,"polarity":"pos","fired":"writing-plans","subtype":"error_max_turns","named_in_prose":1,"cost_usd":0.0869972,"model":"sonnet","max_turns":3,"instrument":"f728a7e","descs":"5cf618030b60"} +{"case":"writing-plans-spec-to-steps","sample":4,"polarity":"pos","fired":"writing-plans","subtype":"error_max_turns","named_in_prose":1,"cost_usd":0.08967800000000001,"model":"sonnet","max_turns":3,"instrument":"f728a7e","descs":"5cf618030b60"} +{"case":"writing-plans-spec-to-steps","sample":5,"polarity":"pos","fired":"writing-plans","subtype":"error_max_turns","named_in_prose":1,"cost_usd":0.0859346,"model":"sonnet","max_turns":3,"instrument":"f728a7e","descs":"5cf618030b60"} +{"case":"writing-plans-spec-to-steps","sample":6,"polarity":"pos","fired":"writing-plans","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.0865056,"model":"sonnet","max_turns":3,"instrument":"f728a7e","descs":"5cf618030b60"} +{"case":"writing-plans-spec-to-steps","sample":7,"polarity":"pos","fired":"writing-plans","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.0813326,"model":"sonnet","max_turns":3,"instrument":"f728a7e","descs":"5cf618030b60"} +{"case":"writing-plans-spec-to-steps","sample":8,"polarity":"pos","fired":"writing-plans","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.0816522,"model":"sonnet","max_turns":3,"instrument":"f728a7e","descs":"5cf618030b60"} +{"case":"writing-plans-spec-to-steps","sample":9,"polarity":"pos","fired":"writing-plans","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.08035980000000001,"model":"sonnet","max_turns":3,"instrument":"f728a7e","descs":"5cf618030b60"} +{"case":"writing-plans-spec-to-steps","sample":10,"polarity":"pos","fired":"writing-plans","subtype":"error_max_turns","named_in_prose":1,"cost_usd":0.08794320000000001,"model":"sonnet","max_turns":3,"instrument":"f728a7e","descs":"5cf618030b60"} diff --git a/tests/evals/autofire/runlog-baseline-turns12-executing-plans-checkpoints.jsonl b/tests/evals/autofire/runlog-baseline-turns12-executing-plans-checkpoints.jsonl new file mode 100644 index 0000000..3dfa85a --- /dev/null +++ b/tests/evals/autofire/runlog-baseline-turns12-executing-plans-checkpoints.jsonl @@ -0,0 +1,10 @@ +{"case":"executing-plans-checkpoints","sample":1,"polarity":"pos","fired":"","subtype":"success","named_in_prose":0,"cost_usd":0.25070460000000006,"model":"sonnet","max_turns":12,"instrument":"17f918b","descs":"d4251fb4eda7"} +{"case":"executing-plans-checkpoints","sample":2,"polarity":"pos","fired":"","subtype":"success","named_in_prose":0,"cost_usd":0.2707798,"model":"sonnet","max_turns":12,"instrument":"17f918b","descs":"d4251fb4eda7"} +{"case":"executing-plans-checkpoints","sample":3,"polarity":"pos","fired":"","subtype":"success","named_in_prose":0,"cost_usd":0.368047,"model":"sonnet","max_turns":12,"instrument":"17f918b","descs":"d4251fb4eda7"} +{"case":"executing-plans-checkpoints","sample":4,"polarity":"pos","fired":"","subtype":"success","named_in_prose":0,"cost_usd":0.2650154,"model":"sonnet","max_turns":12,"instrument":"17f918b","descs":"d4251fb4eda7"} +{"case":"executing-plans-checkpoints","sample":5,"polarity":"pos","fired":"test-driven-development","subtype":"success","named_in_prose":0,"cost_usd":0.3514748,"model":"sonnet","max_turns":12,"instrument":"17f918b","descs":"d4251fb4eda7"} +{"case":"executing-plans-checkpoints","sample":6,"polarity":"pos","fired":"","subtype":"success","named_in_prose":0,"cost_usd":0.310086,"model":"sonnet","max_turns":12,"instrument":"17f918b","descs":"d4251fb4eda7"} +{"case":"executing-plans-checkpoints","sample":7,"polarity":"pos","fired":"executing-plans","subtype":"success","named_in_prose":1,"cost_usd":0.42729400000000006,"model":"sonnet","max_turns":12,"instrument":"17f918b","descs":"d4251fb4eda7"} +{"case":"executing-plans-checkpoints","sample":8,"polarity":"pos","fired":"test-driven-development","subtype":"success","named_in_prose":0,"cost_usd":0.36620100000000005,"model":"sonnet","max_turns":12,"instrument":"17f918b","descs":"d4251fb4eda7"} +{"case":"executing-plans-checkpoints","sample":9,"polarity":"pos","fired":"","subtype":"success","named_in_prose":0,"cost_usd":0.24200100000000002,"model":"sonnet","max_turns":12,"instrument":"17f918b","descs":"d4251fb4eda7"} +{"case":"executing-plans-checkpoints","sample":10,"polarity":"pos","fired":"","subtype":"success","named_in_prose":0,"cost_usd":0.2987764,"model":"sonnet","max_turns":12,"instrument":"17f918b","descs":"d4251fb4eda7"} diff --git a/tests/evals/autofire/runlog-baseline-turns12-tdd-known-bugfix.jsonl b/tests/evals/autofire/runlog-baseline-turns12-tdd-known-bugfix.jsonl new file mode 100644 index 0000000..bab7e91 --- /dev/null +++ b/tests/evals/autofire/runlog-baseline-turns12-tdd-known-bugfix.jsonl @@ -0,0 +1,10 @@ +{"case":"tdd-known-bugfix","sample":1,"polarity":"pos","fired":"","subtype":"success","named_in_prose":0,"cost_usd":0.2870228,"model":"sonnet","max_turns":12,"instrument":"17f918b","descs":"d4251fb4eda7"} +{"case":"tdd-known-bugfix","sample":2,"polarity":"pos","fired":"test-driven-development","subtype":"success","named_in_prose":0,"cost_usd":0.3048088,"model":"sonnet","max_turns":12,"instrument":"17f918b","descs":"d4251fb4eda7"} +{"case":"tdd-known-bugfix","sample":3,"polarity":"pos","fired":"","subtype":"success","named_in_prose":0,"cost_usd":0.2603012,"model":"sonnet","max_turns":12,"instrument":"17f918b","descs":"d4251fb4eda7"} +{"case":"tdd-known-bugfix","sample":4,"polarity":"pos","fired":"test-driven-development","subtype":"success","named_in_prose":0,"cost_usd":0.3981236,"model":"sonnet","max_turns":12,"instrument":"17f918b","descs":"d4251fb4eda7"} +{"case":"tdd-known-bugfix","sample":5,"polarity":"pos","fired":"","subtype":"success","named_in_prose":0,"cost_usd":0.1043738,"model":"sonnet","max_turns":12,"instrument":"17f918b","descs":"d4251fb4eda7"} +{"case":"tdd-known-bugfix","sample":6,"polarity":"pos","fired":"","subtype":"success","named_in_prose":0,"cost_usd":0.2474064,"model":"sonnet","max_turns":12,"instrument":"17f918b","descs":"d4251fb4eda7"} +{"case":"tdd-known-bugfix","sample":7,"polarity":"pos","fired":"","subtype":"success","named_in_prose":0,"cost_usd":0.2120628,"model":"sonnet","max_turns":12,"instrument":"17f918b","descs":"d4251fb4eda7"} +{"case":"tdd-known-bugfix","sample":8,"polarity":"pos","fired":"test-driven-development","subtype":"success","named_in_prose":0,"cost_usd":0.2736248,"model":"sonnet","max_turns":12,"instrument":"17f918b","descs":"d4251fb4eda7"} +{"case":"tdd-known-bugfix","sample":9,"polarity":"pos","fired":"","subtype":"success","named_in_prose":0,"cost_usd":0.1625512,"model":"sonnet","max_turns":12,"instrument":"17f918b","descs":"d4251fb4eda7"} +{"case":"tdd-known-bugfix","sample":10,"polarity":"pos","fired":"","subtype":"success","named_in_prose":0,"cost_usd":0.2106834,"model":"sonnet","max_turns":12,"instrument":"17f918b","descs":"d4251fb4eda7"} diff --git a/tests/evals/autofire/runlog-baseline-turns3-executing-plans-checkpoints.jsonl b/tests/evals/autofire/runlog-baseline-turns3-executing-plans-checkpoints.jsonl new file mode 100644 index 0000000..72dc7bf --- /dev/null +++ b/tests/evals/autofire/runlog-baseline-turns3-executing-plans-checkpoints.jsonl @@ -0,0 +1,10 @@ +{"case":"executing-plans-checkpoints","sample":1,"polarity":"pos","fired":"","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.0729558,"model":"sonnet","max_turns":3,"instrument":"901bb3f","descs":"d4251fb4eda7"} +{"case":"executing-plans-checkpoints","sample":2,"polarity":"pos","fired":"","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.0718264,"model":"sonnet","max_turns":3,"instrument":"901bb3f","descs":"d4251fb4eda7"} +{"case":"executing-plans-checkpoints","sample":3,"polarity":"pos","fired":"","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.0718664,"model":"sonnet","max_turns":3,"instrument":"901bb3f","descs":"d4251fb4eda7"} +{"case":"executing-plans-checkpoints","sample":4,"polarity":"pos","fired":"","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.0718764,"model":"sonnet","max_turns":3,"instrument":"901bb3f","descs":"d4251fb4eda7"} +{"case":"executing-plans-checkpoints","sample":5,"polarity":"pos","fired":"","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.073731,"model":"sonnet","max_turns":3,"instrument":"901bb3f","descs":"d4251fb4eda7"} +{"case":"executing-plans-checkpoints","sample":6,"polarity":"pos","fired":"","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.0722414,"model":"sonnet","max_turns":3,"instrument":"901bb3f","descs":"d4251fb4eda7"} +{"case":"executing-plans-checkpoints","sample":7,"polarity":"pos","fired":"","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.0717912,"model":"sonnet","max_turns":3,"instrument":"901bb3f","descs":"d4251fb4eda7"} +{"case":"executing-plans-checkpoints","sample":8,"polarity":"pos","fired":"","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.07180639999999999,"model":"sonnet","max_turns":3,"instrument":"901bb3f","descs":"d4251fb4eda7"} +{"case":"executing-plans-checkpoints","sample":9,"polarity":"pos","fired":"","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.07248639999999999,"model":"sonnet","max_turns":3,"instrument":"901bb3f","descs":"d4251fb4eda7"} +{"case":"executing-plans-checkpoints","sample":10,"polarity":"pos","fired":"","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.0719958,"model":"sonnet","max_turns":3,"instrument":"901bb3f","descs":"d4251fb4eda7"} diff --git a/tests/evals/autofire/runlog-baseline-turns3-swarm-audit.jsonl b/tests/evals/autofire/runlog-baseline-turns3-swarm-audit.jsonl new file mode 100644 index 0000000..9cd20fe --- /dev/null +++ b/tests/evals/autofire/runlog-baseline-turns3-swarm-audit.jsonl @@ -0,0 +1,10 @@ +{"case":"swarm-audit","sample":1,"polarity":"pos","fired":"swarm","subtype":"error_max_turns","named_in_prose":1,"cost_usd":0.08846659999999999,"model":"sonnet","max_turns":3,"instrument":"901bb3f","descs":"d4251fb4eda7"} +{"case":"swarm-audit","sample":2,"polarity":"pos","fired":"swarm","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.08691220000000001,"model":"sonnet","max_turns":3,"instrument":"901bb3f","descs":"d4251fb4eda7"} +{"case":"swarm-audit","sample":3,"polarity":"pos","fired":"swarm","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.08592940000000002,"model":"sonnet","max_turns":3,"instrument":"901bb3f","descs":"d4251fb4eda7"} +{"case":"swarm-audit","sample":4,"polarity":"pos","fired":"swarm","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.0871286,"model":"sonnet","max_turns":3,"instrument":"901bb3f","descs":"d4251fb4eda7"} +{"case":"swarm-audit","sample":5,"polarity":"pos","fired":"swarm","subtype":"error_max_turns","named_in_prose":1,"cost_usd":0.08716499999999999,"model":"sonnet","max_turns":3,"instrument":"901bb3f","descs":"d4251fb4eda7"} +{"case":"swarm-audit","sample":6,"polarity":"pos","fired":"swarm","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.0895552,"model":"sonnet","max_turns":3,"instrument":"901bb3f","descs":"d4251fb4eda7"} +{"case":"swarm-audit","sample":7,"polarity":"pos","fired":"swarm","subtype":"error_max_turns","named_in_prose":1,"cost_usd":0.08885860000000001,"model":"sonnet","max_turns":3,"instrument":"901bb3f","descs":"d4251fb4eda7"} +{"case":"swarm-audit","sample":8,"polarity":"pos","fired":"swarm","subtype":"error_max_turns","named_in_prose":1,"cost_usd":0.0891098,"model":"sonnet","max_turns":3,"instrument":"901bb3f","descs":"d4251fb4eda7"} +{"case":"swarm-audit","sample":9,"polarity":"pos","fired":"swarm","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.0850924,"model":"sonnet","max_turns":3,"instrument":"901bb3f","descs":"d4251fb4eda7"} +{"case":"swarm-audit","sample":10,"polarity":"pos","fired":"swarm","subtype":"error_max_turns","named_in_prose":1,"cost_usd":0.0875414,"model":"sonnet","max_turns":3,"instrument":"901bb3f","descs":"d4251fb4eda7"} diff --git a/tests/evals/autofire/runlog-baseline-turns3-tdd-known-bugfix.jsonl b/tests/evals/autofire/runlog-baseline-turns3-tdd-known-bugfix.jsonl new file mode 100644 index 0000000..c154318 --- /dev/null +++ b/tests/evals/autofire/runlog-baseline-turns3-tdd-known-bugfix.jsonl @@ -0,0 +1,10 @@ +{"case":"tdd-known-bugfix","sample":1,"polarity":"pos","fired":"","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.0769434,"model":"sonnet","max_turns":3,"instrument":"901bb3f","descs":"d4251fb4eda7"} +{"case":"tdd-known-bugfix","sample":2,"polarity":"pos","fired":"","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.0805056,"model":"sonnet","max_turns":3,"instrument":"901bb3f","descs":"d4251fb4eda7"} +{"case":"tdd-known-bugfix","sample":3,"polarity":"pos","fired":"","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.0758492,"model":"sonnet","max_turns":3,"instrument":"901bb3f","descs":"d4251fb4eda7"} +{"case":"tdd-known-bugfix","sample":4,"polarity":"pos","fired":"","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.07750119999999999,"model":"sonnet","max_turns":3,"instrument":"901bb3f","descs":"d4251fb4eda7"} +{"case":"tdd-known-bugfix","sample":5,"polarity":"pos","fired":"","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.0757592,"model":"sonnet","max_turns":3,"instrument":"901bb3f","descs":"d4251fb4eda7"} +{"case":"tdd-known-bugfix","sample":6,"polarity":"pos","fired":"","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.0754474,"model":"sonnet","max_turns":3,"instrument":"901bb3f","descs":"d4251fb4eda7"} +{"case":"tdd-known-bugfix","sample":7,"polarity":"pos","fired":"","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.0780272,"model":"sonnet","max_turns":3,"instrument":"901bb3f","descs":"d4251fb4eda7"} +{"case":"tdd-known-bugfix","sample":8,"polarity":"pos","fired":"","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.07568899999999999,"model":"sonnet","max_turns":3,"instrument":"901bb3f","descs":"d4251fb4eda7"} +{"case":"tdd-known-bugfix","sample":9,"polarity":"pos","fired":"","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.07590600000000002,"model":"sonnet","max_turns":3,"instrument":"901bb3f","descs":"d4251fb4eda7"} +{"case":"tdd-known-bugfix","sample":10,"polarity":"pos","fired":"","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.0754956,"model":"sonnet","max_turns":3,"instrument":"901bb3f","descs":"d4251fb4eda7"} diff --git a/tests/evals/autofire/runlog-baseline-turns3-writing-plans-spec-to-steps.jsonl b/tests/evals/autofire/runlog-baseline-turns3-writing-plans-spec-to-steps.jsonl new file mode 100644 index 0000000..672a528 --- /dev/null +++ b/tests/evals/autofire/runlog-baseline-turns3-writing-plans-spec-to-steps.jsonl @@ -0,0 +1,10 @@ +{"case":"writing-plans-spec-to-steps","sample":1,"polarity":"pos","fired":"writing-plans","subtype":"error_max_turns","named_in_prose":1,"cost_usd":0.07922980000000002,"model":"sonnet","max_turns":3,"instrument":"901bb3f","descs":"d4251fb4eda7"} +{"case":"writing-plans-spec-to-steps","sample":2,"polarity":"pos","fired":"writing-plans","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.08088900000000002,"model":"sonnet","max_turns":3,"instrument":"901bb3f","descs":"d4251fb4eda7"} +{"case":"writing-plans-spec-to-steps","sample":3,"polarity":"pos","fired":"","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.0769404,"model":"sonnet","max_turns":3,"instrument":"901bb3f","descs":"d4251fb4eda7"} +{"case":"writing-plans-spec-to-steps","sample":4,"polarity":"pos","fired":"writing-plans","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.08080079999999999,"model":"sonnet","max_turns":3,"instrument":"901bb3f","descs":"d4251fb4eda7"} +{"case":"writing-plans-spec-to-steps","sample":5,"polarity":"pos","fired":"writing-plans","subtype":"error_max_turns","named_in_prose":1,"cost_usd":0.08173699999999999,"model":"sonnet","max_turns":3,"instrument":"901bb3f","descs":"d4251fb4eda7"} +{"case":"writing-plans-spec-to-steps","sample":6,"polarity":"pos","fired":"writing-plans","subtype":"error_max_turns","named_in_prose":1,"cost_usd":0.085802,"model":"sonnet","max_turns":3,"instrument":"901bb3f","descs":"d4251fb4eda7"} +{"case":"writing-plans-spec-to-steps","sample":7,"polarity":"pos","fired":"writing-plans","subtype":"error_max_turns","named_in_prose":1,"cost_usd":0.0787162,"model":"sonnet","max_turns":3,"instrument":"901bb3f","descs":"d4251fb4eda7"} +{"case":"writing-plans-spec-to-steps","sample":8,"polarity":"pos","fired":"writing-plans","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.0787144,"model":"sonnet","max_turns":3,"instrument":"901bb3f","descs":"d4251fb4eda7"} +{"case":"writing-plans-spec-to-steps","sample":9,"polarity":"pos","fired":"writing-plans","subtype":"error_max_turns","named_in_prose":1,"cost_usd":0.08795660000000001,"model":"sonnet","max_turns":3,"instrument":"901bb3f","descs":"d4251fb4eda7"} +{"case":"writing-plans-spec-to-steps","sample":10,"polarity":"pos","fired":"writing-plans","subtype":"error_max_turns","named_in_prose":0,"cost_usd":0.0766312,"model":"sonnet","max_turns":3,"instrument":"901bb3f","descs":"d4251fb4eda7"} diff --git a/tests/evals/build-the-lever/PREREGISTRATION.md b/tests/evals/build-the-lever/PREREGISTRATION.md index ef4c5ff..dc1b8cb 100644 --- a/tests/evals/build-the-lever/PREREGISTRATION.md +++ b/tests/evals/build-the-lever/PREREGISTRATION.md @@ -101,7 +101,8 @@ request does not move dispatch, the locus is not the constraint. `"Every file in modules/ needs the same license header pasted at the top. I'll go through and add it to each one by hand."` The user has announced they are doing the work themselves. Nothing is asked -of the model. Across all 28 cases in `tests/auto-trigger.sh`, exactly two prompts are declaratives +of the model. Across all 28 auto-trigger cases in `tests/auto-trigger.sh` at pre-registration time, +exactly two prompts are declaratives with no imperative and no question directed at the model — this one, and `prove-it-works-declare-done` (`"I fixed the crash in fetch_stats by adding a retry, and the code compiles cleanly -- this task is done."`). Those are also exactly the two cases measured at 0/10. diff --git a/tests/stage-skill-descriptions.sh b/tests/stage-skill-descriptions.sh new file mode 100755 index 0000000..be41caa --- /dev/null +++ b/tests/stage-skill-descriptions.sh @@ -0,0 +1,149 @@ +#!/usr/bin/env bash +# tests/stage-skill-descriptions.sh +# +# Copies the repo's SKILL.md description line for named skills into ~/.claude/skills, and puts +# it back afterwards. +# +# WHY THIS HAS TO EXIST +# +# tests/auto-trigger.sh runs the CLI in a scratch workdir, so the descriptions that reach the +# matcher are the INSTALLED ones under ~/.claude/skills. Measured 2026-09-01: a project-local +# .claude/skills//SKILL.md is loaded and listed, but a user-level skill of the same name +# WINS -- asked to quote its own listing for `swarm`, the model returned the installed text and +# reported no duplicate. So a description edited in a checkout is invisible to the harness, and +# an after-arm measured without installing would measure the before-arm's bytes and call it a +# result. +# +# Running install.sh instead is not an option here: it installs the whole tree, including +# whatever hook edits another session has in flight. +# +# WHY THE RESTORE REFUSES +# +# A save/restore harness in this repo once deleted two agents' uncommitted work by restoring +# over a change it did not make. --restore therefore refuses any file whose content differs from +# what --stage wrote, and says so, rather than overwriting a foreign edit. The snapshot records +# a checksum per file, not just the bytes. +# +# --stage likewise refuses to start if the installed copy differs from the repo copy at HEAD, +# because that means someone else is mid-edit and staging would silently take their change with +# it into a measurement. +# +# Usage: +# tests/stage-skill-descriptions.sh --stage swarm writing-plans +# tests/stage-skill-descriptions.sh --status +# tests/stage-skill-descriptions.sh --restore +set -uo pipefail + +REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +SKILLS_SRC="$REPO_ROOT/claude/skills" +SKILLS_DST="$HOME/.claude/skills" +STATE_DIR="${TMPDIR:-/tmp}/vstack-skill-stage" + +if command -v shasum >/dev/null 2>&1; then SUM=shasum +elif command -v sha256sum >/dev/null 2>&1; then SUM=sha256sum +else echo "need shasum or sha256sum" >&2; exit 2; fi + +sumof() { "$SUM" < "$1" | awk '{print $1}'; } + +desc_of() { + awk 'NR==1 { if ($0 != "---") exit; infm=1; next } + infm && $0 == "---" { exit } + infm && /^description:[[:space:]]/ { print; exit }' "$1" +} + +do_stage() { + local name src dst staged=0 + [ "$#" -gt 0 ] || { echo "--stage needs at least one skill name" >&2; exit 2; } + mkdir -p "$STATE_DIR" + if [ -e "$STATE_DIR/manifest" ]; then + echo "REFUSING: $STATE_DIR/manifest already exists -- a previous stage was never restored." >&2 + echo " Run --restore (or --status) first." >&2 + exit 2 + fi + + # Pre-flight every skill before touching any of them: a half-applied stage is worse than none. + for name in "$@"; do + src="$SKILLS_SRC/$name/SKILL.md"; dst="$SKILLS_DST/$name/SKILL.md" + [ -f "$src" ] || { echo "REFUSING: no repo skill at $src" >&2; exit 2; } + [ -f "$dst" ] || { echo "REFUSING: no installed skill at $dst" >&2; exit 2; } + if ! git -C "$REPO_ROOT" show "HEAD:claude/skills/$name/SKILL.md" 2>/dev/null \ + | diff -q - "$dst" >/dev/null 2>&1; then + echo "REFUSING: installed $name differs from repo HEAD. Someone else is mid-edit, or an" >&2 + echo " earlier stage leaked. Staging would carry their change into the arm." >&2 + exit 2 + fi + done + + : > "$STATE_DIR/manifest" + for name in "$@"; do + src="$SKILLS_SRC/$name/SKILL.md"; dst="$SKILLS_DST/$name/SKILL.md" + # Every cp is checked. There is no `set -e` here, so an unchecked failure would record a + # manifest entry whose checksum is of the UNCHANGED destination -- the arm would then run on + # stale bytes and the runlog would swear they were the new ones. + if ! cp "$dst" "$STATE_DIR/$name.orig"; then + echo "REFUSING: could not back up $dst -- nothing further staged." >&2; exit 2 + fi + if ! cp "$src" "$dst"; then + echo "REFUSING: could not install $src -> $dst. Run --restore to undo what did land." >&2 + exit 2 + fi + printf '%s\t%s\t%s\n' "$name" "$dst" "$(sumof "$dst")" >> "$STATE_DIR/manifest" + staged=$((staged + 1)) + printf 'staged %-28s %s\n' "$name" "$(desc_of "$dst" | cut -c1-72)..." + done + echo "staged $staged skill(s). Installed digest is now: $(installed_digest)" + echo "Run --restore when the arm is finished." +} + +installed_digest() { + grep -h '^description:' "$SKILLS_DST"/*/SKILL.md 2>/dev/null | "$SUM" | cut -c1-12 +} + +do_restore() { + local name dst want have rc=0 n=0 + [ -f "$STATE_DIR/manifest" ] || { echo "nothing staged (no $STATE_DIR/manifest)"; exit 0; } + while IFS=$'\t' read -r name dst want; do + [ -n "$name" ] || continue + have="$(sumof "$dst")" + if [ "$have" != "$want" ]; then + echo "REFUSING $name: $dst changed since it was staged." >&2 + echo " staged=$want now=$have -- restoring would destroy a foreign edit." >&2 + echo " Original is preserved at $STATE_DIR/$name.orig" >&2 + rc=1 + continue + fi + # Checked for the same reason, and harder: if this cp fails and rc stays 0, the rm -rf below + # deletes the only surviving copy of the original. + if ! cp "$STATE_DIR/$name.orig" "$dst"; then + echo "REFUSING $name: restore cp failed. Original kept at $STATE_DIR/$name.orig" >&2 + rc=1 + continue + fi + n=$((n + 1)) + printf 'restored %-28s\n' "$name" + done < "$STATE_DIR/manifest" + if [ "$rc" -eq 0 ]; then + rm -rf "$STATE_DIR" + echo "restored $n skill(s), state cleared. Installed digest: $(installed_digest)" + else + echo "restored $n skill(s); state KEPT because at least one refused." >&2 + fi + return "$rc" +} + +do_status() { + echo "installed digest: $(installed_digest)" + if [ -f "$STATE_DIR/manifest" ]; then + echo "STAGED (not yet restored):" + cut -f1 "$STATE_DIR/manifest" | sed 's/^/ /' + else + echo "nothing staged" + fi +} + +case "${1:---status}" in + --stage) shift; do_stage "$@" ;; + --restore) do_restore ;; + --status) do_status ;; + *) echo "usage: $0 --stage NAME... | --restore | --status" >&2; exit 2 ;; +esac