Skill-description tuning: measured fire rates, collision score 23->13, routing table alignment - #6
Merged
Merged
Conversation
ATTEMPTS cannot answer "what fraction of the time does this skill fire". It early-stops on the first hit, so it measures "landed within 3 tries" and biases upward -- the file has said so above case_max_turns since 2026-08-23, and that early-stopping bias is what read encode-lessons-lint as a dead skill when it was turn-starved. Every before/after claim about a description needs a real rate, so SAMPLES=N runs N independent, non-retrying invocations and prints raw k/N. Default 0: with SAMPLES unset every path, output line and exit code is unchanged, and run_case's body is untouched -- the branch is a guard prepended to it. Two numbers ship alongside k/N because k/N alone is not interpretable here. total_cost_usd, read from the terminal result event with no `origin` (the same discriminator top_level_subtype already uses to avoid a subagent's result line). Nothing in this file counted a billed call before spending it; a full default run is up to 80 invocations and no line of output said so. named-not-called, from the assistant's prose. Measured 2026-08-27 in tests/evals/collision/RESULTS.md: when the tool a skill needs is denied, the model names the skill in prose and never calls Skill, which scores identically to a routing miss under fired=[]. DISALLOWED_TOOLS denies Write and Agent, which are exactly what the planning-chain skills and swarm exist to use. A 0/N without this number beside it cannot distinguish "the description does not route" from "it routes and the fence blocks the payoff". Sample rows pin both halves of the measurement: the instrument sha and a digest of the INSTALLED description bytes. Measured 2026-09-01: a project-local .claude/skills/<name>/SKILL.md is loaded, but a user-level skill of the same name wins, so the installed copy is what reaches the matcher and a repo edit is invisible to this harness until installed. Digesting the repo tree would pin a number to bytes the model never saw. This is the artifact discipline 1.47.0 withdrew two published numbers for lacking. A low rate does not fail the run -- only a fence breach does. Wiring a rate to a red exit would make the suite refuse to report the thing it was run to find out. Verified: bash -n; shellcheck -S warning clean; tests/dispatch-static.sh green, 4/4 assertions, 28 cases still declared and matched, 0 model calls. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…keyword writing-plans and test-driven-development had no isolated case. Their only positive coverage was feature-chain's three-way alternation across brainstorming|writing-plans|test-driven-development, which any one of the three satisfies alone. Both skills sit at zero lifetime dispatches across 2536 transcripts (tests/transcript-census.sh, re-run 2026-09-01) and this suite stayed green throughout. A gate that cannot go red for a dead skill is not measuring that skill. Each new case gets a setup_ function, because a fixture without one measures something else. Measured 2026-08-27 (tests/evals/collision/RESULTS.md): in an empty workdir the prompt has no referent, the model asks which repository is meant, and that scores identically to a routing miss. Every prompt here names files that exist. setup_spec seeds a SPEC.md that is settled and is explicitly not a plan, so brainstorming has nothing to explore and executing-plans has nothing to execute. The ask is for ordered steps in the reply rather than a file, because Write is denied by the fence and a fixture whose payoff needs a denied tool cannot distinguish a blocked affordance from a routing miss. setup_bugfix seeds a real defect plus a suite that demonstrably misses it. The prompt states the cause outright, which starves principle-fix-root-causes; the suite existing starves create-verification-skill and maintain-verification-skill. swarm-audit is rewritten rather than added. The old prompt was "Audit this directory three different ways at once" with no setup, and swarm's own description quoted "at once" back at it -- so the case measured whether the model echoes a keyword the description already contains, on an empty directory. That is a keyword test wearing a routing test's name. The new prompt seeds six non-importing integrations with six unrelated defects and states splittability as a property of the work. All four quoted trigger strings are gone. The SessionStart routing table still carries them; this case exercises the other path deliberately. Six count claims move with the suite. Four are live totals and are renumbered 28 -> 30. Two are historical measurements and are reworded instead: docs/how-skills-fire.md's routing-table arm and tests/evals/build-the-lever/PREREGISTRATION.md both describe what a past arm measured, and renumbering them to match today's tree would restate a finished measurement as something it was not. Breaking the number-noun adjacency is the device this repo already uses for "falsifiability rows"; extending .claude/verify.sh's exempt_phrases was not an option, that file is owned elsewhere this week. Verified: bash -n; shellcheck -S warning clean; tests/dispatch-static.sh green, 37 fixtures parsed clean (was 25), 30 cases declared and matched; .claude/verify.sh check 12 reads "30 test cases". The gate's four other failures (version, inventory, payload_digest, derived_at.head) reproduce identically on pristine 1bfb133 and are the in-flight release, not this branch. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
No model calls. Splits each description at its first ':', '.' or em dash into the discriminator clause and the rest, then scores shared 2- and 3-grams per pair, weighted 3/2/1 by whether the overlap lands in both first clauses, one, or neither. The only failing condition is a 3-gram opening two descriptions -- the defect aebeebb fixed by hand when grill-me and interrogate both led with "tear this apart". Everything else prints and passes. The header says what this is not, at length, because the claim it would be mistaken for is withdrawn. "Colliding triggers were measured over 80 samples to suppress both skills" has no surviving runlog and was retracted in 1.47.0; the pre-registered replacement at tests/evals/collision/ found the hypothesis unsupported on a committed instrument and located the real mechanism in the harness instead. So this encodes design discipline -- discriminator in the first clause, no two skills opening on the same words -- and explicitly not a dispatch law. Its own self-test found three defects in it before the live corpus did, which is the argument for carrying one: The stoplist was a wrapped shell string. BSD awk rejects a newline inside a -v assignment and exits non-zero having scored nothing, so the live corpus printed rc=0 with awk failing on every invocation -- a green that measured nothing, the exact shape docs/checks-that-inherit-their-answer catalogues. Now one line, with the reason on it. score() returned awk's status without distinguishing a parse error from "no collisions". It now refuses on rc>1 and on a corpus of fewer than two descriptions, rather than reporting a clean result it never computed. That guard is what caught the third defect on the first live run: BSD awk aborts on the em dash in several descriptions with "illegal byte sequence" under a UTF-8 locale. LC_ALL=C is now pinned, and the byte comparison in the HEAD/TAIL split is written against those bytes deliberately. The planted self-test collision was also wrong to begin with: "tear this apart" does not survive stopword removal, dropping to a 2-gram the check correctly declines to fail. The plant is content words now. Verified: bash -n; shellcheck -S warning clean; --self-test green both directions (planted collision fails and names the pair, clean corpus passes). Live corpus scores 23 across 28 skills, no hard-fail pairs, with the two ordinal-frame overlaps visible as test-driven-development/writing-plans "once shape" (6) and brainstorming/test-driven-development "shape known" (4). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…nt arms Two instrument gaps the baseline arm walked straight into. MAX_TURNS was a literal. The baseline run came back cut off by error_max_turns on 40 of 40 samples, which means two of the four arms scored 0/10 without the run ever reaching a decision point -- the exact turn-starvation signature that read encode-lessons-lint as a dead skill, and which this file has warned about above case_max_turns since 2026-08-23. Probing that needed either a source edit or a throwaway script, and the last time someone chose the throwaway script the resulting numbers were withdrawn for having no committed instrument. It is an env var now, and it lands in every runlog row so arms run at different budgets cannot be silently pooled. stage-skill-descriptions.sh exists because a description edited in a checkout is invisible to this harness. Measured 2026-09-01: tests/auto-trigger.sh runs the CLI in a scratch workdir, so the descriptions that reach the matcher are the installed ones under ~/.claude/skills, and while a project-local .claude/skills/<name>/SKILL.md IS loaded and listed, a user-level skill of the same name wins -- asked to quote its own listing for `swarm`, the model returned the installed text and reported no duplicate. An after-arm measured without installing would measure the before-arm's bytes and publish it as a result. Running install.sh instead is not available this week: it installs the whole tree, including another session's in-flight hook edits. Both directions refuse rather than guess. --stage aborts if any installed copy already differs from repo HEAD, because that means someone else is mid-edit and staging would carry their change into the arm; it pre-flights every named skill before touching any, so a half-applied stage cannot happen. --restore compares each file against the checksum it wrote and refuses that file if it changed, keeping the original rather than overwriting a foreign edit -- a save/restore harness in this repo has already destroyed two agents' uncommitted work by doing the opposite. Verified: bash -n; shellcheck -S warning clean on both; --status reads the live installed digest (d4251fb4eda7) and reports nothing staged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review found both of these after the baseline ran and before any after-arm, which is the only reason the numbers below are still usable. The runlog row was built by printf with a bare %s for `fired`. That value comes from .input.skill, a model-controlled tool-call argument, and nothing constrains it to be quote-free; a value containing a double quote emits a line that fails `jq .`. The runlog is the thing the provenance block calls the reason a published k/N is checkable, so a log its own subject can break is not a record. It is built by `jq -cn --arg` now, verified valid for values containing a quote, a backslash and a newline, with the old printf form reproducing the failure on the same input. No baseline row was affected: every `fired` value across all 60 rows is a plain skill name and the files parse. stage-skill-descriptions.sh had unchecked cp calls in both directions. On the stage side a failed copy would still write a manifest entry whose checksum is of the unchanged destination, so an arm would run on stale bytes while the runlog swore they were new. On the restore side a failed copy still printed "restored" and left rc at 0, after which the rm -rf deleted the only surviving backup. Both are checked and both refuse now. That script took no part in the baseline. Neither fix changes what a sample measures, so the arms stay comparable. THE BASELINE, N=10 per arm, no early stop, unedited descriptions. At the suite default of max-turns=3, 40 calls, $3.17: writing-plans 9/10, swarm 10/10, executing-plans 0/10, test-driven-development 0/10. Every sample terminated error_max_turns, 40 of 40, including both arms that fired -- so being cut off separates nothing and cannot be offered as the reason for the two zeros. It is reason enough to re-take them. At max-turns=12, 20 calls, $5.61, cut-off drops to 0/10: test-driven-development 3/10, executing-plans 1/10. Turn budget was carrying part of the TDD result and almost none of the executing-plans one. executing-plans is the finding. Its prompt and its unedited description are close to the same sentence, and it fires once in ten. Whatever suppresses it, it is not that the description fails to describe the situation. Two of its nine misses fired test-driven-development instead, on a prompt containing no test, which is the only direct evidence in this run that those two descriptions compete for the same prompts. Two of the brief's premises do not survive. writing-plans fires 9/10 unedited. swarm fires 10/10 on a prompt carrying none of the four trigger strings its own description quotes, so the literals are not what carries it. The transcript-census zeros are real; they are about how real prompts are phrased, and this harness cannot separate that from how descriptions are worded. RESULTS.md says so rather than claiming the credit. Verified: .claude/verify.sh check 12 reads 30 test cases and check 31 (referrers) is green at 164 files. The four remaining gate failures -- version, inventory, payload_digest, derived_at.head -- reproduce identically on pristine 1bfb133 and belong to the in-flight release. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…tion 1/10 -> 6/10, N=10 per arm, no early stop, max-turns=12 both arms. The old description was "Use when you have a written implementation plan to execute in a separate session with review checkpoints". The fixture prompt is "PLAN.md is the implementation plan we approved last session. Execute it, and pause for my review after each phase." Those are close to the same sentence and it fired once in ten, so the problem was never that the description failed to describe the situation. Two of the nine baseline misses fired test-driven-development instead, on a prompt with no test in it. That is what changed. Not the wording of the situation, which was already right, but that three sibling descriptions all opened on `Nth, once the shape is X` and competed for the same tokens in the clause a matcher weighs hardest. tests/description-collision.sh scores the corpus 23 before and 13 after, and the entire 10-point drop is those two overlaps. READ THE NUMBER WITH ITS CAVEAT. 6/10 meets the threshold I registered before the arm -- baseline <=1/10 and after >=6/10 -- and does not reach significance. Two-sided Fisher on 1/10 against 6/10 is p=0.0573, which clears neither 0.05 nor the Bonferroni 0.0125 that four tests want. The threshold was derived assuming a 0/10 baseline, where 6/10 would have given p=0.0108. The measured baseline was 1/10 and I never re-derived it. So the check passed while the thing the check stood for did not, which is the defect docs/checks-that-inherit-their-answer.md exists to catalogue, committed here in my own pre-registration. Separating 1/10 from 6/10 at that alpha needs roughly n=20-25 per arm, about $12 more at this budget, and that call is not mine to make. The misroute did not close: still 2/10 to test-driven-development. Whatever makes these two compete is narrowed by the rewrite, not removed. The runlog beside this commit is the after arm for this case. All four descriptions were installed together when it ran (digest 5cf618030b60 on every row), because the four are designed to be mutually exclusive OF EACH OTHER and a rate taken while three still carried ordinals would measure a tree that never ships. So this file's 6/10 is attributable to the shipped set, not to this one description in isolation. Description is 168 chars against check 3's 200 cap. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…it changed nothing 3/10 -> 3/10, N=10 per arm, no early stop, max-turns=12 both arms. Two-sided Fisher p=1.0. The rewrite did not move this skill and I am not going to present it as though it did. What it did do is remove the reason the old wording was indefensible. "Third, once the shape is known, including any bugfix" asserts a position in a chain the matcher cannot see: it was never told there is a chain, and nothing in a single prompt lets it verify that this is step three of four. The chain lines live in the SessionStart routing block, injected once, not attached to any description. The replacement asserts a property of the prompt instead, which is at least checkable against what the model has in front of it. It also cost this skill both of its trigger-token overlaps. It shared "once shape" with writing-plans and "shape known" with brainstorming, in the first clause, which is the zone aebeebb identified as the one a matcher weighs hardest. tests/description-collision.sh scores the corpus 23 before and 13 after; those two pairs are the whole delta. So this commit buys a defensible description and a lower collision score, and buys no measured dispatch. Both statements are in the runlog beside it. The one place the old overlap showed up as behaviour is in the executing-plans arm, not this one: 2 of that skill's 9 baseline misses fired test-driven-development on a prompt with no test in it, and it is still 2/10 after. Narrowed, not removed. A caveat that limits this arm specifically. 3/10 at max-turns=12 is a stochastic rate, and n=10 cannot separate 0.3 from 0.5. It can separate "never fires" from "fires about half the time", which is what it was sized for. This arm establishes that test-driven-development is weak rather than dead, and nothing finer. The runlog was taken with all four rewritten descriptions installed together (digest 5cf618030b60 on every row), because the four are mutually exclusive of each other and an arm run against a half-rewritten tree measures something that never ships. Description is 169 chars against check 3's 200 cap. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ing in for 10/10 -> 10/10, N=10 per arm, no early stop, max-turns=3 both arms. No regression, and no improvement was available. The brief said swarm fires only when the user says "in parallel". The baseline refutes that: swarm fired 10/10 on a prompt containing none of the four strings its own description quoted -- not "in parallel", not "at once", not "all of these", not "try N ways" -- because the fixture describes six non-importing integrations needing six independent verdicts. The situation was carrying it the whole time. The literals were decoration on a description that already worked. So this is a simplification with a measured no-regression, not a fix. The description now states the cardinality-and-independence condition it was really matching on: three or more independent units, none needing another's result. That is also why it is the one skill here that does not sit on the planning-chain axis at all, and can co-fire with any stage of it. The literals are not lost. claude/hooks/inject-session-context.sh still carries them on its swarm routing line, and that file is owned elsewhere this week. The division of labour is the one the hook's own preamble states: the routing block spells out what descriptions do not reliably trigger. This change moves swarm off needing that crutch without removing it. Measuring an arm with no headroom is the point rather than a waste. Changing a description that scores 10/10 and not re-measuring it is how a regression ships as a cleanup, and this repo has shipped enough of those. The runlog beside this commit is what makes "no regression" a reading instead of an assumption. The runlog was taken with all four rewritten descriptions installed together (digest 5cf618030b60 on every row). All 10 samples terminated error_max_turns at max-turns=3, as they did at baseline; the Skill call precedes the cut, so the hits stand, and the arm is budget-matched to its baseline. Description is 186 chars against check 3's 200 cap, down from 194. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
9/10 -> 10/10, N=10 per arm, no early stop, max-turns=3 both arms. Two-sided Fisher p=1.0. The one-sample difference is noise at this n and should not be read as an improvement. The brief listed writing-plans among three skills that have never fired. That is true of the transcript corpus and false of this harness: it fired 9/10 on the unedited description, before anything in this branch touched it. There was no defect here to fix and no headroom to fix it in. What the rewrite buys is the same thing it buys for its two siblings. "Second, once the shape is agreed and before code" asserts a position in a chain the matcher cannot see. The replacement asserts three things about the prompt itself: requirements settled, nothing written down about how, more than one edit of work. It also drops the "once shape" overlap this description shared with test-driven-development in the first clause, which is 6 of the 10 points tests/description-collision.sh loses between 23 and 13. THE CAVEAT THAT LIMITS BOTH ARMS, and it limits them badly enough to state here rather than only in RESULTS.md. This fixture shares its hinges with the description it tests. It says "SPEC.md is agreed" against a description reading "once the shape is agreed", and "before anyone touches app.js" against "before code". That is not a quotation, but it is the same two hinges, and a fixture built from a description's own hinges will fire more often than a real request does. 9/10 here beside 0 across 2536 transcripts is most economically explained by real prompts not stating the situation this plainly -- a claim about how people phrase requests, not about how descriptions are worded. This arm cannot separate those two, and neither the before nor the after number should be quoted as evidence that the skill works in production. An earlier draft of RESULTS.md attributed a phrase to this description that does not appear in it. Review caught it. The overlap named above is the real one. The runlog was taken with all four rewritten descriptions installed together (digest 5cf618030b60 on every row). Description is 178 chars against check 3's 200 cap. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… earning it Four arms, N=10 each side, no early stop, every arm budget-matched to its own baseline. 40 calls after, $7.41; $16.19 and 100 calls across everything published here. executing-plans 1/10 -> 6/10 p=0.0573 moved, not significant test-driven-development 3/10 -> 3/10 p=1.0 unchanged writing-plans 9/10 -> 10/10 p=1.0 no regression, no headroom swarm 10/10 -> 10/10 p=1.0 no regression, no headroom The registered threshold was: fixed when baseline <=1/10 and after >=6/10. executing-plans came in at exactly 1/10 and 6/10, so it passed. It should not have been written that way. I derived "6/10" against an assumed 0/10 baseline, where it gives p=0.0108 and clears the Bonferroni 0.0125 four tests want. The measured baseline was 1/10 and I never re-derived the bound. Against 1/10, 6/10 is p=0.0573 and clears nothing. The check passed while the thing it stood for did not. That is docs/checks-that-inherit-their-answer.md, in my own pre-registration rather than someone else's, and the entry is written up as such rather than quietly repaired. Resolving 1/10 against 6/10 at that alpha needs roughly n=20-25 per arm, about $12 more. What the arms actually establish. Two of the brief's premises are false of this harness: writing-plans fires 9/10 unedited and swarm fires 10/10 on a prompt carrying none of the four trigger strings its own description quoted. The transcript-census zeros across 2536 transcripts are real, and they are about how real prompts are phrased rather than how descriptions are worded. This arm cannot separate those, and RESULTS.md says so instead of taking the credit. The writing-plans fixture shares its hinges with the description it tests -- "SPEC.md is agreed" against "once the shape is agreed", "before anyone touches app.js" against "before code". Neither its 9/10 nor its 10/10 should be quoted as evidence the skill works in production. An earlier draft of this file supported that same caveat with a phrase that does not appear in the description at all; review caught the fabrication and the real overlap replaced it. Two claims were also removed for outrunning their data. The baseline section said the starved arms "never reached a decision point"; the runlog carries fired and subtype and no turn trace, and the arms that fired were also 10/10 cut off, so being cut off separates nothing. It now says only that the zeros were taken at a budget every sample exhausted, which is reason to re-take them and not reason to publish them. Verified: .claude/verify.sh check 12 reads 30 test cases, check 31 green at 168 files, check 3 clean on all four rewrites (168-186 chars against a 200 cap). dispatch-static.sh 4/4, 30 cases, zero model calls. description-collision.sh self-test green both directions, corpus 23 -> 13, no hard-fail pairs. stage-skill-descriptions.sh restored ~/.claude to d4251fb4eda7 with nothing staged. The four remaining gate failures reproduce identically on pristine 1bfb133 and belong to the in-flight release. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Handoff report from the skill-autofire session (verbatim); the release owner picks up the inventory re-point and 1.63.0 bump as explicit merge tasks.
Per-skill fire rates, N=10 per arm, no early stop
100 published calls, $16.19, recorded by the harness. Model sonnet, no seed knob exists. Instrument frozen at 901bb3f and stamped into every row; installed-description digest d4251fb4eda7 (baseline) / 5cf618030b60 (after), one value per arm.
Three of the brief's premises are falsified. swarm fires 10/10 on a prompt containing none of the four trigger strings its own description quoted, so "fires only when the user says 'in parallel'" does not hold in this harness. writing-plans fires 9/10 unedited. executing-plans fired 1/10 on a prompt that is nearly its own description restated, with 2/10 misrouting to test-driven-development on a prompt containing no test, so wording was never the binding constraint. Sibling competition was.
The pre-registered threshold passed without earning it. "after >=6/10" was derived against an assumed 0/10 baseline (p=0.0108, clears Bonferroni alpha=0.0125). Measured baseline was 1/10, making 6/10 p=0.0573, which clears neither 0.0125 nor 0.05. Written up in RESULTS.md as an instance of docs/checks-that-inherit-their-answer.md in the session's own work. Separating those rates needs n~20-25 per arm, roughly $12 more.
Collision check
tests/description-collision.sh, zero model calls. Corpus score 23 -> 13, zero hard-fail pairs either side. The 10 points are exactly the two overlaps the ordinal framing created: test-driven-development/writing-plans on "once shape" (6) and brainstorming/test-driven-development on "shape known" (4). The 80-sample "collisions suppress BOTH" hazard the brief cited is retracted (CHANGELOG 1.47.0); its pre-registered replacement at n=25 found the hypothesis unsupported. The script header says so and the string "80 samples" appears nowhere.
Situation -> skill after the change
Gate status
tests/dispatch-static.sh green (30 cases declared and present). .claude/verify.sh 63 checks, 4 failures, all one cause: four payload files changed without a version bump or inventory re-point. origin/main is green, so these are the branch's.
MERGE TASKS (release owner):
Hook diff for its owner
claude/hooks/inject-session-context.sh was read, never edited. Two lines to match the shipped descriptions:
Line 208 (swarm) and the writing-plans line already agree. Flag alongside it that line 221 and the shipped executing-plans-checkpoints fixture have disagreed since before this branch. claude/CLAUDE.md has no routing table (58 lines); the table lives in this barred hook, which is why the branch edits no hook.
🤖 Generated with Claude Code