feat(plugin): optimize-skill's execution track, measured end to end — plus the skill_triggered and --split fixes it surfaced by uipreliga · Pull Request #109 · UiPath/coder_eval

uipreliga · 2026-08-12T21:36:21Z

What

Adds Dataset.split_field + a --split row filter, ships a seventh plugin skill /coder-eval:optimize-skill, and dogfoods the whole loop in a new tutorial — which is where most of the value in this PR came from.

Implements c/2026-08-12-skill-optimize-p0-p1.md, phase by phase.

The feature

--split (P0). Label dataset rows train / test and select one at run time. The filter runs before either sampler — sampling first would leave an unpredictable (possibly zero) number of rows per split, destroying the comparison the split exists to protect. Unlabelled tasks pass through untouched, so --split is safe in a multi-task run; a labelled task with no matching row raises.

/coder-eval:optimize-skill (P1). A/B tests candidate edits to a skill as experiment variants, promoting only what beats run-to-run noise and then survives a held-out split. Two tracks:

Activation — the frontmatter description, measured against an activation suite. Does it fire when it should?
Execution — the skill body, measured against an outcome suite with real success criteria. Having fired, does it do the job?

Explicit-invocation only — it spends real money.

Why the two gates differ (worth a reviewer's eye)

The tracks share splits, snapshots, reachability, replicate discipline and the ledger. What differs is the instrument, and it is load-bearing:

Activation compares F1, which a pooled suite.json cannot report per replicate — hence three separate invocations.
Execution compares per-row weighted_score, which paired_comparison already computes correctly over replicates it averages per row before pairing — hence --repeats 3 on exactly two variants.

So the paired block that is mere corroboration on the activation track is the primary instrument on the execution track: tested code instead of arithmetic by hand. Unifying the gates would swap in an instrument that cannot see the metric, so both halves are pinned by sensors.

Two smaller inversions, also sensored: activation rows must never name the skill (that tests obedience, not activation) while execution rows must invoke it by slash command — /plugin:skill at the top of initial_prompt — to hold activation constant; and disable-model-invocation: true now routes to the execution track rather than hard-stopping, so init and ci are optimizable after all.

The slash form is not a stylistic preference. A disable-model-invocation skill is not offered to the model at all, so asking in prose returns "no such skill is available" and the row measures nothing; the slash form loads it, emits a real Skill tool call, and is detected by skill_triggered. Both halves verified live (see below).

Validation

Beyond the test suite, the skill itself was live-tested in three layers, because most of its failure modes are silent rather than loud:

Static — every ${CLAUDE_PLUGIN_ROOT} reference, slash command and CLI flag it emits resolves; the experiment YAML in its body validates against ExperimentDefinition.
Cold agents — two agents with no context executed the skill end to end, one per track, on a zero-run budget, building real snapshots and experiment YAML for inspection.
Live probe — one real agent run to settle whether skill_triggered detects a slash-command invocation (it does; scored 1.0).

That surfaced 15 problems, fixed in 9ac72f6. The two worth a reviewer's attention were silent confounds in the snapshot step: the diagram showed only skills/ while the prose said "everything that source contained" (a diagram-faithful snapshot mounts skills whose bundled reference/ files are missing — invisible on the activation track), and omitting .claude-plugin/plugin.json makes the namespace default to the arm's directory name, so arms would differ in the listing name as well as the text under test. Neither would have errored; both would have produced confident, wrong numbers.

It also produced one piece of guidance now in the skill: before proposing an A/B, check whether a lint rule already answers the question for zero runs. The complaint that prompted the ci test — "emits workflows that miss a step" — overlaps CE026 almost exactly, and ~300 agent runs is the wrong way to learn something a static rule catches permanently.

Three bugs caught before shipping

Two were errors in the plan, found by review; the third was already shipped on main.

The sibling-regression gate read the wrong metric. Annexation makes the sibling's criterion expected=yes, observed=no — a false negative. Since precision = tp/(tp+fp), it stays pinned at 1.0 however many requests are stolen. The gate would have been gating on a constant. Now reads recall.yes.
A one-skill candidate snapshot empties the sandbox. A variant's plugins block replaces the task's, so the snapshot is the arm's only skill source. Snapshot one skill and every sibling criterion silently observes no — the sibling gate "passes" by measuring nothing. Snapshots now carry the whole skills tree.
The shipped reachability guidance was wrong (pre-existing). A local plugin path must be a plugin root holding skills/; the template told users to point at the bare skills directory, which loads nothing. Proven by probing three layouts against the real CLI. Every suite check-skill generated would have reported recall 0.0 — exactly what the template's own comment calls "reads exactly like a broken skill". Corrected in 7 surfaces, including ci, which was writing the broken path into users' CI workflows.

The tutorial (docs/tutorials/08)

Runs the loop for real against this repo's own skills, on Sonnet, and reports what happened rather than a tidied version:

lint-tasks measured at ceiling (F1 1.000 both splits), so the loop declined to spend ~224 runs chasing an unreachable number. That also closed the plan's open residual: the earlier 66-character description trim is now measured safe.
A task misfire that reproduced on both splits turned out to be 2-in-3 variance under replicates. Two agreeing runs were not evidence.
The real headroom was a sibling: analyze missed "what regressed" deterministically. Full three-stage A/B → a-regression promoted, gated on the train split (1.000 vs 0.667, non-overlapping, three invocations) and confirmed on the test split (1.000 vs 0.909).
Two comparisons were invalidated mid-run and thrown out — one candidate ranked on an eroded denominator, and a confident p = 0.038 that was a billing limit eating one arm harder than the other. completion_rate is what caught both.
Bare-name collision hazard: skill_triggered strips plugin: prefixes, and Claude Code ships its own unscoped init. A skill_name: "init" criterion would silently score a different skill. Documented in check-skill.

Notes for review

Listing budget held flat at 1,574/1,600 with a seventh skill added — five descriptions trimmed rather than raising the ceiling, since the budget is shared with every skill the user has installed.
analyze's description change is the one behavioural edit to an existing skill, and it is measured.
3 new lint sensors (mutation-tested); 1 harness gap deferred to .claude/harness-candidates.md — an all-skipped run exits 0, now reachable by a one-character --split typo. Fixing it changes exit semantics for every skipped-task path, so it wants a decision rather than a drive-by.

make verify green: 4071 passed, coverage 91.57%.

🤖 Generated with Claude Code

Late changes (after the first review pass)

Two follow-ups from using the thing, both pushed since the PR opened:

Split values renamed tune/holdout → train/test. Docs and content only — split values are open strings, so no schema change and nothing to migrate; split_field still defaults to "split". 215 replacements, followed by a pass disambiguating bare "test" back to "test split" wherever it named the data half rather than a check, since in a testing framework "without a test" parses exactly wrong. (Pedantic but recorded: nothing is trained here — you hand-select a candidate, which is a form of fitting, so "train" is defensible in the ML sense and far more recognisable than "tune".)

The skill now labels the splits itself rather than asking the user to. It previously offered to add labels, leaving a mechanical JSONL edit and a balance judgement to the reader. It now writes them and shows the counts for objection, under two rules that are easy to get wrong by eye and expensive when you do: stratify within each polarity so both halves carry positives and distractors, and assign deterministically so the split never re-rolls between rounds — a reshuffled split is not a test split, because rows already tuned against leak into it.

Not done, deliberately: a hash-based auto-split (--split train --split-ratio 0.6, membership derived from row_id, no labels in the file) would remove the last of the friction, but it conflicts with a load-bearing behaviour — unlabelled tasks currently pass through unfiltered, which is what makes --split safe in a multi-task run. Auto-splitting them would silently change that, so it needs an opt-in flag and its own tests. Left as a follow-up rather than smuggled in here.

Late changes (second pass): the execution track measured for real — and a criterion bug it uncovered

The execution track had never been run end to end. Doing that turned up a bug in skill_triggered that invalidates two claims made earlier in this description, so those are corrected below rather than quietly edited.

⚠️ Behaviour change reviewers should look at

skill_triggered no longer counts an errored Skill call as engagement (src/coder_eval/criteria/skill_triggered.py).

Any suite where a Skill call failed previously scored yes and will now score no. That is the correction — a refused call means the skill's body never loaded — but an existing green activation suite that was passing on refused calls will go red, and correctly so. The file-read signal is deliberately not gated the same way: a refused call loaded nothing, whereas a path reference means the SKILL.md was actually opened. Both cases are pinned by tests.

What the run found

An outcome suite was built to A/B ci's body: 4 arms differing only in that body, 24 rows. All four tied exactly on every criterion — which is a bug report, not a result.

ci sets disable-model-invocation: true, and the Skill tool refuses such a call outright:

<tool_use_error>Skill coder-eval:ci cannot be used with Skill tool
due to disable-model-invocation</tool_use_error>

24 of 24 calls failed this way, result_status: "error" on every one, and no row read the SKILL.md off disk either. The body never entered context; the agent answered from its own knowledge of GitHub Actions, plausibly enough that nothing downstream looked wrong. The tell was in the output the whole time:

body NOT loaded:  uses: anthropics/coder-eval-action@v1   <- does not exist
body loaded:    - uses: UiPath/coder_eval@v0             <- what the body specifies

Re-scoring those same artifacts with the fixed criterion gives 0/6 engagement per arm where the run reported 6/6.

Corrections to claims made above

"asking in prose returns no such skill is available" — wrong. Measured across the same six rows, by how often the model attempted the call: slash form alone 3/6, prose instruction alone 5/6, slash + explicit imperative 6/6. Prose works better than the slash form. Nothing in coder-eval expands a slash command; it arrives as plain text the model may ignore.
"Live probe — … whether skill_triggered detects a slash-command invocation (it does; scored 1.0)" — that 1.0 was the false positive. The call had errored.
"init and ci are optimizable after all" — true, but only with the snapshot fix: delete the disable-model-invocation: line in every arm's snapshot, which is what a real user's slash command effectively does. Verified — the same rows then score 1.000 with the action reference correct.

Also in this pass

Verdict on ci: no promotion, a verified ceiling. With the body loaded, the train split scores 1.000 on every row and every criterion (18/18 assertions, completion_rate 1.0). The three candidates were solving a problem that did not exist. Stages B and C were correctly never reached. ~55 Sonnet runs, ~$20.
The outcome-suite template (reference/templates/outcome.yaml) + a checked-in worked example (tasks/skills/ci-outcome.yaml), runnable in two documented lines.
Engagement is now actually gated (recall.yes: 1.0) in both — they described it as a gate and carried no threshold, which is how a zero-engagement run exited 0.
CE035 (partly-labelled split datasets) and CE036 (a row's prompt must not contain the string its criteria grade — four scenarios here had that leak).
Tutorial 09, written from the real numbers, plus tutorial 08 made reproducible (snapshot layout, per-stage experiment YAML, the -e command lines) and restructured into two parts.
docs/TASK_DEFINITION_GUIDE.md updated — it documented the pre-fix detection rule.

make verify green: 4074 passed, coverage 91.57%.

Third pass — the review's P0/P1/P2 fixes, then a tutorial audit

Implements c/2026-08-13-optimize-skill-review-v2-fixes.md, one commit per phase.

Two behaviour changes to look at first

skill_triggered now requires the signal to have actually delivered. The early-stop watcher evaluates on the tool call, where result_status is still None, so an in-flight or crash-force-closed Skill call live-passed while the frozen check scored it no — live and final verdicts disagreed on the one criterion that decides whether a paid round measured the skill at all. Engagement is now an allowlist: a Skill call counts only on "success" (for that tool the body is the tool result), and a Read/Glob/Grep that errored or has not resolved does not count — the path is in its parameters, but nothing loaded. Bash stays ungated, because cat … | grep exits non-zero after genuinely reading the file. Historical activation P/R/F1 is not directly comparable if a run contained those shapes; two golden-corpus entries were appended to force that acknowledgement rather than editing existing ones.

This reverses a documented design decision, so it is worth a reviewer's eye. Seven tests in tests/test_early_stop.py pinned the opposite semantics as deliberate; each is inverted, not deleted, with the reasoning in its comment, plus a Bash twin for the path that still decides on the call. The cost is stated in the test that pays it: a Skill call whose result never arrives no longer stops the run, so that run continues to its turn cap. The old behaviour stopped it and scored yes — crediting a skill whose body never reached the agent.

A mistyped --split now exits 2 instead of 0. It used to raise, get caught like any load failure, and land in skipped_tasks: one yellow line, zero evaluations, exit 0 — a CI gate reporting success for a one-character typo. SplitSelectorError separates a malformed file (still demoted, so one bad task cannot abort a suite) from a malformed invocation (re-raised — the same selector applies to every task in the run). No new CLI code was needed: the existing except ValueError -> typer.BadParameter supplies the exit.

The rest

coder-eval plan expands datasets and takes --split — prints total/selected row counts and catches a bad ${row.*} at plan time rather than per-row after the sandbox is built. The pre-spend check now costs nothing. It is also louder: a task with a missing dataset JSONL now fails a plan it previously passed.
Partial split labelling stops being silent — a WARNING with the drop count from expand_dataset, a yellow ⚠ from plan, and a check in lint-tasks for adopters CE035 cannot reach. All three call row_split_label, so the rule cannot drift even though the messages differ.
Row-id validation covers the whole dataset, before --split and sampling narrow it. A malformed row in the test half used to validate under every --split train run and surface at promotion time — the most expensive moment to learn it.
The bundled activation.yaml caps and isolates (run_limits, setting_sources: []), mirroring the checked-in worked example key for key. It preached both and shipped neither, so a user who copied it got the opposite of the advice they had just read.
CE036 gets real fixtures and a skill_name exemption. Its only test was a repo scan that passed whether or not the rule could still detect anything; detection now lives in a helper the scan and six fixtures share. CE036_LOCATOR_FIELDS is the source and CLAUDE.md the derived surface, with a both-directions parity sensor — that list had already drifted once.
reference/optimize-method.md — the track-invariant method extracted from an 859-line SKILL.md as a byte-identical move (verified against HEAD), the three named seams fixed, and the prose sensor split into procedure-vs-method sets that state which file each token belongs to and why.
Method caveats the review asked for: promotion verdicts apply to the flag-removed configuration, and Stage B now requires a predeclared primary criterion plus guardrails — scanning every per-criterion aggregate post hoc is uncorrected multiple testing in the rejection direction, so noisy criteria veto real wins.
ci-outcome's both-triggers row asked for two triggers and graded one, so a schedule-only workflow scored 1.000 — and it graded the same string as its sibling, discriminating nothing. Fixed without touching any train row, so the file's own measured "the train split scores 1.000 on every row" claim stays true by construction.

Tutorial audit

All nine checked mechanically — every path, relative link, cross-doc anchor, CLI flag against live --help, and every complete YAML snippet against the real models. All clean. By content, three defects, every one the same shape: a hand-maintained number describing a set the code derives.

"14 criterion types" (×3) against a registry of 15. CLAUDE.md's heading was right, which is why nothing noticed — it was the only guarded surface.
"six commands" in tutorial 07, omitting optimize-skill entirely. A skill-count sensor already existed; this file was not one of the four surfaces it read, and the tutorial's phrasing ("commands") was a fourth the matcher did not know.
Tutorial 09 quoting ci-outcome.yaml excerpts this PR had just changed — a reader copying the shown row would have built a suite that raises at expansion.

Each is now derived rather than asserted, and tutorial 07 joins SKILL_DOC_SURFACES.

Tutorial 08 Part 1 was re-measured rather than annotated — 62 Sonnet runs, three replicates of the train split plus one of the test split. lint-tasks still holds at F1 1.000 on a suite three rows larger. analyze reads 1.000 where Part 1 recorded 0.000, because the description this PR promotes is committed (4c7481c): the baseline is unreproducible precisely because the loop worked. And hard-3 is still unstable at exactly the 2-in-3 Step 6 measured — months later, on a different revision of the suite, which is the page's own "two agreeing runs are not evidence" lesson arriving unprompted. The page now states what a reader gets today and why it differs, instead of quietly refreshing the numbers.

Notes for review

6 new sensors + 6 CE036 fixtures, all mutation-verified — each was confirmed to fail when the property it guards is reverted.
1 harness gap deferred: "the ToolStart seam decides" is now a per-criterion property rather than a global invariant. Not mechanically detectable — it is about what a live_verdict reads, which no AST rule can infer.
No new Pydantic model, criterion, agent, config key or MergeField; one new exception type (SplitSelectorError), one module logger, one CLI option on an existing command.

make verify green: 4115 passed, 390 lint checks, coverage 91.63%.

github-actions · 2026-08-12T21:37:14Z

Claude finished @uipreliga's task in 1m 29s —— View job

Code Review in Progress

I'm reviewing PR #109 following the guidelines in .github/code_review.md.

Todo List

Read .github/code_review.md for review standards
Read CLAUDE.md for project conventions
Run git diff origin/main...HEAD to see full changes
Review changed files with full context
Perform cross-file consistency checks
Analyze what's missing
Provide comprehensive review feedback

Starting review now...

Label dataset rows with a split (tune / holdout / …) and select one at run time with `coder-eval run --split <name>`. The filter runs BEFORE either sampler: sampling first would leave an unpredictable (possibly zero) number of rows per split, destroying the comparison the split exists to protect. - `Dataset.split_field` (default "split") mirrors `stratify_field`'s shape. - The filter is inlined in `expand_dataset` rather than extracted: one call site, a one-line comprehension, and a helper would re-declare the missing-field convention `_stratified_sample` already owns. - A row is unlabelled when the field is absent, null, or "". A task whose rows are all unlabelled passes through unfiltered — `--split` is global to the invocation, so an unlabelled suite beside a labelled one must not fail. A labelled task with no matching row raises, naming the splits that exist; `resolve_all_tasks` records that as a skipped task, so a mistyped selector is a zero-task run that still exits 0. Documented rather than papered over. `--split` unset leaves expansion byte-for-byte unchanged (pinned by test). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…activation template `/coder-eval:optimize-skill` turns an activation suite's confusion matrix into candidate description rewrites, A/B tests them as experiment variants, and promotes only what beats run-to-run noise and then survives a held-out split. Explicit-invocation only: it spends real money across three stages. Two mismeasurements were designed out rather than discovered later: - The sibling-regression gate reads the sibling's `recall.yes`, not its precision. Annexation makes the sibling's criterion expected=yes/observed=no — a false negative — and `precision = tp/(tp+fp)` stays pinned at 1.0 when the sibling never misfires, so a precision gate would gate on a constant. - Each candidate snapshots the WHOLE skills directory, siblings copied unchanged. A variant's `plugins` block replaces the task's, so the snapshot is the arm's only skill source: snapshot one skill and every sibling criterion silently observes `no` in every arm, and the description is tested against a listing it will never face. Supporting changes: the activation template gains `split_field` and per-row tune/holdout labels (both splits carry positives and distractors; no `stop_early:` — that would degrade sibling measurement); run-layout.md documents the suite-rollup path, aggregate shape, `failed_samples` as the only row-identity field, and replicate pooling — the contract that keeps someone from "simplifying" Stage B's three invocations into `--repeats 3`, which pools into one suite.json and leaves the gate nothing to read. Five shipped descriptions trimmed so seven skills fit the listing budget (1,524/1,600) without raising the ceiling: the budget is shared with every skill the user has installed, so growing our own footprint evicts theirs. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…lity guidance it disproved Runs the optimization loop end-to-end against this repo's own `lint-tasks` skill and reports what actually happened, including the parts that did not go to plan. The shipped reachability guidance was wrong. Both the activation template and `check-skill` said a local plugin `path` is "the directory containing the skill's directory" — `.claude/skills`. Probing three layouts against the real CLI shows a plugin path must be a PLUGIN ROOT holding a `skills/` subdirectory (`<path>/skills/<name>/SKILL.md`; a manifest is optional, the namespace then defaults to the directory name). A bare directory of skill directories loads nothing at all. So the correct root for `.claude/skills/my-skill/SKILL.md` is `.claude`. Every suite `check-skill` generated would have reported recall 0.0 — precisely the symptom the template's own comment warns "reads exactly like a broken skill". Corrected in the template, `check-skill` and `optimize-skill`, and pinned by a new sensor. The run itself: `lint-tasks` scores F1 1.000 on both splits, so the loop stops at the diagnosis step rather than spending ~130 further runs chasing a number the gate makes unreachable. That result closes an open question from the previous commit — the 66-character description trim is now measured safe rather than assumed. The headroom turned out to be in the sibling matrix: `task` annexes setup requests on both splits, and `analyze` never fires on "what regressed". The tutorial ends on that, because a walkthrough that only works when the result is positive teaches the wrong lesson about A/B testing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…ing --split Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The reachability correction reached only half its surfaces. `ci`'s workflow snippet still exported `SKILL_SOURCE_PATH=.../.claude/skills` — a shipped skill writing the broken path straight into users' CI, where the failure mode is a permanent red that looks exactly like the drift the schedule exists to detect. Same stale form in docs/PLUGIN.md and tutorial 07, plus a self-contradicting sentence in optimize-skill's own Step 8. All corrected, and the sensor now covers seven surfaces and asserts the specific layout: its previous `"skills/" in text` check passed on the pre-fix text via `.claude/skills/my-skill/SKILL.md`, so it guarded nothing. `check-skill` was never told about splits, so following it to the documented row count produced a PARTLY labelled suite — the one state where `--split` silently drops rows and shrinks the denominator the thresholds gate on. It now labels every row it writes, and optimize-skill gained the missing branch. Two rows of the new repo suite asserted `expected_skill: "init"`, a skill that sets `disable-model-invocation: true` and so can never be engaged: an unsatisfiable criterion that made every run of that suite exit 1. Relabelled to `""`, which asks the question that has an answer. Also: duplicate row ids are now rejected across the whole dataset before `--split` narrows it, so a duplicate in an unselected split can no longer validate under every split and surface only on a full run; run-layout documents `rows_total` / `rows_excluded` / `completion_rate`; and the claim that an excluded row shrinks a suite "silently" was wrong — the rollup reports it and it is gateable. Re-ran both splits after the label fix. `lint-tasks` holds F1 1.000. The `task`-annexes-setup finding, which had reproduced on both splits, did NOT survive the third run on byte-identical prompts — so the tutorial now teaches that instead, which is a better lesson about why the gate demands replicates. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

… two open findings Both follow-ups from the previous round are now settled by measurement rather than left as open questions. **The `task` misfire was intermittent, not phantom.** Three tune invocations show `hard-3` engaging `task` in two of three runs on byte-identical prompts. The earlier pair of agreeing runs was luck. This is the clearest argument yet for the replicate gate, and the tutorial now teaches it with the real trace. **The `analyze` gap was real, stable, and fixable.** Recall 0.500 in all three runs with precision 1.000 throughout — one row ("what regressed") missed every time, no over-claiming. The suite gained analyze rows on both splits so the gap could be measured at all, then the full loop ran: - Stage A (68 runs): three candidates, all above the incumbent. `b-results` looked competitive but lost a row to an error (completion 0.941), so its recall came from 3 rows not 4 — ranked out rather than compared. - Stage B (153 runs, three separate invocations): `a-regression` at F1 1.000 in every run against a rock-steady incumbent 0.667. Non-overlapping, no sibling regression, precision never off 1.000. One incumbent invocation dropped a row and was excluded rather than averaged in. - Stage C: not obtained. The first holdout was uninformative — a flat tie, because every regression-phrased row had been put in tune. Fresh holdout rows were authored and the re-run then hit the org's monthly spend limit, which removed 11 rows from one arm and 6 from the other and produced a confident p = 0.038 pointing the WRONG way. A p-value over an asymmetrically eroded sample is not evidence. `a-regression` is therefore promoted **gated on tune, unconfirmed on holdout** — stated plainly in the tutorial rather than dressed up. Listing budget holds at 1,574/1,600. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…d on holdout Re-ran the holdout confirmation now that budget is available. Erosion was one row against the candidate and none against the incumbent — near-symmetric, and pointing the conservative way — so unlike the spend-limited attempt this run is interpretable. incumbent analyze recall 0.833 precision 1.000 F1 0.909 a-regression analyze recall 1.000 precision 1.000 F1 1.000 The direction reproduces on rows the candidate was never tuned against, which is what Stage C is required to show. One row separates the arms, and it is one of the fresh rows authored at promotion time: an-6 "Which of my tasks got worse after I switched the model?" incumbent 1 of 3 a-regression 3 of 3 No sibling regressed in either arm; the incumbent shows the intermittent `task` misfire once more, consistent with the 2-in-3 rate measured earlier. The paired comparison reads exactly zero (p = 1.000). That is the documented limit of the block rather than a contradiction: it pairs per-row weighted_score across all three criteria, so a gain confined to one criterion on one row out of eleven is diluted below what 11 pairs resolve. F1 is the promotion metric; the tutorial now reports both and says which one the decision rested on. `a-regression` therefore ships gated AND confirmed, and the earlier "unconfirmed" framing is gone. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The Stage C per-row data showed `init` being engaged on the setup rows in both arms — which contradicted a comment claiming coder-eval's `init` "can never be engaged" because it sets `disable-model-invocation: true`. Both halves were wrong in an instructive way. Claude Code ships its own unscoped `init` skill, and `skill_triggered` matches on the BARE name (it strips any `plugin:` prefix), so the two are indistinguishable to the criterion. A `skill_name: "init"` criterion would have silently scored a different skill's activation as though it were the plugin's. That is a second, independent reason not to have written one — beyond the unsatisfiability already fixed. Documented where suite authors will hit it: `check-skill` (which already covers bare-name matching), the tutorial's caveat section, and the suite's own comment. A collision does not error; it measures the wrong skill. Also marks the plan file complete and ticks its 52 acceptance criteria, with a header recording the four deviations the implementation required. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

… body, not just the description A skill can fail two independent ways: it never gets reached, or it gets reached and gives bad instructions. optimize-skill only measured the first. Both tracks now live in the one skill, because they share almost everything — splits, snapshots, reachability, replicate discipline, the ledger, the stop rule. What differs is the instrument, and that difference is load-bearing: `skill_triggered` is a binary one-turn probe that says nothing about the quality of the work that follows, so an activation suite cannot grade a body. The execution track uses an ordinary outcome suite with real success criteria. Three consequences worth calling out, each pinned by a sensor: - **The two gates differ on purpose.** Activation compares F1, which a pooled suite.json cannot report per replicate — hence three separate invocations. Execution compares per-row weighted_score, which `paired_comparison` already computes correctly over replicates it averages per row before pairing. So the paired block that is mere corroboration on the activation track is the PRIMARY instrument on the execution track: tested code instead of hand arithmetic. Unifying the gates would swap in an instrument that cannot see the metric. - **The prompt rule inverts.** Activation rows must never name the skill (that tests obedience, not activation). Execution rows must, to hold activation constant so the body is the only variable. - **`disable-model-invocation: true` now routes instead of stopping.** Such a skill's description never enters the activation decision, but its body still determines whether it does its job — so `init` and `ci` are optimizable after all, just not on the track that was previously the only one. Also: one variable per round (never both tracks at once), regression coverage for rows that already pass (a body edit breaks things silently, where the confusion matrix shows activation regressions for free), and trajectory-based diagnosis with the five failure modes a body edit actually addresses. Tutorial 08 now says which track it walks. Listing budget holds at 1,577/1,600. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Two agents executed the skill with no prior context and a zero-run budget, one per track, building real artifacts. Between them they found 15 problems. The substantive ones: **The execution track never said how to engage the skill it routes to.** Step 2 sends every `disable-model-invocation` skill there, and Step 4 then said "name the skill in the prompt" — which does not work: such a skill is not offered to the model at all, so prose gets "no such skill is available" and the row measures nothing. Verified live, and verified the fix: a slash command in `initial_prompt` DOES load it, emits a real `Skill` tool call, and is detected by `skill_triggered` (probe scored 1.0). The routing was right; the mechanism was missing. Now specified, with a worked `initial_prompt`. **Two silent confounds in the snapshot step.** The diagram showed only `skills/` while the prose said "everything that source contained" — a diagram-faithful snapshot mounts skills whose `${CLAUDE_PLUGIN_ROOT}/reference/` files are gone, invisibly on the activation track. And dropping `.claude-plugin/plugin.json` makes the namespace default to the arm's directory name, so arms differ in the name shown in the listing as well as the text under test — on the one track where activation is a competition between listings. **A version string is not a capability check.** Two binaries reported the same 0.9.6 and differed in whether `--split` exists at all; the pinned-version rule said "carry on" while every run would fail at load. Step 1 now requires `coder-eval plan <suite>` to exit 0 before spending. **The skill writes descriptions and never mentioned the length ceiling** every natural fix pushes against. Now budgeted before candidates are written. Also: the baseline is now a line item in the cost table with its non-redundancy explained (it validates the snapshot wiring against the task's own source); `completion_rate` is documented as per-criterion-aggregate rather than top-level; the execution track's no-suite stop matches the activation track's force and carries the split-label and slash-invocation requirements into the hand-off; suite sizing resolves to a number rather than contradicting itself; Step 3 gains a "check whether a lint rule answers this for zero runs" pre-check; and repo-layout.md now prunes `tmp/` and any already-found run store, which had been returning hundreds of agent-produced SKILL.md files. Fixes two bugs this branch introduced: an off-by-one step reference from the renumbering, and tutorial 08 claiming the skill hard-stops where it now routes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…vention Tutorial 08 shipped without the YAML front-matter every other tutorial carries, and with an H1 that did not follow `# Tutorial NN — Title Case`. The front-matter is not decorative: mkdocs renders `description:` into the page's `<meta name="description">`, so the page had no search or social-preview summary at all. - adds the folded `description: >-` block, matching 01–07's shape and voice - retitles to `# Tutorial 08 — Optimizing a Skill Description` - title-cases the nav and tutorials/README entries, which were the only lowercase ones in either list - regenerates docs/llms.txt via `make docs-indexes` (verified idempotent) Audited the rest of the tree while here: 24 of 24 docs pages now carry a front-matter description; 08 was the only one missing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…, and label them for the user Terminology: `train`/`test` is the vocabulary everyone already has, and split values are open strings, so this is docs and content only — no schema change, no migration. `Dataset.split_field` still defaults to `"split"`; only the values this project's own suites and guidance use have changed. 215 replacements across 19 files, then a pass to disambiguate bare "test" back to "test split" wherever it named the data half rather than a check — in a testing framework "without a test" reads as "without a unit test", which is exactly the wrong parse. One honest imprecision worth noting: nothing is *trained* here. You hand-select a description from candidates, which is a form of fitting, so "train" is defensible in the ML sense of "the data you made your choices on" — and it is far more recognisable than "tune". Keeping the familiar word. Also removes the labelling friction that prompted this. Step 5 previously told the skill to *offer* to add labels, which left the user with a mechanical JSONL edit and the balance to get right by eye. It now does the labelling itself and shows the resulting counts for objection, under two rules that are easy to get wrong manually: stratify within each polarity so both halves carry positives and distractors, and assign deterministically so the split never re-rolls between rounds — a reshuffled split is not a test split, because rows already tuned against leak into it. Verified end to end: `--split train` -> 17 rows, `--split test` -> 11, unsplit -> 28. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

`!!! note` / `!!! danger` are mkdocs-material syntax. GitHub does not understand them: it renders the marker as literal text and turns the indented body into an accidental code block, so both callouts looked broken when read from the repo or a PR diff. Tutorials 01-07 use plain `>` blockquotes, which render correctly in both places — I had introduced the admonitions without checking the convention. Converted both callouts, and added a lint guard so the next tutorial cannot reintroduce them (mutation-tested: it fires on an injected `!!! note`). Scoped to tutorials deliberately. The one admonition under docs/ is in DATASETS.md, is site-first reference material, and predates this branch. Also drops `<sub>` tags from the skill's cost formula — a SKILL.md is read as plain text by a model, so HTML there is noise rather than markup. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…table The execution track's instructions produced silently-wrong measurements. Every fix here is prose; no evaluation behaviour changes. - The outcome suite must be ONE dataset-backed task, one row per scenario. suite.json is written only for tasks the dataset expander touched (rollups group on suite_id), and --split filters dataset ROWS — so a directory of separate task files gives Stage A no rollup to rank and makes Stage C's `--split test` silently re-run the train rows. - Two consequences that decide how rows are written: criteria are copied to every row with ${row.*} substituted into every string leaf (so per-scenario assertions are parameterized, never written per scenario), and substitution never reaches sandbox.template_sources — every row shares ONE fixture, so variation lives in the prompt and the fixture must clear the skill's own hard stops or every arm ties at zero. - Step 4's worked example described a repo with no .github/, which is exactly the hard stop `ci` refuses on; it now shows a repo that clears the precondition and names its output path. - There is no --variant flag, so the arm set changes by authoring a file per stage: round<N>-triage/gate/confirm.yaml. Re-passing the triage file at Stage B/C costs (N+1)/2x the budgeted runs and renders no Paired Comparison block at all. - The paired mean diff subtracts in variant declaration order, so with incumbent declared first a candidate win reads NEGATIVE. Stated in both Stage B and Stage C. - Pin the tool policy under `defaults: agent:` and size the allowlist to the union of every arm's tools, or a candidate whose hypothesis is "use a different tool" is scored on the prohibition. - Name the per-row max_usd brake, and say plainly that an activation suite's tight caps must not be carried over — a truncated outcome row scores as a body failure that never happened. - Cost table: M_tune/M_holdout -> M_train/M_test, matching the split rename. The deletion sensor gains one entry per new instruction, each with its why, plus a counted assertion that the sign rule survives in BOTH stages and a guard that the stale cost-table symbols stay gone. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

… track optimize-skill's execution track had no artifact to hand over: Step 4 pointed users at /coder-eval:task with a list of requirements attached, which came back half-applied. It now hands over a template, the way the activation track has always handed over activation.yaml. outcome.yaml is the execution twin of activation.yaml and carries the two constraints that are expensive to discover by running: - ONE fixture serves every row (${row.*} never reaches sandbox:), so scenario variation lives in the prompt and the fixture must clear whatever precondition the skill checks before it will act — else every arm ties at the floor and reads as three bad candidates. - Criteria are copied to every row, so per-scenario assertions are parameterized by row fields (path: "${row.expected_path}"). Its run_limits are deliberately unlike an activation suite's: an outcome row is a full task run, so max_turns is 20 rather than 2 and max_usd is the per-row brake. The run_command slot ships commented out — file_check already asserts what it would, and a row-supplied shell command would drag quoting and sandbox-tool-availability into every copy. Tests are loader-backed, mirroring the activation template's. The structural contract lives in ONE shared helper because Phase 4's checked-in sample asserts the same four properties; the threshold vocabulary is the union of a real aggregate() call and a real _attach_row_accounting() call, since completion_rate comes from the latter and a check against aggregate() alone would fail on the very template this ships. Also closes a harness gap found reviewing 1/8: nothing verified that a ${CLAUDE_PLUGIN_ROOT}/... pointer in a skill resolves to a file that exists, so the pointer at this template shipped one commit before the template did, past 344 green lint tests. Now 28 such references are checked. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…issed them docs/PLUGIN.md said "All six skills read it" while seven ship, and both READMEs named three of the four skills that shell out to the CLI. The count sensor was supposed to catch the first and did not: it substring-matched raw file text, and the offending line reads "All six\n skills read it", so the newline sat exactly between the two words the check needed adjacent. 91 lint tests passed over it. - docs/PLUGIN.md: six -> seven; both READMEs name all four members of SKILLS_REQUIRING_THE_CLI in the install paragraph; both `templates/` blurbs now name the outcome suite alongside the activation suite. - Extract `_normalized()` and read every prose surface through it. The idiom was already hand-copied at three sites, all now converted. - Extract the count matcher into `_wrong_skill_count_offenders` so its wrapped-phrase self-test can run the REAL matcher against a built file. Asserting only that `_normalized()` collapses whitespace left the sensor free to be reverted with every test still green — mutation-checked: reverting it now fails the self-test, and previously did not. - New `test_cli_driving_skills_are_named_in_the_install_prose`, derived from SKILLS_REQUIRING_THE_CLI with no names in the assertion, so a fifth CLI-driving skill cannot ship with the prose still listing four. Scoped to the install paragraph, since every skill name appears somewhere in both files. Note the strengthened sensor immediately caught this commit's own first draft ("the four skills that shell out to the CLI"). Reworded rather than exempted, as the sensor is deliberately blunt about any count word next to "skills". Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…orked example Stands to the content track as tasks/skills/lint-tasks-activation.yaml stands to the trigger track: a real, commented, runnable suite built from the bundled outcome template. `ci` is the subject because it sets disable-model-invocation: true (so the rows MUST use the slash form, dogfooding the mechanism), its output is a real workflow file, and it needs no extra credentials in the sandbox. 10 rows, 6 train / 4 test. Because row substitution never reaches `sandbox:`, all ten run against ONE fixture, so each row is a different REQUEST against the same repository and each names its own output path — which takes the agent's filename choice out of the measurement without hinting at the content graded. The fixture is not scenery. It carries `.github/workflows/lint.yml` because `ci` stops outright on a repo with no `.github/` (an empty fixture ties every arm at zero and reads as three bad candidates), and that workflow avoids the string `coder_eval` or `ci` takes its don't-clobber branch instead. Its eval tree sits at `evals/` rather than `tasks/`, at two depths, with one task interpolating $SKILL_SOURCE_PATH, one experiment and a version pin — each making one load-bearing body rule observable rather than unreachable. The fixture lives under templates/, NOT tasks/ as drafted: tasks/**/*.yaml is swept recursively by CE034, test_yaml_migration, test_tags and the CLI's own default task discovery, all of which load every match as a TaskDefinition — and the fixture must contain an experiment file. templates/ is the repo's existing home for sandbox template dirs and is swept by nothing. One row was smoke-run for real ($0.43, SUCCESS, skill_triggered observed 'yes', file_check resolving a file the agent actually wrote) and it paid for itself twice: - max_usd was drafted at 0.50, which the measured row came within 15% of. A slightly longer row would have aborted COST_BUDGET_EXCEEDED and scored as a body failure that never happened. Raised to 2.00. - The emitted workflow invented `anthropics/coder-eval-action@v1` and still scored 1.0. The real action reference is now asserted as its OWN criterion rather than a second `includes` entry: file_check scores found/total, so folding a constant sub-check into the gated criterion would put a fixed contribution in every row of every arm and quietly weaken `mean: 0.7` on the one dimension that discriminates. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

optimize-skill calls a partly-labelled dataset "the dangerous state, because it does not look like one", and it is right: --split keeps the rows whose label matches and SILENTLY DROPS the unlabelled ones, so the run succeeds, the report renders, and every metric is computed over a smaller suite than the file suggests. Nothing in the output says how many rows went missing. That is mechanically detectable, so per CLAUDE.md it becomes a rule rather than a paragraph. - Extract `row_split_label()` in task_loader.py as the single definition of the SPLIT-FILTER convention, and rewrite expand_dataset's inline predicate to use it in one pass. Behaviour-preserving: the old `x not in (None, "")` and the new `x is None or x == ""` were checked over None/""/0/0.0/False/True/[]/{}/ b""/"0"/" "/"None"/missing-key and agree on every one, so a falsy 0 remains a real label compared via str(). - _stratified_sample keeps its DIFFERENT convention on purpose (it folds a missing key into the "" stratum, which turns an explicit None into "None") and now carries a comment naming the divergence — the old comment read as though the two agreed. - CE035 as a @pytest.mark.lint class, mirroring CE034: every tasks/**/*.yaml with a dataset: block must label every row or none. Both legal states pass; only the mixture is a finding. It keys on the CONFIGURED split_field, not the literal "split", and covers inline rows as well as paths. Four dataset-backed repo tasks are exercised for real, not skipped. Also closes a second definition of "labelled" that had survived in the harness: the two template row tests used truthiness (`r.get("split")`), which would have reported a legitimate `"split": 0` as unlabelled — the exact disagreement this phase exists to remove. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Both tracks carried their own copy of the promotion loop: `holm_promote` and `holm_promote_execution`, ~110 lines each, 700 lines apart, each with its own `promoted` conjunction. The two had already drifted in spelling — the refusal term read `refusal is None` on one side and `not refused` on the other — and the two trailing notes were appended from different places, one inside the note ladder and one inline. `gate.decide_family` is now the single loop. It owns the `holm_family` and `resamples_for_family` calls, the outside-the-family branch, the ONE `promoted` conjunction, both trailing notes and both `copy_with` calls. Each track supplies only a `decide(verdict, FamilyFacts) -> TrackDecision` hook, returning its refusal and its notes. The refusal is RETURNED from both because the two tracks disagree about where it comes from and agree about what it means: activation computes it from the family's rank-dependent threshold, execution reads the one `execution_gate` already set. Returning it from both is what makes the conjunction one expression rather than two spellings of one. `FamilyFacts` carries no `threshold`, as planned, and no `family_resamples` either — no hook reads it, since `decide_family` emits the resolution note itself and the activation ladder reads `verdict.n_resamples`, a different number. A field written every iteration and read by nobody is the speculative field this tuple already refuses a `threshold` for. Behaviour-preserving, and that claim was falsified rather than asserted: a 10,982-state differential against both old loops — every combination of p, mean_diff and interval sign, pre-set refusal, failing veto per list, discreteness floor, three family sizes, mixed draw counts and three alphas — is byte-identical on notes, order, count, `promoted`, `holm_rejected`, `holm_alpha` and `gate_refusal`. The one delta is `model_fields_set`, documented where it happens. The new sensor pins `family_size` to the MEASURED count, which needed a family with an unmeasured member to say anything: `len(family)` -> `len(verdicts)` passed the entire suite before this. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Both noise floors opened with the same three guards — reconcile the tree, load the rows, check the split provenance — in the same order, differing only in a rendered label. `gate.floor_preflight` owns them now, and owns the order with them: the reconcile runs BEFORE the load, so a contaminated tree costs no parse and a mistyped variant id still reports the path rather than a stale tree. Split across two functions, that ordering was a comment in each of them. It returns the per-invocation row maps and the split provenance. Both are load-bearing: the activation floor halves the invocations, and both floors key their cached `NoiseFloor` on `provenance.value`, so a preflight that swallowed the second value would serve a train floor to a test lookup. `gate.FirstCause` replaces four identical `nonlocal` refusal closures — one in `execution_gate`, one in each confirm gate, one in `_execution_diagnostics` — three of whose docstrings said they mirrored another, which is a tree asking for one declaration. A class rather than a closure factory because three of the four sites read the value at a distance, from inside a verdict builder called long after the last record; an attribute survives being passed and a captured cell does not. Every refusal message is byte-identical, and the render pins are the witness. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…n function Three functions carried most of the family's complexity: `execution_gate` at radon cc 34 over 427 lines, `_execution_diagnostics` at 31, `_dead_weight` at 29. They are now 14, 8 and 8, and the file's worst function is the untouched `_integrity_checks` at 19. The gate's causes were already ordered by precedence in program order, and that order was the only declaration of the rule — a later cause is usually an earlier one's consequence, so reporting the wrong one sends the reader to the wrong remedy. Each cause is now a named stage returning its message, called in that same order, with `FirstCause` keeping the first. The order is still program order; it is now readable in five lines instead of four hundred. `_read_gate_experiment` returns either a resolved comparison or a refusal, and its refusal carries the row counts on the two paths that HAVE them — a verdict that dropped those would hide an eroded sample behind a message about a variant id. Nothing asserted that before; two tests do now. Behaviour-preserving, and falsified rather than claimed: a differential against the previous commit over 33 end-to-end gate scenarios, a 3,840-case diagnostics grid and 17 dead-weight cases is byte-identical on every message, note and number. Fifteen mutations killed every claim the new tests make except two, and both were real gaps: the untested row counts above, and an unreachable fallback arm in the dead-weight note — deleted, with its invariant asserted where the two lists are built rather than guarded where they are read. Extracting the reconcile into `_refuse_stale_tree` made CE053 fire on the gate, since the rule matches call names within one function body. The rule was taught the stage rather than suppressed on the gate — a `# noqa` there would have retired the check on the family's most consequential reader — and because that match is by bare name, a new test asserts every accepted wrapper really calls the primitive reconciler. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

`load_and_pair` was radon cc 30 and interleaved five concerns in one body: loading both arms, pairing, dropping rows that scored on only one side, balancing replicate counts, and five notes about why the sample shrank. It is cc 9 now, composed from `_pair_rows`, `_balance_clusters` and two note builders, all private to the module — `_PairedRows` already crosses a boundary CE059 cannot see, and none of these widen that. `n_discordant` moved into the balancing stage because that is what it has to describe. Computed on the raw clusters, a row trimmed from three replicates to one can read as discordant on observations the comparison never saw, and the discreteness floor derived from it would then bound a sample that does not exist. `rows_excluded` stays where it was, at the end, since that is the only point that knows both of its causes. Every note is byte-identical and in the same order, which one comment now states rather than four separated appends implying. `pyproject.toml`'s complexity block is re-measured rather than extended. Under mccabe — the tool the ceiling enforces — the `optimize/` family now tops out at 10 where `execution_gate` alone was 23, and under radon it carries no E- or F-grade at all. The ratchet does not move: the binding function is `isolation/docker_runner.py::_build_argv` at 29, which this plan does not touch. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Five import rules read `node.module` and none read `node.level`, so `from ..models import X` — the spelling most of `src/` uses — evaded all of them, and the failure was silent: an import rule that never matches reports zero violations exactly like a clean tree. The fix at the time was to route each rule through `resolved_module`. This makes that the DEFAULT rather than a thing each rule remembers. `BaseRule.visit_ImportFrom` resolves the module once and hands it to `check_import`, which rules override. All six real definers moved onto the hook; `ce020` keeps its own `ast.walk` over `node.module` because it matches a third-party package, where a relative import can never resolve and there is no blindness to fix. CE051 is augmented rather than narrowed, because narrowing it would have been a weakening: its scope is all of `tests/`, and the same blindness lives in `test_optimize_layering.py::_coder_eval_imports`, which is not a `BaseRule` and can never use the hook. So the resolver check stays exactly as it was, and a second check under the same id forbids a rule in `tests/lint/rules/` from defining `visit_ImportFrom` at all — closing the opt-out the hook creates. That second check is anchored against vacuity on `base.py` itself: checking that file asserts it still defines both halves of the hook. Renaming `check_import` would otherwise leave a check that forbids the only shape there is, and every rule file in the tree would pass it while nothing was being verified. No caching was added. The resolver now runs for every rule on every import — measured at 0.6% of the lint suite, which `pytest -m lint` timings confirm unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…, and CE044 retires CE039's restricted expression evaluator declared its allowed operators twice: a tuple the AST walk checked membership against, and a `match` that computed them. Two halves of one decision, kept in step by CE044 — because a wildcard arm returning a value would compute an unhandled operator as something else, which is how `ast.Mod` in the whitelist would have been reported as division by the one sensor class whose entire purpose is catching arithmetic that lies. The halves are now one: two dicts mapping an operator type to the function that computes it. Admitting an operator and implementing it are a single edit, so there is no parity left to pin, and CE044 is retired rather than kept as a rule policing a shape that can no longer occur. Both halves converted — the unary arm too, which left hard-coded as `ast.USub()` would still have been a second declaration. What replaces the parity scan is behaviour. Every admitted operator must compute correctly, since a dict cannot be partially implemented but can be mis-mapped: `ast.Sub` pointing at `operator.add` type-checks and passes every other test in the file. An unadmitted operator must raise naming itself, which is what tells prose drift apart from a parser bug. And the two tables are pinned non-empty and disjoint by arity — the anti-vacuity guard for the other two, which would otherwise still pass against emptied tables by raising. The id stays reserved. `runner.py`'s uniqueness assert covers `ALL_RULES` only, so a class-wired rule could claim CE044 with nothing failing, and `make lint` would then report findings under a number whose documented meaning is something else. The guard is now a parametrized register of both directions — CE056 reserved, CE044 retired — beside its rationale in harness-candidates. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…ontract `tests/test_custom_lint.py` was 9,680 lines and 52 top-level classes, one of which was 3,226 — a third of the file. It is 95 lines now, holding the three runner-level invariants; the per-rule tests live in `tests/lint_tests/`, grouped by what each rule reasons over. `TestPluginArtifacts` became five classes over five modules, sharing a base for the class attributes and grader helpers its methods reached through `self.`. A pure move, and pinned as one: 518 qualified test functions before and after, and `pytest -m lint` collected the same 770 immediately after the split. `test_lint_suite_parity.py` is the standing witness — no test name claimed by two modules, every module importing and holding tests, and the count a floor rather than a literal so adding a lint test is not a chore. Three whole-tree scans had silently narrowed to their own file, which is what this kind of split does when nothing is watching. Each would have kept passing while checking a thirteenth of its subject: the cross-surface rule-id check found zero ids (caught by its own anti-vacuity guard), the normalization-idiom sensor policed one of thirteen modules, and ~25 path constants read `Path(__file__).parent.parent` — correct in `tests/`, one directory short in `tests/lint_tests/`. That last one surfaced as 178 anti-vacuity assertions firing at once, each reporting that the tree it scans had vanished. They now derive from one declared `REPO_ROOT`. `make lint` selected by path and now selects by marker. The two disagreed about what a lint test is, in both directions and invisibly: five classes — CE047, CE050 and CE051 among them — carried no `lint` marker but ran because the target named their file, while `tests/test_lint_no_top_level_run_limits.py` was marked and had never run under `make lint` at all. The five are marked, a test keeps them marked, and `make lint` runs 768 where it ran 749. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The optimize family's docstrings were 40-line essays, and an essay is not read: the contract a caller needs was buried in the history of how the code got there. Those are three different kinds of sentence and they now live in three places — contract in the docstring, why-not-the-obvious-alternative as a comment at the decision site, and what-broke-once in a dated file under `.claude/decisions/`, reached from the code by a one-line pointer. Docstrings over 25 lines in `optimize/` and `reports_optimize.py`: 25 down to 8. `Field(description=…)` blocks over 8 lines in `models/optimize.py`: 9 down to 4. Three of the eight surviving docstrings are module-level and are architectural contract, so they stayed. Nothing was deleted that a caller needs, and nothing was deleted that a sensor reads: `make lint`'s skipped count is identical and its passed count only grew. The field descriptions were trimmed rather than moved wherever they are user-visible in the JSON schema. The convention is prose, deliberately: "is this sentence a contract or a defect history" is a judgement no rule can make, and a heuristic for it would be a rule policing wording. What IS checked is mechanical — no orphaned decision file, no pointer to a file that does not exist, and two counted ratchets so the next essay is a visible decision rather than a drift. The ratchets are ceilings, not equalities: a docstring that resists trimming because every line really is contract should stay, and the README says so. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

89,090 characters down to 44,908. The file is prepended to the context of every request in every session, so a paragraph nobody reads is paid for on every call. One line was 33,247 characters — 37% of the file — naming 35 CE ids with a rationale essay for each, every one of which already existed in the rule's own module docstring. Duplicated prose is worse than absent prose there, because the copy drifts and a reader cannot tell which is current. It is now the standing instruction plus four bullets: where a rule lives by kind, where its tests live, how to claim a number, and the statement that each rule's own docstring is the authority on what it does and what it deliberately does not catch. Exactly one sentence in that line was a DERIVED surface — CE036's exemption list, bound in both directions by a test — and it is kept verbatim, now introduced by a clause saying why it lives here rather than in the rule, so the next person to trim does not move it. Nine more long lines point at their authoritative home instead of restating it. In every case that home already existed: the early-stop module and the task guide, the harness-parity doc, the module docstrings, and the new decision log. Two assertions guard it. A character ceiling, and — because a ceiling can be satisfied while one line still holds a third of the file — a per-line limit. The long line is also the shape that hides a derived sentence: nobody diffs 33,000 characters on one line, which is how CE036's list drifted from the code inside one in the first place. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The worst of it: Phase 9's docstring trim replaced the specification of `cost_quality_front`, `headroom_ceiling` and `candidate_leaks` with descriptions written from their NAMES rather than their bodies, and Phase 10 then canonized one of those in the decision log. `cost_quality_front` does not rank by a ratio — it is a 2-D Pareto filter gated on row-set coverage, and that conjunct is what stops an arm which crashed 5 of 6 rows from dominating the incumbent. `headroom_ceiling` cannot see other arms and never returns None, so a caller writing the documented `is None` guard got dead code; its real invariant, that the denominator is the FULL row count and not the selected subset, was the line that went missing. `candidate_leaks` takes strings: `skill_text` is the reader, and handed one file the preflight comes back CLEAN — byte-identical to a genuinely clean candidate, which is the worst shape a preflight can have. All three are restored from the code, and the decision log with them. The ratchet that caused it now has slack. It sat exactly at the count while its own README told authors to keep a docstring whose lines are contract — advice that reddened the build. A cost control must not be able to win an argument against a specification. Two sensors were open in the case they were written for. The reserved/retired-id register checked `ALL_RULES` and `tests/lint/rules/`, so a class-wired `TestCE044` — the exact shape its docstring names — passed everything; it now intersects the `TestCE<NNN>` class set too. And CE036's exemption parity read every backticked name in the sentence, so `skill_name` could be deleted from the list and still pass, because the sentence names it again downstream; it reads the parenthesised list now. Both verified by mutation. CE034, CE036 and CE060 hand-rolled `tasks/` discovery, so a `.yml` task was invisible to all three — a leaking `probe.yml` produced zero cases and a green `make lint`. They route through `task_yamls` now, from one declaration with a non-emptiness assert, so a path break fails instead of collecting nothing. The rest is prose that had stopped being true: six stale `optimize/` line counts in the very pyproject block that claims to be re-measured, a function attributed to the wrong module, `no_floor` "called four times" where Phase 3 left one, three disagreeing counts of the execution gate's refusal causes (none of them the right eight), "TWO consumers" of the leak primitive where there are three, and fifteen wiring pointers still naming a file the classes no longer live in. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Both were live after the lint split, and both fail OPEN — a stale pointer reads exactly like a correct one, which is why fifteen of them survived a phase whose whole subject was moving those files. `<path>::<Name>` wiring references now have to name the file that declares the name. Each is the "where is this rule wired" pointer a reader follows from a detection body to the test that runs it, and a wrong one sends them to a 95-line file holding none of the subject. The decision-log pointers next door were already checked in both directions; the asymmetry is what made this worth closing rather than deferring. And a module nested under `tests/` may no longer reach for the repo root with `Path(__file__).parent.parent`. The expression is CORRECT one directory up, which is what makes it dangerous: about 25 constants carried it into `tests/lint_tests/` unchanged, and the failure was not a red import but 178 anti-vacuity assertions firing at once, each reporting that the tree it scans had vanished. Depth-aware, so it says nothing about the files where the idiom is right. Both mutation-verified. The rest of the run's candidates are deferred with the measurement that shows why they are not cheap — the unread-tuple-field rule reports 15 false positives against 2 real hits, because a field read by tuple unpacking or by a consumer outside the package looks identical to a dead one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…or composites `coder_eval/optimize/api.py` is the module `SKILL.md` will import and nothing else: rank 4, it composes the ranks below it and returns the markdown block the skill prints. Two composites land here — `activation_floor_report` and `execution_floor_report` — absorbing the guards, the cache read and the model resolution that the two Step 6 / Step 8 fences spelled out in markdown. The ladder accepts a fourth rank through five amendments in `tests/test_optimize_layering.py`, of which one is substantive: the "nothing in the family imports the RENDERER" loop narrows to the DECISION layer. That prohibition exists to stop a module that decides from also owning its presentation; rank 4 decides nothing, and returning a rendered block is its whole contract. `_RENDERER_CONSUMERS` is asserted to be exactly one module at the maximum rank, and `_renderer_prohibited()` is shared by the assertion and the test that proves it can still fail. `execution.resolve_arm_model` gives the four-times-repeated `resolve_model(load_arm_rows(...))` idiom one declaration, and therefore ONE reasoned `# noqa: CE053` instead of four. Its reason argues from the return value rather than from immunity: a contaminated tree CAN change what it returns, but only toward `UNRESOLVED_MODEL`, which bars the cache — and every consumer's own `floor_preflight` refuses the tree first. `reports_optimize.render_noise_floor` keeps `api.py` free of formatting. It says what the two `print()` calls could not: that no floor is not a floor of zero, and that a floor of exactly zero is a real answer with three causes — one of which (a criterion index pointing at something already perfect) has happened on the bundled outcome template. The two metric labels move to `models/optimize.py` beside `TARGET_LABEL`, on the same cycle-free-leaf precedent. `NoiseFloor.metric`'s default, `activation.py`'s probe, `_f1_yes`'s selector and `execution.py`'s probe all read them now, so the label a block renders cannot disagree with the record it describes. SKILL.md: two fences, 6 lines each, importing `coder_eval.optimize.api` alone. Five paragraphs rewritten — every one described code these fences no longer run. Python-fence lines: 427 -> 400. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

`discreteness_report`, `row_matrix_report` and `cost_quality_report` are straight-line compositions, so what moves here is mostly arithmetic the fences made the reader do: `DEFAULT_ALPHA / survivors`, and the two front calls that have to be passed to the same renderer in the right argument positions. `render_discreteness` is the block that stops a reader reaching for the wrong lever. Adding rows the two arms AGREE on makes the discreteness floor WORSE, so "buy more rows" is advice that can leave a user strictly further from a promotion — the shipped fence printed a bare integer and left that to the prose beside it. Its `None` branch names the family and the draw count as the remedy, which is only honest because `discreteness_report` now rejects an empty suite: that is the other cause of a `None` from `min_discordant_rows`, and the two want opposite advice. `criterion_index=None` is documented once, in the parameter's docstring, instead of in a comment on every call. The skill keeps one prose sentence covering both Stage A blocks, since a reader of a markdown fence cannot read a docstring. `assert_matches_render_pin` moves to `tests/optimize_fixtures.py` and loses its underscore: two suites now pin the same rendered blocks — one through the renderer, one through the composite above it — and a second copy of that helper would be a second place a committed pin could be quietly regenerated from. A pure move; no pin file is touched, so `tests/lint/estimator_ledger.py` sees no modification. `_ENTRY_POINTS` in the composites' tests becomes the running list of every composite taking `run_dirs`, asserted through the public surface rather than through the private guard they share — a composite that forgets to call it is invisible to any test of the guard itself. Python-fence lines: 400 -> 390. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…econciled Both fences carry a fallback whose absence changes what the block MEANS, and one of them carried a live bug: `SKILL.md`'s Step 7 fence read rows and built the ceilings table from them without reconciling the tree. `run.json` is written per INVOCATION while the tree is APPEND-ONLY, so a re-used `--run-dir` leaves an earlier call's rows on disk — they load, parse and are pooled into a confident number. CE053 exists to force exactly that reconcile and cannot see markdown, so the fence sat outside its reach. Moving it into `api.py` puts it inside. And a composite can do better than the library does. `reconcile_arms`' docstring says the RESPONSE is deliberately not shared, because `ArmRowScores` has nowhere to put a refusal — so `arm_row_scores` only warns, which a skill session never sees. A composite returns markdown, which HAS somewhere to put it: the staleness goes into the printed block and therefore into the ledger. That rule is not a headroom detail, so it applies to every composite that reports a number from rows it read, with `TestEveryReportingCompositeNamesAContaminatedTree` covering all four in both directions — a note that fired on a clean tree would stop being read. `render_corpus_check` owns the hole-versus-loss distinction the fence left to the reader. A missing score is not a loss and not a pass, and the two causes the corpus cannot tell apart — the row errored, or it belongs to this skill's other suite — are named rather than collapsed. It also owns the empty-CORPUS block, beside the no-ARMS one it is not: both are normal and they mean opposite things. `SystemExit` is gone from the library. It was markdown's error channel; from a library it kills an interpreter that had other work to do, and the skill's session is exactly that interpreter. Two things this migration nearly lost and did not: Step 11's fence had been depending on an `arms` binding Stage A stopped providing in 2/11, so it now binds its own inputs and runs in a fresh interpreter; and the round-1 "no floor is expected, re-print after the control arm" remedy is back in the prose. Python-fence lines: 390 -> 351. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

`load.row_replicate_scores` is `row_score` reduced over a whole arm without averaging, and the per-replicate vector is the one reading a mean cannot give: a row with zero variance on both arms and a non-zero delta is a REPRODUCIBLE behavioural change, not noise. Measured on a real round, two of them with opposite signs cancelled to a suite delta of +0.0001. The skill hand-wrote this as a `replicates()` closure reaching into `r.success_criteria_results[grader_index].score` with no bounds check, so a mistyped position raised `IndexError` from markdown after the round was paid for. Now an index past every row's criteria list is a `ValueError` naming the real count — while a row whose own list is SHORTER stays a hole, because a crashed row is not a reason to refuse a reading of the other fourteen. Scoping the bound to the WIDEST row is what makes those two rules non-overlapping. `replicates_report` reconciles both arms in ONE sweep, not two: a dir carrying both arms is a single re-used `--run-dir`, so it is one fault and gets one sentence naming both. That is the second half of the fix 3/11 started — the shipped fence built its table from rows it never reconciled. Three ways this block could have read as a result instead of a fault, all now loud: an arm that scored nothing (a full column of holes, which the block's own prose calls "present on one arm only"), an arm compared against itself (every row dead), and an index past the end. The one case that cannot be attributed — nothing scored anywhere, where a bad index and a crashed arm are the same input — is documented rather than guessed at, and the caller names the arm. The fence had also been showing the activation track's three-directory layout and a `criterion_index` under an execution-track heading, so its deltas could not reconcile with the verdict beside them. One run dir, no index. Python-fence lines: 351 -> 349. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

… fence `leak_report` absorbs the anti-memorization scan, and with it the `def body(arm)` closure that was the last function definition in any of the skill's fences. Two things it deliberately does not expose. There is no `split` parameter, because both alternatives are wrong: scanning the whole suite flags content a candidate is entitled to be fitted to, and scanning the test rows reports on a split the proposer is blinded to. And `baseline_dir` stays required, because a candidate is diffed against what it was edited FROM — from round 2 that is the lineage head, whose snapshot lives under the round that produced it, and defaulting it would re-report every span the head added on every round. The missing baseline is now a hard error, where a missing candidate directory is only named. That asymmetry is the whole point: an absent baseline makes `skill_text` return an empty string, so the diff silently becomes the absolute scan the diff exists to replace — measured on this repo's own `ci` skill, an absolute scan flags five strings that are simply the output contract its suite grades, and a checker that fires on the shipped skill is one you learn to ignore. An arm that could not be scanned gets its own channel rather than a fake span. The first cut folded "no skill directory" into the findings list, which rendered a wiring fault as "1 span(s) the baseline does not have" — a mis-snapshotted arm reported as a candidate that memorized something. `LEAK_SCAN_BOUNDARY` follows the `COST_FRONT_ADVISORY` precedent its comment cites, and now follows both halves of it: the constant keeps the rendered block from drifting, and a new sensor keeps the prose beside it from drifting. A clean scan is the moment a reader most needs to be told what clean does not prove. Python-fence lines: 349 -> 327. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

`search_report` is the accept-or-revert reading for one explored arm, and it is emphatically not a gate: it corrects for no multiplicity, because the alternative to accepting is reverting a step rather than shipping a skill. It is also the first caller `lineage_head_scores` has ever had in `src/`. Two guards the fence expressed as crashes become errors that name their cause. No recorded lineage was a `SystemExit`, which from a library kills an interpreter that had other work to do — and the skill's session is exactly that interpreter. And an arm that scored nothing is now `wrong_path_reason`, naming the variant, the suite and every run dir. That second one is worth stating precisely, because the plan mispredicted it. The plan expected `arms[0]` to raise `IndexError` on a wrong variant id; `arm_row_scores` in fact returns one arm per requested id unconditionally, so the index is safe and the shipped fence did not crash. What it did was hand the empty arm to `search_compare`, which refused with "the two rounds share no rows … a wiring fault" — sending a reader to check sampling seeds and snapshot mounts for what is a mistyped slug. Both `arms[0]` sites in the module now record why the index cannot raise and what the real fault is. Python-fence lines: 327 -> 304. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

`candidate_variants` is a SEQUENCE, and that is the whole design. The shipped fence's most available mistake was correcting one candidate at a time, which silently reverts to an uncorrected alpha — every token still present, every single-verdict test still green, and a false promotion with nothing in the output to show it. A sequence in and one `holm_promote` out means there is no single-candidate call shape left to get wrong, and a monkeypatched counter pins it rather than a reading of the code. Four ways that family could still be inflated, all now hard errors and all silent before: a bare string (a `Sequence[str]`, so "cand-a" is one candidate per letter), a duplicate, a one-shot iterable (it passes the guard, then yields nothing to the gate loop and renders an empty block), and the INCUMBENT in the candidate list. The last is one copy-paste away, because Stage A's `variant_ids` legitimately starts with the incumbent — and it gates an arm against itself while tightening the threshold for every real candidate, whose blocks say nothing about why. `seed_stability_report` had two ways to render a confident lie. An arm gated against itself, and DUPLICATE seeds: re-running one draw three times reports 3/3 agreement at a spread of 0.0000, which is the most confident stability claim available and rests on a single bootstrap. Both refuse now. And when every seed's gate REFUSES — a cross-split pair, a contaminated tree — the block used to read "STABLE — would promote at none of 3 seeds". `SeedStability` has no channel for a refusal, so a comparison that was never made rendered as a confident negative. The renderer now says NOT A STABILITY READING and sends the reader to the gate's own block, which is where the refusal is. No estimator knob is exposed: every one is a way to produce a number that is not comparable with the floor recorded beside it. `sibling_indices` is the exception and the docstring now has the reason the right way round — `None` DERIVES and checks every sibling position, so omitting the argument is the safe state and the parameter exists to disarm the veto deliberately. Python-fence lines: 304 -> 289. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The mapping IS the family on this track: `paired_comparison` fires only for exactly two variants, so each candidate is gated in its own run dir and the Holm family lives ACROSS them. One `holm_promote_execution` call over all of it, pinned by a counter rather than by a reading of the code. **And the family can shrink without anyone saying so.** A verdict with no p-value is not a family member, so an arm that REFUSED drops out and `m` falls — which is right for that arm and wrong for its siblings, who were predeclared against the larger family and are then decided against a looser threshold. Measured: two keys pointing at one run dir promoted the good arm "across a family of 1" while the round had predeclared two. Every other guard here fails closed; this one fails open, and this composite is the only thing that holds the predeclared count. It now says so, in the block, above the verdicts it is warning about. `engagement_criterion_index` was exposed with no word about what it does. `None` disarms a veto — the engagement reading feeds `integrity_checks`, and a failed one forces `promoted` False — so it is documented like the twin's `sibling_indices`, and a test now passes a non-default value: hardcoding both indices had left every test green. The incumbent-as-candidate guard is shared with the activation twin now. One claim, one message; the two copies had already drifted in their wording. The tests were also measuring the wrong thing. A candidate that beats the incumbent by the SAME amount on every row is refused for zero variance, so a family built from a uniform shift exercises the refusal path in every test — including the ones about promotion and ordering. They use `WINNER` now, and the family reads PROMOTED. Python-fence lines: 289 -> 280. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

`confirm_gate` needs the HOLM-CORRECTED Stage B verdict, because `promoted` is what Stage C classifies against — and `measurements.json` is `extra="forbid"` with nowhere to put one. So the composites take the Stage B FAMILY and recompute it: gate everything, correct once, select the named candidate. The bootstrap is seeded, so that is the same verdict Stage B printed, and it costs CPU over rows already on disk. It also removes the failure the skill's own prose warned about, where the fence needed a `promoted_verdict` from an earlier snippet and raised `NameError` in a fresh interpreter after the round had been paid for. The first cut refused any candidate whose `promoted` was not True. That was wrong, and rank 1 says so in writing: `confirm_train_note`'s docstring is "A NOTE, not a refusal: a reader may legitimately want to confirm a candidate that separated and was then vetoed by a guardrail", and `confirm_train_refusal` renders the not-a-result case. A rank-4 composite whose contract is that it decides nothing was deciding that the other way, and made both helpers unreachable from the only surface the skill uses. It now refuses only a verdict with NO STATISTIC — a gate that could not measure is not a candidate that lost, and calling it one sends a reader to rewrite a candidate whose gate never ran. `_family_shrink_note` is one declaration across all four Stage B / Stage C surfaces. The activation side had none, and it needs one most: a candidate that changed nothing has no discordant rows, hence no p-value, hence silently leaves the family — so the block printed "recomputed over a family of 2" directly above a verdict reading "Holm applied across a family of 1". `confirm_one_candidate` runs FIRST now. A shortlist used to die on an unhashable dict key after the entire family had been re-gated, which on a five-candidate round is fifteen seconds of bootstrap for an error rank 1 already words. And the test that claimed no `track:` discriminator existed was decoration: the module legitimately carries `_track_verdict(…, track_name)`, so the grep was satisfied by a rename. It now asserts the real invariant — two public entry points, neither taking a track selector, neither carrying the other's parameters. Python-fence lines: 280 -> 258. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…tself The last and largest fence: 136 lines of markdown driver, including a commented-out track half the reader had to choose between and a `TRACK = "activation"` string they had to hand-edit. Two writers now, one per track, and the activation one has no grader-fingerprint parameter at all — which is what makes the impossible combination unrepresentable rather than asserted against. `fronts.lineage_head` moves a rule out of prose that the prose itself warned had a wrong answer: the head is the arm with the highest MEAN of `row_scores`, not the top `f1.yes` arm, because the search loop compares means. A NaN cell made it order-dependent — every comparison against NaN is False, so the winner became whichever arm was listed first, defeating the tie-break the docstring promises. It routes through `_finite_scores`, which is the module's existing answer for a NaN. Two ways a recorded round used to lie about itself, both found in review: `lineage_head_variant=None` meant DERIVE, so a REVERTED search round advanced the head. A search round has one arm, so deriving names the rejected candidate — and the bar every later round is measured against then moves on a step that was rejected. There was no correct call to make: the previous head is not an arm of this round, so the validator refuses it. Three states now, and `None` means no head, which is exactly what `lineage_head_scores` skips. And the grader fingerprint was recorded unvalidated. `check=True` rules out a non-zero exit, but the grader exits non-zero ONLY in that mode — its own docstring says "a score-shaped line would be recorded by a caller as the fingerprint itself". A grader predating the flag reads `--fingerprint` as a row id and prints a score line carrying an absolute path: constant on one machine while the real grader moves, different on the next machine. It is checked against the digest shape now, and the test grader emits a real one and verifies its own argv. Smaller, all real: the first round claimed "no fingerprint on one of the two rounds" with both freshly recorded; `record_promotion` reported the submitted count as recorded, hiding the de-duplication the corpus exists for; the grader path came from `split()[1]` rather than the `$TASK_DIR` token, so `uv run python $TASK_DIR/...` would have fingerprinted `run`; a second `run_command` won silently; the subprocess ran after the bootstrap so its raise paths threw the bootstrap away; and no arm needed to have scored anything, so a mistyped id wrote a suite digest over zero rows and made the next round report a suite change. Four of the new tests were vacuous and are not any more — each was mutated and watched to fail. `record_noise_floor` in particular was never exercised at all: the fixture yields no floor, so deleting the write kept everything green. Python-fence lines: 258 -> 126, from 427 at the start. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Before this plan the library's skill-facing API was whatever the snippet binder happened to resolve: fifteen fences reaching into five decision modules, carrying 427 lines of guards, fallbacks and track branches that no test could reach because markdown does not execute. CE066 is what converts that into a declared surface — `SKILL.md` imports `coder_eval.optimize.api` and nothing else, so a fence that still needs a primitive is a fence not finished, mechanically. Its reader is a shared reader in the `action_docs.py` mould rather than a `BaseRule`, because it reasons over markdown. CE051 fired on it and was right: it matched `node.module` without routing through `resolved_module`, which is the exact habit four import rules failed OPEN on. The first fix was a reasoned suppression; the second was to call the resolver, so the rule is satisfied by construction. A relative import inside a fence is reported by shape instead — a fence has no parent package, so it is code that cannot run. Both binder thresholds are exact counts now, measured on the real file: 13 and 16, where they were floors of 5 and 15. A floor was right while fifteen multi-line fences could disagree about a keyword; with one call per fence it is unfailable in practice and would not notice ten of them going quiet. The hardcoded name list is extended rather than replaced — the four library names that survive in prose keep their guard, five composites the procedure must name join them — and word-bounded, because four of the nine are prefixes of the others and a substring match would have let the library name vanish while staying green. Also here: the three ledger notes get direct tests rather than transitive ones, and `.claude/decisions/2026-08-20-the-skill-facing-api.md` records why composites and not a facade, why every one returns `str`, why Stage C recomputes, why the tracks get two functions instead of a discriminator, and the §7 declination. Python-fence lines: 427 -> 126. `make verify` passes, coverage 92.85%. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

`_ENTRY_POINTS` claimed to cover every composite taking a run-dir sequence and to grow each phase. It fell behind twice — the second time in the very phase whose comment justified widening `_require_run_dirs` by pointing at the ledger writers' two sequences. The failure is silent: the parametrization just runs one case fewer, and a composite's boundary guard goes untested while `api.py`'s docstring keeps promising every entry point has one. `test_every_run_dir_parameter_has_a_boundary_entry` derives the set by inspecting the module for a `Sequence[Path]` annotation and compares it BOTH ways, so a stale entry is a failure too. It found one real gap on its first run — `record_round_execution(run_dirs)`, the only one of the four ledger sequences with no entry. Each registry row now carries the composite it exercises, which is what makes the comparison exact rather than a name-prefix guess. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…ence that could not run The per-phase reviews each saw one phase. These are what only the whole diff shows. **The ledger writers persisted a contaminated tree with nothing to doubt.** Every reporting composite sweeps with `reconcile_arms` and prepends a staleness note; `record_round_*` did neither, and CE053 could not see them because they reach the tree through `arm_row_scores` rather than `load_arm_rows`. A leftover row from a re-used `--run-dir` therefore landed in the persisted vectors, both fronts, the lineage head and the suite digest — and a later CLEAN round then reported "The SUITE CHANGED" for a suite nobody touched. A reporting composite at least hands a reader a block to doubt; a written round is doubted by nobody. Both writers reconcile and name it now, and the class is recorded as a harness candidate, because widening CE053 to `arm_row_scores` would fire on every current caller. **Stage C's activation twin had no family-shrink note** while three separate docstrings — the helper's, the renderer's and the decision doc's — all said "all four Stage B / Stage C surfaces". It is the worse of the two to miss, because that surface PRINTS the family size it recomputed against, so a silent shrink makes that line a false claim about the threshold the winner cleared. The activation track had no shrink test on either surface, which is exactly why it shipped. **The headroom fence could not run its first print.** It passed `variant_id="incumbent"` against `baseline-1`, which a plain `coder-eval run` writes under `default` — so the composite raised. The skill says this correctly 245 lines earlier and contradicted itself here. Also: the budget table said Stage B execution is "one invocation", where the family lives across one run PER CANDIDATE and the method file says so; the corpus fence's `criterion_index=0` fails OPEN on the execution track, where position 0 is the engagement criterion the skill requires to be 1.0 on every row — a silent all-clear on the one check that catches what an aggregate hides; two paragraphs still told the reader to call primitives CE066 now forbids importing; the activation ledger paid for a bootstrap before a check its twin documents as cheaper first; `headroom_report` paraphrased a rank-0 message instead of using it, dropping the glob that makes it actionable; `variant_ids="incumbent"` rendered one arm per letter; a NaN `threshold` cleared every arm silently; and the grader subprocess had no timeout where every other external-tool call in `src/` does. Stage C's activation composite is now a shown fence rather than a sentence, so every one of the 18 is reachable by copying rather than by reconstruction. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The rebase onto main put two independently-grown lint registries in one tree. CE036-CE039 and CE043 each named a DIFFERENT rule on each side, and CE037 named two `BaseRule`s at once — a hard failure at import, where `runner.py`'s id-uniqueness assert cascaded into 84 collection errors. Main is shipped, so this branch renumbers, which is what `runner.py`'s own comment prescribes ("the loser must renumber") and what this branch already did once when its CE035 became CE060: CE036 -> CE061 a dataset row's prompt may not contain what its criteria grade CE037 -> CE062 F1 is computed in exactly one place CE038 -> CE063 `extra="forbid"` must reach the models it appears to protect CE039 -> CE064 a prose surface's arithmetic is checked by COMPUTING it CE043 -> CE065 `run` and `plan` declare the same row selectors Scoped per file rather than tree-wide, because BOTH meanings of every one of those numbers is live in this tree: main's CE036 is the `live_verdict` contract, its CE037 forbids a dead private helper, its CE043 forbids truncating captured command output. A blanket rewrite would have renamed those too. Main's five ids are untouched. Three fixes fall out of main's API changes rather than the numbering: * `tests/lint/live_verdict_contract.py` pointed at main's CE036 test class in `tests/test_custom_lint.py`, which this branch's split emptied. The class now lives beside CE025 in `tests/lint_tests/test_lint_ast_rules.py`, and the branch's own wiring-pointer sensor is what caught the stale reference. * two fixtures still built a `reference` from the `code:` form main removed in favour of directory-only references. CLAUDE.md is back under its 46,000-character budget. Main's per-rule paragraph (CE037/CE038/CE039) is gone and its reference-solution bullet is a pointer now, which is the policy the file states two paragraphs further down: a rule's own module docstring is the authority. Every load-bearing fact from that bullet — the stacking rule, the caps deliberately NOT dropped, the hard error on a window that cannot be applied, both known gaps — was verified present in `fs_permissions.py`'s docstring, `orchestration/evaluation.py`'s, or docs/DOCKER_ISOLATION.md before the text was cut. Note for whoever adds to CLAUDE.md next: it now sits 2 characters under the cap. The budget was written when this branch had 496 characters of headroom and main had not yet added `fs_permissions.py`; the next addition raises the number deliberately or displaces something. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

+# re-declare a base field absent from here, and every entry here must genuinely differ from the
+# base — in default, in description, or both. That second half is the CE063 `EXEMPT` pattern: a
+# stale licence must not outlive the trade it recorded.
+_FIELD_OVERRIDES: tuple[tuple[str, str], ...] = (


uipreliga requested review from akshaylive, bai-uipath and tmatup as code owners August 12, 2026 21:36

This comment was marked as outdated.

Sign in to view

uipreliga changed the title ~~feat(dataset,plugin): --split row filter + /coder-eval:optimize-skill, measured end to end~~ feat(plugin): optimize-skill's execution track, measured end to end — and fix skill_triggered counting refused calls Aug 13, 2026

github-advanced-security Bot found potential problems Aug 13, 2026

View reviewed changes

Comment thread tests/test_custom_lint.py Fixed

uipreliga changed the title ~~feat(plugin): optimize-skill's execution track, measured end to end — and fix skill_triggered counting refused calls~~ feat(plugin): optimize-skill's execution track, measured end to end — plus the skill_triggered and --split fixes it surfaced Aug 14, 2026

github-advanced-security Bot found potential problems Aug 14, 2026

View reviewed changes

Comment thread tests/lint/computed_claims.py Fixed

Comment thread tests/test_optimize_gate.py Fixed

Comment thread tests/test_optimize_measurements.py Fixed

Comment thread tests/test_optimize_measurements.py Fixed

This comment was marked as outdated.

Sign in to view

uipreliga force-pushed the feat/plugin-optimize-skill branch from ecbabea to 8ea19ce Compare August 20, 2026 01:52

uipreliga and others added 19 commits August 20, 2026 16:32

style: apply ruff format to the reachability lint assertion

2c30397

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

docs(harness): record the all-skipped-run-exits-0 gap found while add…

ae3c39b

…ing --split Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

uipreliga and others added 25 commits August 20, 2026 16:46

uipreliga force-pushed the feat/plugin-optimize-skill branch from 1c353a0 to 68fa740 Compare August 21, 2026 00:31

github-advanced-security Bot found potential problems Aug 21, 2026

View reviewed changes

Navigation Menu

Uh oh!

feat(plugin): optimize-skill's execution track, measured end to end — plus the skill_triggered and --split fixes it surfaced - #109

feat(plugin): optimize-skill's execution track, measured end to end — plus the skill_triggered and --split fixes it surfaced#109
uipreliga wants to merge 202 commits into
mainfrom
feat/plugin-optimize-skill

uipreliga commented Aug 12, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Aug 12, 2026 •

edited

Loading

Uh oh!

This comment was marked as outdated.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as outdated.

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

uipreliga commented Aug 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

The feature

Why the two gates differ (worth a reviewer's eye)

Validation

Three bugs caught before shipping

The tutorial (docs/tutorials/08)

Notes for review

Late changes (after the first review pass)

Late changes (second pass): the execution track measured for real — and a criterion bug it uncovered

⚠️ Behaviour change reviewers should look at

What the run found

Corrections to claims made above

Also in this pass

Third pass — the review's P0/P1/P2 fixes, then a tutorial audit

Two behaviour changes to look at first

The rest

Tutorial audit

Notes for review

Uh oh!

github-actions Bot commented Aug 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Review in Progress

Todo List

Uh oh!

This comment was marked as outdated.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as outdated.

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

uipreliga commented Aug 12, 2026 •

edited

Loading

github-actions Bot commented Aug 12, 2026 •

edited

Loading