feat(cli): plan expands datasets and takes --split, plus four correctness fixes

.claude/harness-candidates.md

@@ -457,14 +457,21 @@ with the two `action.yml` items above — one considered change to the action's
   depend on, mirroring how CE030 pins doc/schema parity.
 ## From the split-field / optimize-skill plan (2026-08-12)
 
-- [ ] **A run whose every task is skipped exits 0 — a green run of zero tasks.** When
-  `resolve_all_tasks` demotes every task to `skipped_tasks` (a load failure, `skip: true`,
-  or now a `--split` selector matching no labelled row), the run reports success: nothing
+- [ ] **A run whose every task is skipped exits 0 — a green run of zero tasks.** *(Narrow case
+  CLOSED 2026-08-13: a `--split` selector that matches no labelled row now raises
+  `SplitSelectorError` out of `expand_dataset`, which `resolve_all_tasks` re-raises and the CLI
+  turns into a `typer.BadParameter` — exit 2. The GENERAL case below stays open: `skip: true`,
+  load failures and tag filters that match nothing all keep today's exit-0 behaviour, because
+  making those fatal changes exit semantics for deliberate quarantine workflows and needs its own
+  decision plus tests per case.)* When
+  `resolve_all_tasks` demotes every task to `skipped_tasks` (a load failure or `skip: true` —
+  no longer a `--split` typo, see above), the run reports success: nothing
   failed, so the exit gate in `cli/run_command.py` — which keys only on failed/errored tasks
-  and suite gates — passes. Verified directly: `coder-eval run <suite> --split holdou`
-  prints one yellow "1 task file(s) skipped" line and exits 0. This is pre-existing, but
-  `--split` makes it reachable by a one-character CLI typo rather than a broken file, and
-  the whole point of a test confirmation is that you trust its verdict. Not guarded, and
+  and suite gates — passes. Verified directly before the narrow fix: `coder-eval run <suite>
+  --split holdou` printed one yellow "1 task file(s) skipped" line and exited 0. `--split` was
+  what made it reachable by a one-character CLI typo rather than a broken file, and
+  the whole point of a test confirmation is that you trust its verdict. Still unguarded for
+  the remaining paths, and
   not a five-minute fix: making an all-skipped run non-green changes exit semantics for
   every skipped-task path (including deliberate `skip: true` suites and tag filters that
   match nothing), so it needs a decision about which of those should be fatal, plus tests
@@ -474,3 +481,7 @@ with the two `action.yml` items above — one considered change to the action's
 - [ ] **Semantic answer-leak in a task prompt** — a prompt that describes the graded behaviour in *different words* ("list the paths explicitly rather than with a recursive wildcard" while grading an explicit glob) scores well whether or not the behaviour happened, and in an A/B an arm that deleted the rule still passes. CE061 catches only the verbatim form; the semantic form needs an LLM judge or a `lint-tasks` pass over this repo's own `tasks/`, neither of which is cheap or deterministic. — caught in the final review of c/2026-08-13-optimize-skill-fixes.md, where 4 of 10 rows in a shipped worked example had it.
 - [ ] **A doc claim that contradicts merge semantics** — `optimize-skill` told users to declare `allowed_tools` in an experiment's `defaults: agent:`, which is a silent no-op because those fields merge by `replace` and the task layer outranks experiment defaults. Detecting "this prose recommends a config location that the merge order makes ineffective" would need the rule to model the layer stack against prose, which no existing rule shape supports. — caught in the final review of c/2026-08-13-optimize-skill-fixes.md.
 - [x] **`_normalized()` not used by every prose sensor** — CLOSED: all 9 sites converted, and `test_no_sensor_inlines_the_normalization_idiom` now forbids the raw form. Original note: — 8 sensors in `tests/test_custom_lint.py` still inline `" ".join(path.read_text().split())`, so a future one copied from the wrong neighbour is defeated by a line wrap (the bug that let a stale skill count ship past 91 green tests). A rule forbidding the raw idiom in that file is easy; the conversion sweep was out of scope. — caught in the final review of c/2026-08-13-optimize-skill-fixes.md.
+
+## From the optimize-skill review v2 plan (2026-08-13)
+
+- [ ] **"The ToolStart seam decides" is now a PER-CRITERION property, not a global invariant.** `command_executed`'s verdict is decidable from the tool call's inputs; `skill_triggered`'s is not (for the `Skill` tool the body is delivered AS the result, so an in-flight call engaged nothing). A new `LiveSuccessCriterion` must state which seam its `live_verdict` is decidable at, and a criterion that decides at the ToolStart on information only the result carries silently diverges from its own frozen check. Not mechanically detectable today: the property is about what a `live_verdict` implementation *reads*, which no AST rule can infer — a rule would have to know that `result_status` is the field distinguishing the two seams. A cheaper partial guard would be a test-level convention (every live criterion has a "not decided before the result" or "decided on the call" test), which is a sweep rather than a rule. — caught implementing Phase 1 of c/2026-08-13-optimize-skill-review-v2-fixes.md.

docs/DATASETS.md

@@ -149,11 +149,16 @@ Three behaviours are worth knowing before you rely on it:
   safe direction (an unlabelled row never leaks into a named split), but during an incremental
   migration it silently *shrinks* the suite, which moves the aggregate metrics `suite_thresholds`
   gates on. Finish labelling before you compare two runs.
-- **A labelled task with no row in the requested split is skipped, not fatal.** Expansion raises,
-  naming the splits that do exist, and the run records it in `run.json`'s `skipped_tasks` and carries
-  on with whatever else resolved. So a mistyped selector (`--split holdou`) produces a run of **zero
-  tasks that still exits 0** — check the skipped-task count, not just the exit code, when a split run
-  comes back suspiciously clean.
+- **A labelled task with no row in the requested split aborts the run.** Expansion raises
+  `SplitSelectorError`, naming the splits that do exist, and — unlike every other dataset error —
+  `resolve_all_tasks` re-raises it rather than recording a `skipped_tasks` entry. So a mistyped
+  selector (`--split holdou`) fails loudly with a non-zero exit instead of producing a green run of
+  zero tasks. The distinction is deliberate: the other errors describe a malformed *file*, and one
+  bad task must not abort a suite; this one describes a malformed *invocation*, and the same
+  selector applies to every task in the run.
+- **A partly labelled dataset logs a WARNING** naming how many rows were dropped, because that run
+  is legitimate but is measuring a smaller suite than the file suggests. `coder-eval plan --split
+  <name>` shows the same thing before you spend anything.
 
 **Stage 2 — the samplers**, over whatever survived the filter:
 

docs/PLUGIN.md

@@ -49,7 +49,7 @@ Running a suite additionally needs credentials for whichever agent the tasks use
 | --- | --- |
 | `/coder-eval:init` | Scans the repository for what is worth evaluating (Claude Code skills, an MCP server, a CLI), reports the findings, then scaffolds a task directory with one real task. |
 | `/coder-eval:check-skill` | Builds and runs an activation suite for one of your skills — does the agent engage it when it should, and leave it alone when it shouldn't? |
-| `/coder-eval:optimize-skill` | Improves a skill along either of two tracks — the **description**, measured against an activation suite, or the **body**, measured against an outcome suite. Candidate edits become experiment variants; only what beats run-to-run noise and survives a held-out split is promoted. Walked end to end in [tutorial 08](tutorials/08-optimizing-a-skill.md) (description) and [tutorial 09](tutorials/09-optimizing-a-skill-body.md) (body). |
+| `/coder-eval:optimize-skill` | Improves a skill along either of two tracks — the **description**, measured against an activation suite, or the **body**, measured against an outcome suite. Candidate edits become experiment variants; only what beats run-to-run noise and survives a held-out split is promoted. Walked end to end in [tutorial 08](tutorials/08-optimizing-a-skill-description.md) (description) and [tutorial 09](tutorials/09-optimizing-a-skill-body.md) (body). |
 | `/coder-eval:task` | Turns a natural-language description into task YAML with criteria that check output *content*, validated through `coder-eval plan`. |
 | `/coder-eval:lint-tasks` | Reviews task YAML that already exists and reports, per task, criteria that cannot fail, prompts that leak the answer, fixtures with no cleanup and near-duplicates — each with a severity and a fix. Read-only. |
 | `/coder-eval:analyze` | Reads a finished run directory and writes `analysis.md`: systemic failure patterns, per-task findings, and concrete fixes. |
@@ -190,6 +190,11 @@ directories, so every file a skill reads travels with it under `reference/`:
   writes and `lint-tasks` applies to task files already on disk: could this task pass for
   the wrong reason, does it grade behavior or a self-report, do its fixtures reset and
   clean up.
+- `optimize-method.md` — the track-invariant method behind `optimize-skill`'s three stages:
+  the cost table, what each stage does and does not bound, why the activation and execution
+  gates use different machinery, and the paired-diff sign rule. Split out of `SKILL.md` for
+  the same reason as `task-rubric.md` — the method is identical on both tracks, so it lives
+  once and the skill keeps the procedure.
 - `cli-setup.md` — how the CLI-driving skills handle a missing `coder-eval`
   binary: offer the install, ask first, verify it worked.
 - `run-layout.md` — the on-disk run-directory contract `analyze` reads: what is *inside*

docs/TASK_DEFINITION_GUIDE.md

@@ -94,7 +94,7 @@ dataset:
 | `sample_per_stratum` | `null` | Stratified random sample: keep up to N rows per stratum. Overridden by CLI `--sample`. |
 | `stratify_field` | `"expected_skill"` | Row field whose value defines the stratum for `sample_per_stratum`. |
 | `sample_seed` | `null` | Seed for the stratified draw. Unset means the sample is **re-drawn every run**; set an integer to pin it. CLI `--sample` is separately fixed-seed and always reproducible. |
-| `split_field` | `"split"` | Row field naming the row's split (e.g. `train` / `test`). CLI `--split <name>` keeps only rows whose value here matches, **before** any sampling. A task whose rows never set this field is unaffected by `--split`. Splits are open strings. |
+| `split_field` | `"split"` | Row field naming the row's split (e.g. `train` / `test`). CLI `--split <name>` keeps only rows whose value here matches, **before** any sampling. A task whose rows never set this field is unaffected by `--split`; a task that sets it on *some* rows keeps the matching ones, drops the rest, and logs a WARNING naming the drop count; a task that sets it on every row but has none in the requested split aborts the run. Splits are open strings. |
 
 Full guide — row sources, substitution rules, sampling precedence, suite-level scoring, and worked
 examples: **[Bring Your Own Dataset](DATASETS.md)**.
@@ -663,7 +663,7 @@ This pattern is especially useful for A/B testing whether additional context imp
 
 ## Success Criteria
 
-Every task needs at least one success criterion. The framework supports 14 criterion types.
+Every task needs at least one success criterion. The framework supports 15 criterion types.
 
 ### Continuous Scoring
 
@@ -1272,7 +1272,11 @@ Like `skill_triggered`, this criterion emits a `ClassificationCriterionResult`,
 
 ### `skill_triggered`
 
-Binary classifier: **did the agent engage the target skill during the run?** Agent-agnostic — scans the run's `turn_records` for either signal: Claude's explicit `Skill` tool call whose `skill` parameter matches `skill_name` (namespace prefixes like `plugin:skill` are stripped, so `skill_name: uipath-agents` matches `Skill(skill="uipath-coded-agents:uipath-agents")`), or — for an agent with no `Skill` tool, e.g. Codex — a command that reads the skill's files off disk (a parameter contains `skills/<skill_name>/`, matching both the repo path and the `.agents/skills/` symlink).
+Binary classifier: **did the agent engage the target skill during the run?** Agent-agnostic — scans the run's `turn_records` for either signal: Claude's explicit `Skill` tool call whose `skill` parameter matches `skill_name` **and which succeeded** (namespace prefixes like `plugin:skill` are stripped, so `skill_name: uipath-agents` matches `Skill(skill="uipath-coded-agents:uipath-agents")`), or — for an agent with no `Skill` tool, e.g. Codex — a command that **successfully** reads the skill's files off disk (a parameter contains `skills/<skill_name>/`, matching both the repo path and the `.agents/skills/` symlink).
+
+> **Only a successful `Skill` call is engagement**, and the distinction matters more than it sounds. The common cause of a failed one is a skill carrying `disable-model-invocation: true`: the tool refuses the call outright (`cannot be used with Skill tool due to disable-model-invocation`), so the skill's body never loads and the agent continues on its own background knowledge — producing output plausible enough that nothing downstream looks wrong. Counting the attempt would report `yes` for a run the skill took no part in. The same reasoning excludes a call that is still in flight or was force-closed by a turn crash: for the `Skill` tool the body IS the tool result, so anything short of a delivered result loaded nothing.
+>
+> The file-read signal is gated too, but only where a failure proves nothing was read: a `Read`, `Glob` or `Grep` that errored (or has not resolved yet) names the path in its parameters while loading nothing, so it does not count. `Bash` is deliberately left ungated — `cat skills/x/SKILL.md | grep foo` exits non-zero *after* genuinely reading the file, which is exactly how an agent with no `Skill` tool engages a skill.
 
 Observed label is `"yes"` when either signal is found, else `"no"`. Expected label is `"yes"` iff `expected_skill == skill_name`. **Binary scoring:** `1.0` when observed matches expected, else `0.0`.
 
@@ -1288,7 +1292,7 @@ Observed label is `"yes"` when either signal is found, else `"no"`. Expected lab
 
 | Field | Default | Description |
 |-------|---------|-------------|
-| `skill_name` | *required* | The skill to detect — a `Skill` call whose `skill` parameter matches, or a file read under `skills/<skill_name>/` |
+| `skill_name` | *required* | The skill to detect — a **successful** `Skill` call whose `skill` parameter matches, or a **successful** file read under `skills/<skill_name>/` |
 | `expected_skill` | *required* | The row's expected skill (after `${row.*}` substitution); empty string `""` for negative rows where the skill should **not** fire |
 
 **Requires agent telemetry.** This criterion reads `turn_records`, so it only works against a real agent run (not a static check). With no turn records it reports `score=0.0` and an `error`.

docs/llms.txt

@@ -50,7 +50,7 @@ and A/B plumbing.
 - [05 · Comparing two models](https://coder-eval.com/docs/tutorials/05-comparing-models)
 - [06 · Docker isolation](https://coder-eval.com/docs/tutorials/06-use-docker-isolation)
 - [07 · Driving Coder Eval from Claude Code](https://coder-eval.com/docs/tutorials/07-plugin-in-claude-code)
-- [08 · Optimizing a Skill Description](https://coder-eval.com/docs/tutorials/08-optimizing-a-skill)
+- [08 · Optimizing a Skill Description](https://coder-eval.com/docs/tutorials/08-optimizing-a-skill-description)
 - [09 · Optimizing a Skill Body](https://coder-eval.com/docs/tutorials/09-optimizing-a-skill-body)
 <!-- docs-index:end:tutorials -->
 

docs/tutorials/04-writing-a-task.md

@@ -69,7 +69,7 @@ What each block does:
   (default 1.0) and `pass_threshold` (default 0.9). `run_command` here only
   checks the exit code; it can also match stdout (`expected_stdout`) or read a
   continuous score from stdout — see the
-  [Task Definition Guide](../TASK_DEFINITION_GUIDE.md) for all 14 criterion
+  [Task Definition Guide](../TASK_DEFINITION_GUIDE.md) for all 15 criterion
   types and every field.
 
 ## 2. Validate without spending tokens
@@ -101,7 +101,7 @@ created are preserved under `runs/latest/<variant>/<task>/<NN>/` (`task.json`,
 
 ## Where to go deeper
 
-- **All 14 criterion types, weights, thresholds** → [Task Definition Guide](../TASK_DEFINITION_GUIDE.md)
+- **All 15 criterion types, weights, thresholds** → [Task Definition Guide](../TASK_DEFINITION_GUIDE.md)
 - **Stop a run early once the key criteria are decided** (an opt-in `stop_early:` block on a criterion; `run_limits.stop_early: false` is the run-level kill switch) → [Task Definition Guide → `stop_early`](../TASK_DEFINITION_GUIDE.md#stop_early-opt-in-early-stop)
 - **Fan one task out over a dataset of rows** → [Bring Your Own Dataset](../DATASETS.md)
 - **Full CLI & config reference** → [User Guide](../USER_GUIDE.md)

docs/tutorials/07-plugin-in-claude-code.md

@@ -41,12 +41,18 @@ Code prompt, not in your shell:
 /plugin install coder-eval@coder-eval
 ```
 
-Verify by typing `/coder-eval:`. You should see six commands: `init`,
-`check-skill`, `task`, `lint-tasks`, `analyze`, `ci`. Three of them — `init`,
-`check-skill` and `task` — drive the same `coder-eval` CLI you would type by hand,
-so what they write is a normal file you can commit, diff and run in CI. The other
-three never invoke it: `analyze` reads a finished run directory, `ci` writes a
-workflow, and `lint-tasks` only reads task files and reports.
+Verify by typing `/coder-eval:`. You should see seven commands: `init`,
+`check-skill`, `task`, `optimize-skill`, `lint-tasks`, `analyze`, `ci`. Four of them
+— `init`, `check-skill`, `task` and `optimize-skill` — drive the same `coder-eval`
+CLI you would type by hand, so what they write is a normal file you can commit, diff
+and run in CI. The remaining three never invoke it: `analyze` reads a finished run
+directory, `ci` writes a workflow, and `lint-tasks` only reads task files and reports.
+
+Three of the seven are deliberately **not** model-invokable — `init`, `ci` and
+`optimize-skill` carry `disable-model-invocation: true`, so you reach them by typing the
+command rather than by describing the job. That matters most for `optimize-skill`: it
+spends real money across a baseline and three A/B stages, and is never something to start
+because a message happened to mention a skill's wording.
 
 ## 2. Scaffold a suite
 
@@ -170,6 +176,13 @@ remove it, `/plugin uninstall`.
 - `/coder-eval:ci` emits the CI workflow from [Tutorial 02](02-ci-pipeline.md) for
   you — least-privilege by default, and it provides the agent runtime (Node plus
   the Claude CLI) that the Action deliberately does not install.
+- `/coder-eval:optimize-skill` is the follow-on from step 5. `check-skill` tells you
+  *whether* a skill fires; this A/B-tests edits to its description — or to its body — as
+  experiment variants, and promotes one only when it beats run-to-run noise on rows it was
+  never fitted to. It is the expensive one: a baseline plus three stages of agent runs, so
+  it states the projected count and asks before each. Walked end to end in
+  [Tutorial 08](08-optimizing-a-skill-description.md) (description) and
+  [Tutorial 09](09-optimizing-a-skill-body.md) (body).
 - [Claude Code plugin](../PLUGIN.md) — the reference page: every skill, what ships
   in the plugin, and the activation-budget mechanics in full.
 - [Writing a task](04-writing-a-task.md) — the same authoring loop by hand, worth