UiPath · uipreliga · Aug 21, 2026
@@ -464,9 +464,13 @@ with the two `action.yml` items above — one considered change to the action's
   and suite gates — passes. Verified directly: `coder-eval run <suite> --split holdou`
   prints one yellow "1 task file(s) skipped" line and exits 0. This is pre-existing, but
   `--split` makes it reachable by a one-character CLI typo rather than a broken file, and
-  the whole point of a holdout confirmation is that you trust its verdict. Not guarded, and
+  the whole point of a test confirmation is that you trust its verdict. Not guarded, and
   not a five-minute fix: making an all-skipped run non-green changes exit semantics for
   every skipped-task path (including deliberate `skip: true` suites and tag filters that
   match nothing), so it needs a decision about which of those should be fatal, plus tests
   per case. A narrower option is to fail only when a CLI *selector* (`--split`, `--tags`)
   eliminated everything, since that is unambiguously a user error rather than repo state.
+
+- [ ] **Semantic answer-leak in a task prompt** — a prompt that describes the graded behaviour in *different words* ("list the paths explicitly rather than with a recursive wildcard" while grading an explicit glob) scores well whether or not the behaviour happened, and in an A/B an arm that deleted the rule still passes. CE061 catches only the verbatim form; the semantic form needs an LLM judge or a `lint-tasks` pass over this repo's own `tasks/`, neither of which is cheap or deterministic. — caught in the final review of c/2026-08-13-optimize-skill-fixes.md, where 4 of 10 rows in a shipped worked example had it.
+- [ ] **A doc claim that contradicts merge semantics** — `optimize-skill` told users to declare `allowed_tools` in an experiment's `defaults: agent:`, which is a silent no-op because those fields merge by `replace` and the task layer outranks experiment defaults. Detecting "this prose recommends a config location that the merge order makes ineffective" would need the rule to model the layer stack against prose, which no existing rule shape supports. — caught in the final review of c/2026-08-13-optimize-skill-fixes.md.
+- [x] **`_normalized()` not used by every prose sensor** — CLOSED: all 9 sites converted, and `test_no_sensor_inlines_the_normalization_idiom` now forbids the raw form. Original note: — 8 sensors in `tests/test_custom_lint.py` still inline `" ".join(path.read_text().split())`, so a future one copied from the wrong neighbour is defeated by a line wrap (the bug that let a stale skill count ship past 91 green tests). A rule forbidding the raw idiom in that file is easy; the conversion sweep was out of scope. — caught in the final review of c/2026-08-13-optimize-skill-fixes.md.
@@ -72,3 +72,8 @@ refs/
 
 # Derived pyright config for the CE036 contract engine (tests/lint/pyright_config.py)
 .pyright-tests.json
+
+# /coder-eval:optimize-skill working tree: candidate skill snapshots,
+# per-stage experiment files and the round ledger. A snapshot is a full plugin
+# root copied per arm, so a round writes several megabytes of duplicated skills.
+.optimize-skill/
@@ -355,7 +355,7 @@ if any listed metric is below its minimum.
 | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ |
 | `-e, --experiment <path\|name>`                                                                                                                                   | Experiment YAML. Bare name → `experiments/<name>.yaml`.                                                |
 | `--sample N`                                                                                                                                                      | For dataset-backed tasks, use a fixed-seed random N-row sample (reproducible, unbiased across paths; cheap smoke test). |
-| `--split NAME`                                                                                                                                                    | For dataset-backed tasks, keep only rows whose `dataset.split_field` value matches (e.g. `tune` / `holdout`). Applied before `--sample`. Unlabelled tasks unaffected. |
+| `--split NAME`                                                                                                                                                    | For dataset-backed tasks, keep only rows whose `dataset.split_field` value matches (e.g. `train` / `test`). Applied before `--sample`. Unlabelled tasks unaffected. |
 | `--repeats N`                                                                                                                                                     | Run each `(task, variant)` N times; overrides YAML `repeats`.                                          |
 | `--driver tempdir\|docker`                                                                                                                                        | Override sandbox driver for all tasks.                                                                 |
 | `-j, --max-parallel N`                                                                                                                                            | Run up to N tasks concurrently.                                                                        |

@@ -124,7 +124,7 @@ Load-time errors, with their message shapes:
 | Two rows share an id | `Duplicate dataset row id for task '<task_id>': '<x>'` |
 | Both/neither `rows` and `paths` | `Dataset must specify either 'paths' or 'rows'` / `... only one of ...` |
 | `paths: []` | `Dataset.paths must be a non-empty list` |
-| `--split X` on a labelled dataset with no `X` rows | `Dataset for task '<task_id>' has no rows in split 'X' (split_field='split'); labelled splits present: ['holdout', 'tune']` |
+| `--split X` on a labelled dataset with no `X` rows | `Dataset for task '<task_id>' has no rows in split 'X' (split_field='split'); labelled splits present: ['test', 'train']` |
 
 ## Selecting a subset
 
@@ -144,8 +144,8 @@ Three behaviours are worth knowing before you rely on it:
 - **A task whose rows carry no split label at all passes through unfiltered.** `--split` is global to
   the invocation, so an unlabelled dataset sitting beside a labelled one in the same run must not
   fail. A row counts as unlabelled when the field is absent, `null`, or `""`.
-- **Partial labelling drops the unlabelled rows.** If only some rows carry a label, `--split tune`
-  keeps just the `tune` rows — the unlabelled ones are excluded rather than folded in. That is the
+- **Partial labelling drops the unlabelled rows.** If only some rows carry a label, `--split train`
+  keeps just the `train` rows — the unlabelled ones are excluded rather than folded in. That is the
   safe direction (an unlabelled row never leaks into a named split), but during an incremental
   migration it silently *shrinks* the suite, which moves the aggregate metrics `suite_thresholds`
   gates on. Finish labelling before you compare two runs.
@@ -182,31 +182,31 @@ Three behaviours are worth knowing before you rely on it:
     coverage over time matters more than run-to-run comparability. Note the contrast with
     `--sample N`, which is fixed-seed and reproducible by default.
 
-### Tune and holdout splits
+### Train and test splits
 
 Label each row with a split and you can develop against one half and confirm on the other, which is
-what keeps a measured improvement from being an artifact of the rows you tuned on:
+what keeps a measured improvement from being an artifact of the rows you trained on:
 
 ```jsonl
-{"id": "pos-1", "prompt": "review my task files", "expected_skill": "lint-tasks", "split": "tune"}
-{"id": "pos-2", "prompt": "are my evals any good?", "expected_skill": "lint-tasks", "split": "holdout"}
+{"id": "pos-1", "prompt": "review my task files", "expected_skill": "lint-tasks", "split": "train"}
+{"id": "pos-2", "prompt": "are my evals any good?", "expected_skill": "lint-tasks", "split": "test"}
 ```
 
 ```bash
-coder-eval run tasks/skills/activation.yaml --split tune      # iterate here
-coder-eval run tasks/skills/activation.yaml --split holdout   # confirm here, once
+coder-eval run tasks/skills/activation.yaml --split train      # iterate here
+coder-eval run tasks/skills/activation.yaml --split test   # confirm here, once
 ```
 
 **The filter runs before either sampler, and that ordering is load-bearing.** Sampling first would
 leave an unpredictable — possibly zero — number of rows per split, so the two arms of the comparison
-would no longer be the same size or the same rows. Filter-then-sample means `--split tune --sample 8`
-is always drawn from the tune rows alone — at most eight of them, and all of them if `tune` holds
+would no longer be the same size or the same rows. Filter-then-sample means `--split train --sample 8`
+is always drawn from the train rows alone — at most eight of them, and all of them if `train` holds
 fewer than eight.
 
 Two consequences worth planning for. A split **halves each side of the suite**, so a dataset sized
 for a single one-shot measurement is undersized once split — budget roughly double the rows you
-would otherwise want. And a holdout is only worth what its independence buys: consult it to confirm
-a decision already made on `tune`, not to choose between candidates, or it becomes a second tune set.
+would otherwise want. And a test split is only worth what its independence buys: consult it to confirm
+a decision already made on `train`, not to choose between candidates, or it becomes a second train set.
 
 ## Suite-level scoring
 

@@ -31,9 +31,9 @@ references, not packages, so the `coder-eval` binary is a separate step:
 uv tool install coder-eval    # or: pip install coder-eval
 ```
 
-You do not have to do it in advance. `init`, `task` and `check-skill` — the three
-skills that shell out to the CLI — check `coder-eval --version` before doing any
-work and, if it is missing, **offer to install it and ask first**. They never
+You do not have to do it in advance. `init`, `task`, `check-skill` and
+`optimize-skill` — the four that shell out to the CLI — check `coder-eval --version`
+before doing any work and, if it is missing, **offer to install it and ask first**. They never
 install unprompted: that writes outside your repository, so it is your call, and
 they verify the install worked before continuing. `analyze` and `ci` do not invoke
 the CLI, and `lint-tasks` needs neither the CLI nor credentials — it only reads
@@ -49,7 +49,7 @@ Running a suite additionally needs credentials for whichever agent the tasks use
 | --- | --- |
 | `/coder-eval:init` | Scans the repository for what is worth evaluating (Claude Code skills, an MCP server, a CLI), reports the findings, then scaffolds a task directory with one real task. |
 | `/coder-eval:check-skill` | Builds and runs an activation suite for one of your skills — does the agent engage it when it should, and leave it alone when it shouldn't? |
-| `/coder-eval:optimize-skill` | Takes an activation suite's confusion matrix and proposes description rewrites, A/B tests them as experiment variants, and promotes only what beats run-to-run noise and survives a held-out split. |
+| `/coder-eval:optimize-skill` | Improves a skill along either of two tracks — the **description**, measured against an activation suite, or the **body**, measured against an outcome suite. Candidate edits become experiment variants; only what beats run-to-run noise and survives a held-out split is promoted. Walked end to end in [tutorial 08](tutorials/08-optimizing-a-skill.md) (description) and [tutorial 09](tutorials/09-optimizing-a-skill-body.md) (body). |
 | `/coder-eval:task` | Turns a natural-language description into task YAML with criteria that check output *content*, validated through `coder-eval plan`. |
 | `/coder-eval:lint-tasks` | Reviews task YAML that already exists and reports, per task, criteria that cannot fail, prompts that leak the answer, fixtures with no cleanup and near-duplicates — each with a severity and a fix. Read-only. |
 | `/coder-eval:analyze` | Reads a finished run directory and writes `analysis.md`: systemic failure patterns, per-task findings, and concrete fixes. |
@@ -195,10 +195,12 @@ directories, so every file a skill reads travels with it under `reference/`:
 - `run-layout.md` — the on-disk run-directory contract `analyze` reads: what is *inside*
   a run directory.
 - `repo-layout.md` — how a skill finds *where* your eval tree is, by globbing for
-  `task_id:` files and `run.json` rather than assuming `tasks/` and `runs/`. All six
+  `task_id:` files and `run.json` rather than assuming `tasks/` and `runs/`. All seven
   skills read it, which is what lets them work in a repository that names or nests the
   tree differently.
-- `templates/` — the canonical activation suite `check-skill` copies.
+- `templates/` — the two canonical suites the skills copy: the activation suite
+  `check-skill` writes, and the outcome suite `optimize-skill`'s execution track starts
+  from.
 
 ## Related
 

@@ -94,7 +94,7 @@ dataset:
 | `sample_per_stratum` | `null` | Stratified random sample: keep up to N rows per stratum. Overridden by CLI `--sample`. |
 | `stratify_field` | `"expected_skill"` | Row field whose value defines the stratum for `sample_per_stratum`. |
 | `sample_seed` | `null` | Seed for the stratified draw. Unset means the sample is **re-drawn every run**; set an integer to pin it. CLI `--sample` is separately fixed-seed and always reproducible. |
-| `split_field` | `"split"` | Row field naming the row's split (e.g. `tune` / `holdout`). CLI `--split <name>` keeps only rows whose value here matches, **before** any sampling. A task whose rows never set this field is unaffected by `--split`. Splits are open strings. |
+| `split_field` | `"split"` | Row field naming the row's split (e.g. `train` / `test`). CLI `--split <name>` keeps only rows whose value here matches, **before** any sampling. A task whose rows never set this field is unaffected by `--split`. Splits are open strings. |
 
 Full guide — row sources, substitution rules, sampling precedence, suite-level scoring, and worked
 examples: **[Bring Your Own Dataset](DATASETS.md)**.

@@ -44,7 +44,7 @@ coder-eval run tasks/hello_date.yaml --stream full  # live LLM output
 | `--resume` | Resume an interrupted run: skip tasks already finalized in `--run-dir` and run the rest, folding prior results into `run.json`. Requires `--run-dir`. A task with *any* final status (incl. FAILED/ERROR) counts as finalized, so resume does **not** retry failures — delete a task's `task.json` to force a re-run. A config mismatch is warned, not refused. |
 | `--sample N` | For dataset-backed tasks, run a fixed-seed random N-row sample (reproducible; cheap smoke test). See [Bring Your Own Dataset](DATASETS.md). |
 | `--sample-per-stratum N` | For dataset-backed tasks, keep up to N rows per stratum (`stratify_field`). Overridden by `--sample`. Nondeterministic unless `dataset.sample_seed` is set — see [Bring Your Own Dataset](DATASETS.md). |
-| `--split NAME` | For dataset-backed tasks, keep only rows whose `dataset.split_field` value (default field: `split`) matches — e.g. `--split tune` / `--split holdout`. Applied **before** `--sample` / `--sample-per-stratum`. Tasks whose rows carry no split label are unaffected. See [Bring Your Own Dataset](DATASETS.md). |
+| `--split NAME` | For dataset-backed tasks, keep only rows whose `dataset.split_field` value (default field: `split`) matches — e.g. `--split train` / `--split test`. Applied **before** `--sample` / `--sample-per-stratum`. Tasks whose rows carry no split label are unaffected. See [Bring Your Own Dataset](DATASETS.md). |
 | `--include-skipped` | Also run tasks marked `skip: true` in their YAML (off by default so CI keeps excluding them). |
 | `--exclude-tags` | Skip tasks matching any of these tags (comma-separated) |
 | `--tags, -t` | Only run tasks matching any of these tags (comma-separated) |

@@ -50,7 +50,8 @@ and A/B plumbing.
 - [05 · Comparing two models](https://coder-eval.com/docs/tutorials/05-comparing-models)
 - [06 · Docker isolation](https://coder-eval.com/docs/tutorials/06-use-docker-isolation)
 - [07 · Driving Coder Eval from Claude Code](https://coder-eval.com/docs/tutorials/07-plugin-in-claude-code)
-- [08 · Optimizing a skill description](https://coder-eval.com/docs/tutorials/08-optimizing-a-skill)
+- [08 · Optimizing a Skill Description](https://coder-eval.com/docs/tutorials/08-optimizing-a-skill)
+- [09 · Optimizing a Skill Body](https://coder-eval.com/docs/tutorials/09-optimizing-a-skill-body)
 <!-- docs-index:end:tutorials -->
 
 ## Source