PRD-002 and PRD-002a: a rederivation eval, its dev baseline, and a check that it catches a missing block

AGENTS.md

@@ -19,6 +19,8 @@ uv run explainer coldread <slug> --run        # first-viewing read by a fresh ag
 uv run explainer findings --stats             # how many review findings authors adopt, per tool
 uv run explainer eval chat                    # chat answers under each system prompt, graded blind
 uv run explainer eval artifact --split dev    # held-out subjects built under each condition, read and graded blind
+uv run explainer eval rederive --builds <run> # a fresh reader works out a result each explanation never states, graded blind
+uv run explainer eval cuts && uv run explainer eval controls --out <folder> # does rederivation notice a cut building block? (PRD-002a)
 uv run explainer render <slug> --draft        # fast silent layout pass
 uv run pytest -q                              # toolkit tests
 ```

CHANGELOG.md

@@ -1,5 +1,15 @@
 # Changelog
 
+## Unreleased
+
+Rederivation: can a reader work out what an explanation never states?
+
+- **A rederivation target for every subject** in `evals/artifact/questions.yaml`: a question the explanation was never asked to answer, the expected steps (building blocks a good explanation gives, and derived steps), and the result. A fresh agent checked each one (`explainer eval targets`, `evals/artifact/rederive-checks.json`), and all 30 were frozen before the first run.
+- **The rederivation stage** (`explainer eval rederive --builds <run>`; `eval artifact` runs it too). A fresh reader works the target out from the renderings, naming the part of the explanation each step rests on; a grader marks each expected step supported, unsupported, missing, or stated, blind, in three passes. Stated steps are left out of the score. [docs/evals.md](docs/evals.md#rederivation).
+- [What it found](docs/evals.md#what-the-dev-baseline-found) on the v0.8 dev builds: it is also near its ceiling (90–100% of steps supported). Under the rule fixed before the run, one pair separates, none > published, by about two steps of 38, from one subject where the published explanation lacks part of a building block (when the Sun's and Moon's tides add), a gap the quiz also found. Reported with it: one target that overclaims (float-sum), explanations that state the derived steps in 3 of 10 subjects, and a partial building block graded inconsistently.
+- `explainer eval human` and `evals/artifact/human-check.md`: a person works out a target, graded with the same rubric.
+- **Positive controls for rederivation (PRD-002a)** (`explainer eval cuts`, `explainer eval controls`). One building block was cut from each eligible dev subject's `principles` explanation, the cuts were frozen, and the unchanged stage was run on them. [Verdict](docs/evals.md#does-the-test-catch-a-missing-block): trusted, at the floor. The cut block was caught in 4 of 5 subjects (bloom-filter could not be cut cleanly). Float-sum was the miss: its reader rebuilt the block from what the cut left plus its own knowledge, and the grader credited it. In heat-pump, a block rebuilt the same way was not credited, and the verdict depends on that. Only 6 of the 10 dev targets have an eligible block: one the reader cannot get from the question or from background, and that the intact build was reliably credited with.
+
 ## 0.8.0 — 2026-10-19
 
 Measure the toolkit, not the examples.

CLAUDE.md

@@ -19,6 +19,8 @@ uv run explainer coldread <slug> --run        # first-viewing read by a fresh ag
 uv run explainer findings --stats             # how many review findings authors adopt, per tool
 uv run explainer eval chat                    # chat answers under each system prompt, graded blind (needs the claude CLI)
 uv run explainer eval artifact --split dev     # held-out subjects built under each condition, read and graded blind
+uv run explainer eval rederive --builds <run>  # a fresh reader works out a result each explanation never states, graded blind
+uv run explainer eval cuts && uv run explainer eval controls --out <folder>  # does rederivation notice a cut building block? (PRD-002a)
 uv run explainer render <slug> --draft        # fast silent layout pass
 uv run explainer render <slug> --review       # 1080p60, voice, captions, chapters, review sheet
 uv run pytest -q                              # toolkit tests

ROADMAP.md

@@ -221,6 +221,7 @@ Rounds 2 and 3 also produced general rules: names match pictures (call it an L o
 - The examples are frozen as reference implementations and regression fixtures: `explainer check` must pass; no more review rounds.
 - `evals/artifact/questions.yaml`: 30 held-out subjects (10 dev, 20 test), none an example or a chat-eval question, each with a fixed quiz of four questions; the last is a prediction about a case the explanation need not mention.
 - `explainer eval artifact`: per subject and condition, a fresh agent builds the explainer (none, the principles, or the explain skill at the quick or published tier), a second reads only the renderings and answers the quiz with a quote for each answer (an answer whose quote is not in the renderings scores 0), and a third grades blind. [docs/evals.md](docs/evals.md#the-artifact-eval).
+- Rederivation (PRD-002): each subject also has a frozen, model-checked target, a result its explanation was never asked to state. A fresh reader works it out from the renderings, naming the part each step rests on; a grader marks each expected step blind, in three passes. `eval artifact` runs it; `eval rederive --builds <run>` runs it on saved builds. [docs/evals.md](docs/evals.md#rederivation).
 
 **Rules.**
 
@@ -233,12 +234,16 @@ Rounds 2 and 3 also produced general rules: names match pictures (call it an L o
 - The quiz is at its ceiling (94–100% in every condition, with a strong reader and with Haiku): a model reader cannot measure the difference. The same holds for the published tier's blind test.
 - The principles alone cut length 40% and rank first on effort in every grading pass, at the lowest cost. The quick tier is shortest but ranks below them and makes the most errors; the published tier costs 13× as much for no measured gain.
 - The first run was invalid: builders inside the repo loaded its CLAUDE.md in every condition. Builds now run outside every project.
+- Rederivation is also near its ceiling (90–100% of steps supported). One pair separates under the rule fixed before the run, none > published, by about two of 38 steps, from one subject (two-tides) where the published explanation lacks part of a building block (when the Sun's and Moon's tides add), a gap the quiz also found. Two target defects and three subjects whose explanations state the derived steps are reported with it ([docs/evals.md](docs/evals.md#what-the-dev-baseline-found)).
 
 **Next.**
 
 - A person ranks a sample (principles, quick, published on three subjects); the grader's ranking is used only if it agrees.
 - If it agrees: decide what the skill keeps. Candidates: the model file and the reviews only for the published tier, or the reviews dropped where they do not change the result.
 - A quiz that discriminates: harder subjects, or a reader limited in another way (time, words read), since a weaker model was not enough.
+- Open: whether PRD-003 adopts rederivation as its measure. With a strong model reader it mostly re-detects missing building blocks. Options: a weaker or constrained reader, harder targets with more derived steps (and a checker that tests "not stated" against real explanations), or human readers (`evals/artifact/human-check.md`). Recommendation: get two or three human checks before adopting it.
+- PRD-002a positive controls: **trusted, at the floor.** A cut building block was caught in 4 of 5 subjects ([docs/evals.md](docs/evals.md#does-the-test-catch-a-missing-block)). PRD-003 may use rederivation, but only on targets with an eligible block. The verdict depends on the grader withholding credit from a block the reader rebuilt (heat-pump) while it credited another (float-sum, the miss). Heat-pump also kept a derived step supported while the block it rests on was marked unsupported; making derived credit depend on the blocks would remove that inconsistency.
+- Before PRD-003 relies on the test-split targets, classify the 20 of them by the PRD-002a eligibility rule.
 - `plugin/skills/explain/references/html.md` says "the seven understanding questions from principles.md §7", but §7 has eight. Fix it with the principles rewrite, because skill text is measured.
 - The blind test (`explainer_kit/review.py`) does not ask "Could anything be cut with no loss?" Decide whether it should when the objective becomes least *wasted* effort.
 

docs/cli.md

@@ -22,7 +22,12 @@ With the plugin installed, run `explainer <command>` from any project. In the Ex
 | `explainer say "<text>" [--voice V] [--speed S]` | Synthesize one line and play it. Use it to choose a voice or test a lexicon spelling. |
 | `explainer frames <slug> [--draft] [--every N]` | Rebuild the contact sheet from an existing render. |
 | `explainer eval chat [--conditions name=source ...]` | Answer the questions in `evals/chat/questions.yaml` under several system prompts with fresh `claude -p` calls, grade them blind, and write a report. See [Evals](evals.md). |
-| `explainer eval artifact [--split dev\|test\|all] [--only id ...] [--conditions name=source ...]` | Build an explainer for each held-out subject in `evals/artifact/questions.yaml` under each condition (`none`, `principles`, `git:<rev>`, a file, `quick`, `published`), have a fresh reader answer its quiz from the renderings alone, grade blind, and write a report. A rerun into the same `--out` resumes. See [Evals](evals.md#the-artifact-eval). |
+| `explainer eval artifact [--split dev\|test\|all] [--only id ...] [--conditions name=source ...]` | Build an explainer for each held-out subject in `evals/artifact/questions.yaml` under each condition (`none`, `principles`, `git:<rev>`, a file, `quick`, `published`), have a fresh reader answer its quiz from the renderings alone, grade blind, and write a report; then run the rederivation stage on the same builds. A rerun into the same `--out` resumes. See [Evals](evals.md#the-artifact-eval). |
+| `explainer eval rederive --builds <run> [--only id ...] [--out folder]` | The rederivation stage alone, on the builds of an earlier artifact run (only read, never changed): in three passes, a fresh reader works out each subject's target question from the renderings and names the part each step rests on, and a grader marks each expected step blind. Writes `rederive.md` and `rederive-scores.json`. See [Evals](evals.md#rederivation). |
+| `explainer eval targets [--only id ...]` | Check each rederivation target in `evals/artifact/questions.yaml` with a fresh agent, and record the verdicts in `evals/artifact/rederive-checks.json`. Only targets changed since their last check are checked. |
+| `explainer eval human --builds <run> --chain <file>` | Grade a person's working on a rederivation target with the same rubric as the model readers. See [`evals/artifact/human-check.md`](../evals/artifact/human-check.md). |
+| `explainer eval cuts [--only id ...] [--redo] [--out folder]` | Positive controls for the rederivation stage (PRD-002a): for each subject in `evals/artifact/controls.yaml`, cut one building block from its saved `principles` explanation with a fresh agent, check the cut with another, and keep the diff. The source builds are only read. A rerun skips finished cuts; `--redo` makes one more attempt (at most three in all). It refuses a folder that `eval controls` has run on. Default folder: `evals/artifact/results/<date>-dev-controls`. |
+| `explainer eval controls --out <folder>` | Run the rederivation stage on the cut copies in a folder `eval cuts` made, reusing the baseline's reader chains for the uncut conditions, check uncaught cuts for leaks, and write `controls.md` and `controls.json` with the verdict under the bar in `controls.yaml`. See [Evals](evals.md#rederivation). |
 
 ## `check`