diff --git a/AGENTS.md b/AGENTS.md index 63d9db5..96d1299 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -19,6 +19,8 @@ uv run explainer coldread --run # first-viewing read by a fresh ag uv run explainer findings --stats # how many review findings authors adopt, per tool uv run explainer eval chat # chat answers under each system prompt, graded blind uv run explainer eval artifact --split dev # held-out subjects built under each condition, read and graded blind +uv run explainer eval rederive --builds # a fresh reader works out a result each explanation never states, graded blind +uv run explainer eval cuts && uv run explainer eval controls --out # does rederivation notice a cut building block? (PRD-002a) uv run explainer render --draft # fast silent layout pass uv run pytest -q # toolkit tests ``` diff --git a/CHANGELOG.md b/CHANGELOG.md index bad7aa0..d50e75f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,15 @@ # Changelog +## Unreleased + +Rederivation: can a reader work out what an explanation never states? + +- **A rederivation target for every subject** in `evals/artifact/questions.yaml`: a question the explanation was never asked to answer, the expected steps (building blocks a good explanation gives, and derived steps), and the result. A fresh agent checked each one (`explainer eval targets`, `evals/artifact/rederive-checks.json`), and all 30 were frozen before the first run. +- **The rederivation stage** (`explainer eval rederive --builds `; `eval artifact` runs it too). A fresh reader works the target out from the renderings, naming the part of the explanation each step rests on; a grader marks each expected step supported, unsupported, missing, or stated, blind, in three passes. Stated steps are left out of the score. [docs/evals.md](docs/evals.md#rederivation). +- [What it found](docs/evals.md#what-the-dev-baseline-found) on the v0.8 dev builds: it is also near its ceiling (90–100% of steps supported). Under the rule fixed before the run, one pair separates, none > published, by about two steps of 38, from one subject where the published explanation lacks part of a building block (when the Sun's and Moon's tides add), a gap the quiz also found. Reported with it: one target that overclaims (float-sum), explanations that state the derived steps in 3 of 10 subjects, and a partial building block graded inconsistently. +- `explainer eval human` and `evals/artifact/human-check.md`: a person works out a target, graded with the same rubric. +- **Positive controls for rederivation (PRD-002a)** (`explainer eval cuts`, `explainer eval controls`). One building block was cut from each eligible dev subject's `principles` explanation, the cuts were frozen, and the unchanged stage was run on them. [Verdict](docs/evals.md#does-the-test-catch-a-missing-block): trusted, at the floor. The cut block was caught in 4 of 5 subjects (bloom-filter could not be cut cleanly). Float-sum was the miss: its reader rebuilt the block from what the cut left plus its own knowledge, and the grader credited it. In heat-pump, a block rebuilt the same way was not credited, and the verdict depends on that. Only 6 of the 10 dev targets have an eligible block: one the reader cannot get from the question or from background, and that the intact build was reliably credited with. + ## 0.8.0 — 2026-10-19 Measure the toolkit, not the examples. diff --git a/CLAUDE.md b/CLAUDE.md index b20121c..828374e 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -19,6 +19,8 @@ uv run explainer coldread --run # first-viewing read by a fresh ag uv run explainer findings --stats # how many review findings authors adopt, per tool uv run explainer eval chat # chat answers under each system prompt, graded blind (needs the claude CLI) uv run explainer eval artifact --split dev # held-out subjects built under each condition, read and graded blind +uv run explainer eval rederive --builds # a fresh reader works out a result each explanation never states, graded blind +uv run explainer eval cuts && uv run explainer eval controls --out # does rederivation notice a cut building block? (PRD-002a) uv run explainer render --draft # fast silent layout pass uv run explainer render --review # 1080p60, voice, captions, chapters, review sheet uv run pytest -q # toolkit tests diff --git a/ROADMAP.md b/ROADMAP.md index df5b402..14102d0 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -221,6 +221,7 @@ Rounds 2 and 3 also produced general rules: names match pictures (call it an L o - The examples are frozen as reference implementations and regression fixtures: `explainer check` must pass; no more review rounds. - `evals/artifact/questions.yaml`: 30 held-out subjects (10 dev, 20 test), none an example or a chat-eval question, each with a fixed quiz of four questions; the last is a prediction about a case the explanation need not mention. - `explainer eval artifact`: per subject and condition, a fresh agent builds the explainer (none, the principles, or the explain skill at the quick or published tier), a second reads only the renderings and answers the quiz with a quote for each answer (an answer whose quote is not in the renderings scores 0), and a third grades blind. [docs/evals.md](docs/evals.md#the-artifact-eval). +- Rederivation (PRD-002): each subject also has a frozen, model-checked target, a result its explanation was never asked to state. A fresh reader works it out from the renderings, naming the part each step rests on; a grader marks each expected step blind, in three passes. `eval artifact` runs it; `eval rederive --builds ` runs it on saved builds. [docs/evals.md](docs/evals.md#rederivation). **Rules.** @@ -233,12 +234,16 @@ Rounds 2 and 3 also produced general rules: names match pictures (call it an L o - The quiz is at its ceiling (94–100% in every condition, with a strong reader and with Haiku): a model reader cannot measure the difference. The same holds for the published tier's blind test. - The principles alone cut length 40% and rank first on effort in every grading pass, at the lowest cost. The quick tier is shortest but ranks below them and makes the most errors; the published tier costs 13× as much for no measured gain. - The first run was invalid: builders inside the repo loaded its CLAUDE.md in every condition. Builds now run outside every project. +- Rederivation is also near its ceiling (90–100% of steps supported). One pair separates under the rule fixed before the run, none > published, by about two of 38 steps, from one subject (two-tides) where the published explanation lacks part of a building block (when the Sun's and Moon's tides add), a gap the quiz also found. Two target defects and three subjects whose explanations state the derived steps are reported with it ([docs/evals.md](docs/evals.md#what-the-dev-baseline-found)). **Next.** - A person ranks a sample (principles, quick, published on three subjects); the grader's ranking is used only if it agrees. - If it agrees: decide what the skill keeps. Candidates: the model file and the reviews only for the published tier, or the reviews dropped where they do not change the result. - A quiz that discriminates: harder subjects, or a reader limited in another way (time, words read), since a weaker model was not enough. +- Open: whether PRD-003 adopts rederivation as its measure. With a strong model reader it mostly re-detects missing building blocks. Options: a weaker or constrained reader, harder targets with more derived steps (and a checker that tests "not stated" against real explanations), or human readers (`evals/artifact/human-check.md`). Recommendation: get two or three human checks before adopting it. +- PRD-002a positive controls: **trusted, at the floor.** A cut building block was caught in 4 of 5 subjects ([docs/evals.md](docs/evals.md#does-the-test-catch-a-missing-block)). PRD-003 may use rederivation, but only on targets with an eligible block. The verdict depends on the grader withholding credit from a block the reader rebuilt (heat-pump) while it credited another (float-sum, the miss). Heat-pump also kept a derived step supported while the block it rests on was marked unsupported; making derived credit depend on the blocks would remove that inconsistency. +- Before PRD-003 relies on the test-split targets, classify the 20 of them by the PRD-002a eligibility rule. - `plugin/skills/explain/references/html.md` says "the seven understanding questions from principles.md §7", but §7 has eight. Fix it with the principles rewrite, because skill text is measured. - The blind test (`explainer_kit/review.py`) does not ask "Could anything be cut with no loss?" Decide whether it should when the objective becomes least *wasted* effort. diff --git a/docs/cli.md b/docs/cli.md index 9ad7f7e..cb736bd 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -22,7 +22,12 @@ With the plugin installed, run `explainer ` from any project. In the Ex | `explainer say "" [--voice V] [--speed S]` | Synthesize one line and play it. Use it to choose a voice or test a lexicon spelling. | | `explainer frames [--draft] [--every N]` | Rebuild the contact sheet from an existing render. | | `explainer eval chat [--conditions name=source ...]` | Answer the questions in `evals/chat/questions.yaml` under several system prompts with fresh `claude -p` calls, grade them blind, and write a report. See [Evals](evals.md). | -| `explainer eval artifact [--split dev\|test\|all] [--only id ...] [--conditions name=source ...]` | Build an explainer for each held-out subject in `evals/artifact/questions.yaml` under each condition (`none`, `principles`, `git:`, a file, `quick`, `published`), have a fresh reader answer its quiz from the renderings alone, grade blind, and write a report. A rerun into the same `--out` resumes. See [Evals](evals.md#the-artifact-eval). | +| `explainer eval artifact [--split dev\|test\|all] [--only id ...] [--conditions name=source ...]` | Build an explainer for each held-out subject in `evals/artifact/questions.yaml` under each condition (`none`, `principles`, `git:`, a file, `quick`, `published`), have a fresh reader answer its quiz from the renderings alone, grade blind, and write a report; then run the rederivation stage on the same builds. A rerun into the same `--out` resumes. See [Evals](evals.md#the-artifact-eval). | +| `explainer eval rederive --builds [--only id ...] [--out folder]` | The rederivation stage alone, on the builds of an earlier artifact run (only read, never changed): in three passes, a fresh reader works out each subject's target question from the renderings and names the part each step rests on, and a grader marks each expected step blind. Writes `rederive.md` and `rederive-scores.json`. See [Evals](evals.md#rederivation). | +| `explainer eval targets [--only id ...]` | Check each rederivation target in `evals/artifact/questions.yaml` with a fresh agent, and record the verdicts in `evals/artifact/rederive-checks.json`. Only targets changed since their last check are checked. | +| `explainer eval human --builds --chain ` | Grade a person's working on a rederivation target with the same rubric as the model readers. See [`evals/artifact/human-check.md`](../evals/artifact/human-check.md). | +| `explainer eval cuts [--only id ...] [--redo] [--out folder]` | Positive controls for the rederivation stage (PRD-002a): for each subject in `evals/artifact/controls.yaml`, cut one building block from its saved `principles` explanation with a fresh agent, check the cut with another, and keep the diff. The source builds are only read. A rerun skips finished cuts; `--redo` makes one more attempt (at most three in all). It refuses a folder that `eval controls` has run on. Default folder: `evals/artifact/results/-dev-controls`. | +| `explainer eval controls --out ` | Run the rederivation stage on the cut copies in a folder `eval cuts` made, reusing the baseline's reader chains for the uncut conditions, check uncaught cuts for leaks, and write `controls.md` and `controls.json` with the verdict under the bar in `controls.yaml`. See [Evals](evals.md#rederivation). | ## `check` diff --git a/docs/evals.md b/docs/evals.md index 1836f83..ab825a0 100644 --- a/docs/evals.md +++ b/docs/evals.md @@ -46,6 +46,88 @@ Effort rank: the grader's ranking of each subject's four explanations by the rea Two faults in the first run, both fixed: builders ran inside this repo, so Claude Code loaded its CLAUDE.md (and the principles) in every condition, including `none`; and the grader read Markdown with inline code removed, so formulas looked like blanks. Builds now run in a temp folder outside every project, with a guard. +## Rederivation + +The quiz checks what an explanation says. A reader who has grokked a subject can do more: from its building blocks, work out a result nobody told them. An explanation of entropy as "the average length of the best code for a source" never mentions KL divergence, yet a reader who holds that idea can derive it: messages from p sent with a code built for q cost −log q each instead of −log p, so the average extra cost is Σ p log(p/q), never negative, because no code beats the one built for the true source. The rederivation stage tests that, on the same held-out subjects. + +**Targets.** Each subject in `evals/artifact/questions.yaml` has a `rederive` target: a question the explanation was never asked to answer, 3–6 expected steps, and the result. Each step is a building block (an idea a good explanation gives) or a derived step (worked out from the steps before it). A fresh agent checked every target (`explainer eval targets`, verdicts in `evals/artifact/rederive-checks.json`) against five tests: every step and the result are true; every building block comes from the subject's question, the quiz's expected answers, or background this reader already has; every derived step follows from earlier steps plus that background and brings in no new fact about the subject; each step is one any valid derivation needs; and a good answer to the subject's question would not state the result. The second and third tests matter most: a target that needs an idea a focused explanation may leave out would reward length, not understanding. All 30 passed (two after a revision) and were frozen in one commit before the first run; a test fails if a results folder's targets differ from the file. + +**Reader and grader.** The reader gets the renderings and the target's question, never the steps or the result. It works the question out step by step, and for each step names the part of the explanation it rests on (a heading, paragraph, figure, or a control on a page) and says what that part gives it. Reader and grader get the same rule: background this reader already has may be used without a citation; every fact about the subject must come from a part the reader names. The grader sees each subject's explanations and the readers' working under shuffled labels, and marks each expected step, credited in any order, merged, split, or by an equivalent route: + +- **supported:** made, and the cited parts carry the ideas it rests on (or it uses only background on top of earlier steps); +- **unsupported:** made from the reader's own knowledge, or from a part that does not carry the idea; +- **missing;** +- **stated:** a derived step the explanation states outright, so the reader looked it up. A stated step, or a stated result, is left out of the score and counted, so a long explanation is neither rewarded nor penalized for spelling out what the reader was meant to derive. + +The headline is the **supported** share of expected steps; **raw** is the share made at all; **result right** the share of results that match. A run makes three **passes**: each is a fresh reader run over every build, graded once under its own label shuffle, so the spread covers both the reader's and the grader's variation. **The separation rule**, fixed before the first run: condition A beats condition B when A's supported score is higher than B's in all three passes. + +```bash +uv run explainer eval rederive --builds evals/artifact/results/2026-10-04-dev # the stage alone, on saved builds +uv run explainer eval artifact --split dev # builds, quiz, then this stage +``` + +### What the dev baseline found + +Dev baseline, 2026-10-20: the 40 v0.8 builds (10 subjects × 4 conditions), read and graded three times by the `claude` CLI's default model; no new builds. [Report](../evals/artifact/results/2026-10-20-dev-rederive/rederive.md). + +| condition | supported (3 passes) | result right | stated steps | words | quiz (grounded) | +|---|---|---|---|---|---| +| none | 100%, 100%, 100% | 100%, 100%, 100% | 2, 2, 3 | 1,689 | 100% | +| principles | 95%, 100%, 97% | 89%, 89%, 100% | 4, 4, 5 | 1,006 | 99% | +| quick (skill) | 92%, 100%, 90% | 100%, 100%, 89% | 2, 2, 2 | 606 | 95% | +| published (skill) | 95%, 95%, 95% | 89%, 78%, 89% | 3, 3, 3 | 1,016 | 95% | + +**Under the separation rule, one pair separates: none > published** (100% against 95% in every pass). No other pair does. + +**What that result is made of.** About 38 expected steps count per condition, so 95% is two steps short. The gap comes from one subject, two-tides, and from half of one building block. Block 2 bundles two ideas: the Sun's tide is about half the Moon's, and at new and full Moon the two line up and add. The published explanation gives the ratio but never says when the tides add (no spring or neap tides), so the grader marked the block unsupported in all three passes. That is the gap the quiz already found (published scored 6 of 8 on two-tides, 0 on the spring-tide question). The quick explanation also gives the ratio, and only mentions in a "not covered" line that the Sun's bulges add to or partly cancel the Moon's; the grader marked its block unsupported, supported, and missing in the three passes, so it does not separate. + +**How much a single subject moves between passes.** The same quick two-tides build scored 3/4, 4/4, and 1/4 supported steps in the three passes. A difference of a step or two in one subject is within what one fresh reader and one grading change on their own. + +**Known defects, found after the run** (the targets are frozen, so they stay and are reported): + +- **The float-sum target overclaims.** Its last derived step argues that adding 0.1 ten times cannot land exactly on 1.0 because each addition rounds. Rounding errors can cancel (`0.1` added five times does give exactly `0.5`), so the honest answer from the building blocks is "not guaranteed", which is what the principles and published readers said. They were marked wrong in passes 1 and 2; in pass 3 the published reader gave the same hedged answer and was marked right, so the grader was not consistent either. This lowers principles' and published's "result right" (published's also falls on two-tides in passes 2 and 3) and cost each one supported step in pass 1. The separation does not depend on it: two-tides alone carries none > published. +- **Real explanations stated derived steps outright in 3 of 10 subjects,** more often than the target checker predicted: the gps-relativity clock argument (three conditions), the greenhouse no-lapse-rate case (both derived steps, in principles, quick, and published), and the two-tides right-angle step (none and principles). Those steps are left out of the score, so for greenhouse three conditions were scored on building blocks alone. +- **Compound building blocks are graded inconsistently when half present.** The quorum block "any two majorities share a server, so the next group sees every committed write" is half present in the principles and quick explanations (they give the overlap, for conflicting writes, not for reads); the grader marked it unsupported in two of three passes. The two-tides block above is the same kind, and it carries the only separation. + +**Does the order follow length?** Roughly. On the mean supported share the order is none (100%) > principles (97%) > published (95%) > quick (94%); by words it is none (1,689) > published (1,016) ≈ principles (1,006) > quick (606). The one separation goes in length's direction. The word count also leaves out what the grader reads in pages' scripts, and only none made pages (5 of 10). + +**What we take from it (interpretation, not the rule).** With a strong model reader, rederivation mostly re-detects missing building blocks, which the quiz already catches; it does not yet show a difference in understanding between conditions. The one separation is small, comes from half of one building block in one subject, and favors the longest explanation. Whether PRD-003 adopts this stage, and what to change first (a weaker or constrained reader, harder targets with more derived steps, or human readers), is open: see ROADMAP §15. + +**Human check:** none yet. A brief for testers is in [`evals/artifact/human-check.md`](../evals/artifact/human-check.md); results will be recorded here. + +### Does the test catch a missing block? + +PRD-002a, 2026-10-29. The baseline sat near its ceiling. That could mean the v0.8 builds are good, or that the test is blind: a strong reader fills in a missing block from its own knowledge and the grader credits it. To tell these apart, one building block was cut from saved `principles` explanations, and the unchanged stage was run on the cut copies. [Report](../evals/artifact/results/2026-10-29-dev-controls/controls.md). [The cuts and their review](../evals/artifact/results/2026-10-29-dev-controls/cuts/review.md), frozen before the run. + +**What was cut, and why only six subjects.** A cut block can show up as missing only if the reader cannot get it elsewhere. So a block was eligible only if it is not in the target question or presupposed by it, is not background for this reader, cannot be derived from the other blocks, and was credited in all three baseline passes. Six dev targets have such a block. Four do not (antibiotic-resistance, gps-relativity, quorum, vaccine-memory): the question or school background supplies every block, or every block the intact build was reliably credited with (quorum's second block was not). For each of the six, a fresh agent was asked to remove the block, keep everything else word for word, and reword only where a removal broke a sentence. A second agent checked three things: the block could not be recovered, the removal was not marked, and the other reliably credited blocks were still there (the clean set, 4 blocks). Each cut got at most three attempts. Bloom-filter's cut failed all three: its block (adding an item sets bits to 1, and bits are never cleared) is implied by the worked example and the error maths. So five subjects count. + +Two accepted cuts went further than asked. In two-tides, the cutter added a sentence restating block 1 for the Sun, in the target step's own wording: it had been shown the clean blocks' text, which the PRD does not give it, and so the reader saw target text. That can only raise marks; block 1 stayed supported, and the verdict does not depend on it. In float-sum, the cutter reworded a removed sentence into one the reader later used (below). The cutter now gets only the reader, the question, the block and the explanation; the frozen cuts were made before that change. + +**The bar, fixed before the run.** A cut is caught when the grader marks the cut block unsupported or missing in all three passes. Trusted needs three things: at least 80% of counted subjects caught, at most one clean-set block losing support, and at least five subjects counted. An uncaught cut can leave the count, but only if a leak checker quotes a passage that still carries the block and code finds that quote in the cut copy. The verdict is also given with every leaked cut counted as a miss. If only the exclusions make it trusted, it is inconclusive. + +**Verdict: trusted, at the floor.** 4 of 5 cuts were caught, and 4 were needed. No clean-set block lost support, no leak was found, and the verdict is the same with leaks counted as misses. One more miss, or one more subject dropped or leaked, would have made it blind or inconclusive. The verdict also rests on how the grader treats a block the reader rebuilt. In heat-pump, the reader rebuilt the cut formula from a table the cut left, and the grader withheld credit: a catch. In float-sum, the reader rebuilt the cut block the same way, and the grader gave credit: the miss. The checker had passed heat-pump's copy while noting that the table's numbers fit the formula. Had the grader credited heat-pump's rebuilt block too, the verdict would have been blind. + +| subject | cut block | intact (baseline) | cut | caught | supported, intact → cut | result right, intact → cut | +|---|---|---|---|---|---|---| +| float-sum | 1: a value is exact only if its denominator is a power of 2 | s, s, s | s, s, s | no | 3/4, 4/4, 4/4 → 4/4, 4/4, 4/4 | 1 of 3 → 2 of 3 | +| greenhouse | 1: Earth radiates from the height where infrared escapes; CO2 raises it | s, s, s | m, m, m | yes | 2/2 → 0/4 in each pass | 3 of 3 → 0 of 3 | +| heat-pump | 2: ideal heating COP = T_hot ÷ (T_hot − T_cold) | s, s, s | u, u, u | yes | 4/4 → 3/4 in each pass | 3 of 3 → 3 of 3 | +| key-exchange | 3: the discrete logarithm is infeasible for large p | s, s, s | m, m, m | yes | 5/5 → 3/5 in each pass | 3 of 3 → 0 of 3 | +| two-tides | 2: the Sun's tide is about half the Moon's | s, s, s | m, m, m | yes | 3/3 → 2/4 in each pass | 3 of 3 → 0 of 3 | + +Marks per pass: s supported, u unsupported, m missing. The supported counts change their denominators because stated steps are left out of the score. Greenhouse's intact explanation stated the result outright in all three passes; the cut copy did not. + +- **The miss: float-sum.** The reader rebuilt the block from two sentences in the cut copy, plus its own knowledge of binary fractions. One sentence, the cutter's rewording of a sentence it removed, says a double holds a limited number of binary digits, so its values form a grid; the other says 1/10 repeats forever in binary. The grader credited the rebuilt block in every pass. The leak checker found no passage that carries the block, so the subject counts as a miss. This is either the blind case the controls exist to find, or a cut that left too much in; this run cannot tell which. +- **Caught does not mean the reader failed.** In heat-pump, the reader worked the formula out from the table the cut left: 318 K, a 50 K lift and an ideal COP of 6.4, a leftover the review named before the run. The reader got the result right in all three passes. The grader still marked the block unsupported, because the explanation does not give it. Here the test caught what the explanation leaves out; in float-sum, with a block rebuilt the same way, it did not. +- **Derived steps.** Where a derived step rests on the cut block, it lost credit in three subjects: key-exchange step 5 (the eavesdropper needs a discrete logarithm) and two-tides step 4 (a third of a spring tide's range) went from supported to missing, and greenhouse steps 3–4 went from stated to missing or unsupported. In heat-pump it did not. Step 4 there is algebra on the cut formula, and it stayed supported in every pass while the formula was marked unsupported. That reader did rebuild the formula, so the step's credit may be right; what it shows is a block and the step resting on it graded inconsistently. Making a derived step's credit depend on the blocks it rests on would remove that. +- **Greenhouse's catch comes with a thinner explanation.** Block 2 presupposes block 1, so the cut removed most of block 2 as well. The design allows this, because block 2 is not in the clean set. + +**Differences from the baseline.** The reader read only the cut copies. For none, quick and published, the baseline's chains were reused, and the grader graded all four fresh. One more difference was missing from the PRD's list. The stage shuffles labels once per subject, from one generator per pass, and this run covers five subjects, not ten. So each subject's label shuffle differs from its baseline pass. + +**What it means.** On targets with an eligible block, the test notices a missing block. On the four subjects where the cut was caught, the baseline's near-ceiling score is then a fact about the v0.8 builds, not a sign of a blind test; on float-sum, the miss, it may not be. PRD-003 may use the test, but only on targets with an eligible block. The evidence is thin: five subjects, one model, a verdict at the floor that depends on the grader withholding credit from one rebuilt block and giving it to another, and one miss of the kind the controls look for. + +**What the eligibility rule found.** In four of the ten dev targets, the question or background supplies every block, or every block the intact build was reliably credited with. Those targets cannot show a missing block in any condition, so on them the ceiling comes from the target, not from the builds. PRD-002's blocks test allowed blocks from the question on purpose, so the 20 test-split targets may share the problem. They need the same sort before PRD-003 relies on them. + ## The chat eval `evals/chat/questions.yaml` holds six questions about phenomena (why ice floats, why TCP starts slowly, how a wing makes lift, why a hash table is fast, why we see one side of the Moon, why salt melts ice). For each one it lists: diff --git a/docs/prds/PRD-002-rederivation-eval.md b/docs/prds/PRD-002-rederivation-eval.md new file mode 100644 index 0000000..f55894b --- /dev/null +++ b/docs/prds/PRD-002-rederivation-eval.md @@ -0,0 +1,183 @@ +# PRD-002: A rederivation eval + +Status: approved, 2026-10-20 · Part 2 of 3 (PRD-001: ship v0.8; PRD-003: Olah-compass principles) + +## Why + +The v0.8 quiz is at its ceiling. Every condition scores 94–100%, because a model reader answers from any competent explanation (`docs/evals.md`). So the quiz cannot tell PRD-003 whether a change to the principles helps. + +An explanation here aims for the reader to grok the subject: once they hold its building blocks, they can work out results nobody told them. Rederivation tests exactly that. + +An example from outside the 30 subjects: an explanation of entropy as "the average length of the best code for a source" never mentions KL divergence, but a reader who holds that idea can derive it. + +- Messages from p sent with a code built for q cost −log q each instead of −log p. +- So the average extra cost is Σ p log(p/q). +- Its properties follow without being taught: + - it is never negative, because no code beats the one built for the real source; + - it is zero only when p = q; + - it is not symmetric; + - it is infinite when q gives zero probability to something p produces. + +Each quiz already ends with a one-step "predict" item. This PRD adds the several-step version and grades the chain of reasoning, not only the answer. + +The reader is a model that already knows most results, so a right answer alone proves nothing. The fix: every step must rest on a part of the explanation that the reader names, and the grader checks that the part carries the idea. + +## Goals + +### 1. A rederivation target for every subject, frozen before any run + +Each of the 30 subjects in `evals/artifact/questions.yaml` gets a target with three parts: + +- **the question** the reader works out; +- **the expected steps:** 3–6 of them, one sentence each. Each step is an idea that any valid derivation needs, not the bookkeeping of one route. Each is marked as one of: + - a **building block**, which a good explanation gives; + - a **derived step**, which the reader works out from earlier steps. +- **the result:** the expected conclusion, specific enough to be right or wrong. + +Each target passes three tests: + +1. **Allowed building blocks only.** Every step uses one of: + - an idea from the subject's question; + - an idea from the quiz's expected answers; + - background at the reader's level. That level is set by the `reader` line in `questions.yaml`: school mathematics and science, and basic programming. + + A target that needs an idea a focused explanation can leave out rewards length, not understanding. The conditions average from 606 to 1,689 words, so that bias would be large. PRD-003 would then tune the principles toward length, the opposite of their aim. +2. **Several steps:** at least three. +3. **Not stated:** a good answer to the subject's question would not state the result. + +Claude drafts the targets. A fresh `claude -p` checker sees the subject's question, its quiz with the expected answers, and the target, then applies the three tests and returns pass or fail with a reason. + +- A failed target is rewritten and checked again, at most twice. +- A subject whose target still fails gets no target, and the report lists it. +- The checker's verdicts are saved with the targets. + +All 30 targets, test subjects included, are committed in one commit before the first rederivation run. Writing them now means the PRD-003 changes cannot shape them. The owner may skim them; nothing waits on that. + +### 2. A rederive stage in the artifact eval + +**Reader.** The reader gets the renderings only, as today, and the target's question. It does not see the expected steps or the result. It works the question out step by step. For each step it gives: + +- the idea the step uses; +- the part of the explanation that idea comes from: a heading, a paragraph, a figure or diagram, or a control on a page; +- that idea in its own words. + +If the explanation lacks an idea the reader needs, it says so and stops there. A gap is evidence. + +**Outside-knowledge rule.** The same text goes in the reader prompt and the grader prompt: + +- background at the reader's level may be used without citing anything; +- any fact about the subject must come from a part of the explanation that the reader names. + +**Grader.** The grader works blind, under shuffled labels, as today. For each explanation it sees the text, the reader's chain, and the target's expected steps and result. The expected steps are a checklist of ideas, not a script. A step counts when the reader's chain establishes the same idea in any order, whether merged with another step, split in two, or reached by an equivalent route. The grader marks each expected step as one of: + +- **supported:** the reader made the step, and either the cited parts carry the ideas it rests on, or it uses only reader-level background on top of earlier steps; +- **unsupported:** the reader made the step from its own knowledge, or from a part that does not carry the idea; +- **missing;** +- **stated:** a derived step that the explanation states outright, so the reader looked it up instead of working it out. + +The grader also marks whether the result is right, and whether the explanation states the result outright. + +**Scores.** Each condition gets three scores: + +- **supported:** the share of expected steps that are supported; +- **raw:** the share made at all; +- **result right.** + +These mirror the quiz's grounded and raw scores. + +A stated derived step, or a stated result, is left out of that explanation's score: out of both the count and the total. The report counts these per condition. That way a long explanation is neither rewarded nor penalized for spelling out what the reader was meant to derive. + +**Passes.** Every run of the stage makes three passes. In each pass, a fresh reader works through every build, and the grader grades it once under its own label shuffle. The spread across passes then covers both the reader's variation and the grader's. + +The stage runs in every `eval artifact` run from now on, because PRD-003 needs it on new builds. It can also run alone on the saved builds of an earlier run. + +### 3. A dev baseline on the saved v0.8 builds + +- **No new builds.** The results go in a new folder that reuses the 40 builds saved in `evals/artifact/results/2026-10-04-dev/` (10 subjects × 4 conditions). +- **Models.** The reader and the grader use the `claude` CLI's default model, as the v0.8 baseline did. +- **Three passes,** as defined in Goal 2, each reported. The v0.8 ranges came from three different gradings, one of them the faulty first run, so they are no model to copy. +- **The report** gives, per condition, the three scores (with their range across passes), next to words and the v0.8 quiz score, plus a table per subject. + +### 4. The result is written up, whatever it is + +`docs/evals.md` gets a "Rederivation" section. It says what the eval measures and gives the three target tests. Then it reports what the dev baseline found, judged by the rule below. + +**The separation rule, fixed before the first run.** Condition A beats condition B when A's supported score is higher than B's in all three passes. + +The section says: + +- which pairs of conditions separate; +- whether the order follows word count, which would mean the score measures length; +- if no pair separates, that the eval does not yet rank the conditions, and that PRD-003 should not adopt it as its measure without changes. + +The "Next" list in ROADMAP §15 and the "Unreleased" section of `CHANGELOG.md` record the stage and its result. + +### 5. A human check that does not block + +`evals/artifact/human-check.md` is a one-page brief for a person who reads one explanation and works out its target on paper, before seeing the expected steps. Their chain is graded with the same rubric. `docs/evals.md` then records whether the person's score and the model reader's score on that explanation agree. + +PRD-002 is done without any human results; they are added when they come in. + +## Non-goals + +- **No change to the principles, the skills, or other measured text.** That is PRD-003. +- **No new builds and no run on the test split.** A test run needs new builds for all 20 test subjects. It runs when PRD-003 confirms a change. +- **No changed-world variants and no reader run without an explanation.** They were cut to keep this simple. +- **No change to the quiz, its scoring, or the v0.8 reports.** + +## Anti-goals + +- Changing a target, its steps, or the separation rule after seeing any rederivation score. +- A target that rewards length because it needs an idea outside the allowed building blocks. +- Claiming a separation the rule does not support, or presenting a 10-subject dev run as more than that. + +## Constraints + +- **The repo is public (MIT).** Targets and docs contain no personal or private-project context. +- **The shipped v0.8 record stays as it is.** `evals/artifact/results/2026-10-04-dev/` and `2026-10-04-dev-haiku-reader/` stay byte-identical: `git diff --exit-code` on them is clean. +- **The builder never sees a target.** Build prompts are unchanged. +- **The reader never sees** the expected steps or the result. +- **Grading stays blind** under shuffled labels. +- **Every stage saves its output,** and a rerun skips what exists, as today. +- **Tests run offline,** without the `claude` CLI, like the existing artifact eval tests. + +## Decided + +All decided by the owner on 2026-10-19. + +- **Rederivation is the test of grokking.** Once the reader has the building blocks, they can rederive the results and the rest of the theory from first principles. +- **A citation names a part of the explanation and restates its idea.** Word-for-word quotes are too strict, and an idea carried by a figure cannot be quoted. +- **The owner's time is not required.** Claude drafts the targets, a model checks them, and the owner may skim. +- **Simpler beats complete.** No changed-world variants and no control run without an explanation. +- **Human calibration happens when a tester is free.** It blocks nothing. +- **Targets are frozen before the first run.** + +## Calls made by Claude (check these) + +Approving this PRD approves these calls. Nothing else waits on the owner. + +- **The allowed-building-blocks test:** the question, the quiz's expected answers, and reader-level background. This keeps the score from rewarding length. +- **Targets for all 30 subjects now;** only dev runs in this PRD. +- **The separation rule:** higher in all three passes. +- **A pass is a fresh reader run, graded once under its own shuffle.** This triples the reader calls in every run. Reads are cheap next to builds, and only this design tests whether a difference survives the reader's own variation. +- **Credit for other routes.** Expected steps are a checklist of ideas, credited in any order or form. Each step must be one that any valid derivation needs. +- **Stated derived steps and stated results are left out of the score and counted.** They are neither rewarded nor penalized. +- **Scores:** supported, raw and result-right, mirroring the quiz. +- **The rederive stage joins every artifact eval run** from now on. +- **No release is decided here.** The owner decides at the end whether PRD-002 ships as v0.9.0 or waits for PRD-003. + +## Left to the builder + +- Field names in `questions.yaml`, and the prompt wording beyond the outside-knowledge rule. +- The CLI form for running the stage on an earlier run's builds. +- How the new folder reuses the builds, by copy or by reference, provided the v0.8 folders stay unchanged. +- Whether the grader sees a subject's four explanations in one call, as today, or one at a time, provided the labels stay shuffled. + +## Done when + +- Each of the 30 subjects has a target whose checker verdict is pass, or is listed as having none. The targets were committed before any rederivation result. +- The dev baseline is in a new results folder with three passes. Its report shows the scores next to word count and the quiz. +- `docs/evals.md` states the result under the separation rule, including "no separation" if that is the outcome. +- The v0.8 results folders are unchanged. +- `evals/artifact/human-check.md` exists. +- Tests pass offline, `explainer check` passes, and CI is green. diff --git a/docs/prds/PRD-002a-rederivation-controls.md b/docs/prds/PRD-002a-rederivation-controls.md new file mode 100644 index 0000000..40bd0dc --- /dev/null +++ b/docs/prds/PRD-002a-rederivation-controls.md @@ -0,0 +1,224 @@ +# PRD-002a: Does the rederivation test catch a missing block? + +Status: approved, 2026-10-29 · Amends PRD-002, before PRD-003 (Olah-compass principles) + +## Why + +PRD-002's dev baseline came out near the ceiling. Every condition was credited with 90–100% of the expected steps in every pass. The one separation, none > published, rests on half of one compound building block in one subject (two-tides), and it follows word count (`docs/evals.md`, "Rederivation"). + +That result has two readings, and they lead to different next steps: + +- **The test works, and the explanations really do contain the blocks.** Then PRD-003 can use the test, and the near-ceiling score is a fact about the v0.8 builds. +- **The test is blind.** A strong model reader may fill in a missing fact from its own knowledge, and the grader may credit it. Then the near-ceiling score says nothing, and the test must be fixed before PRD-003 relies on it. + +Tuning the test until the four conditions come out different would fit it to the answer we want. This PRD checks the test against cases where the right answer is known instead: explanations with one building block cut out on purpose. A test we can trust marks the cut block as not supported. + +## Goals + +### 1. Six cut explanations, frozen before any run + +**The starting build.** The saved v0.8 `principles` build in `evals/artifact/results/2026-10-04-dev/work//principles/`. Each is a single `explanation.md` with no page or script, so a cut fact cannot survive somewhere else in what the reader is shown. + +**Which blocks can be cut.** A cut is a fair test only if a working test must then withhold credit. That is true only of a block the explanation alone can supply. So a block is eligible when all four of these hold: + +1. the target question neither states nor takes it for granted; +2. it is not background at the reader's level (the `reader` line in `questions.yaml`); +3. it does not follow from the subject's other blocks plus that background; +4. the intact `principles` build was marked supported for it in all three passes of the PRD-002 baseline (`evals/artifact/results/2026-10-20-dev-rederive/`). Otherwise the cut would be a free catch. + +In each subject, the last eligible block is cut. A subject with no eligible block is left out. Claude classified the 20 dev blocks against this rule, using only the targets and results that already exist. + +| subject | cut | the block, in short | +|---|---|---| +| bloom-filter | 1 of 1 | adding an item sets its k bits to 1, and bits are never cleared | +| float-sum | 1 of 2 | doubles are binary fractions: a value is exact only if its denominator is a power of 2 | +| greenhouse | 1 of 2 | the Earth radiates to space from the height where infrared escapes, and more CO2 raises that height | +| heat-pump | 2 of 2 | the ideal heating limit is COP = T_hot ÷ (T_hot − T_cold), in kelvin | +| key-exchange | 3 of 3 | recovering an exponent from g^x mod p is infeasible for large p | +| two-tides | 2 of 2 | the Sun's tidal effect is about half the Moon's, and in line the two add | + +| left out | why | +|---|---| +| antibiotic-resistance | The question gives the pre-existing resistant mutations (block 1). Selection (block 2) is school biology, and the question takes for granted that one drug alone ends in resistance. | +| gps-relativity | The question states the receiver's clock error as a fourth unknown (block 2). It ties a satellite's clock error to its distance, and distance = speed × time is school physics (block 1). | +| quorum | The question states the write group and the committed write (block 1). Block 2 was marked unsupported in two of the three baseline passes. | +| vaccine-memory | The question's model states both blocks: matching cells multiply until there are enough, and vaccination leaves 1,000 times more of them. | + +Float-sum's block 2 and greenhouse's block 2 are not eligible either. Each follows from block 1 or leans on it. + +**The cut.** A fresh `claude -p` cutter gets the explanation and the block. + +- It removes every sentence that states the block or lets a reader recover it, including a stated derived step that gives the block away. +- It keeps the rest word for word, rewording only where a removal breaks a sentence. + +**The check.** A separate fresh `claude -p` checker sees the cut copy, the target question, the reader line, and the subject's building blocks. It quotes any passage behind its answers. + +- It confirms that this reader cannot get the cut block from the cut copy, the question, or background. This also re-checks the eligibility rule. +- It confirms that every block in the clean set (Goal 3) is still in the copy. +- A cut that fails is redone, at most twice. +- A subject whose cut still fails leaves the count, and the report lists it. + +**The diff** of each cut is saved. Claude reads every diff before the freeze, and redoes (as one of the two retries) any cut that removed more than the block and what implies it. + +**Freeze.** The cut copies, the checks, the diffs, and the bar in Goal 3 are committed in one commit before any rederivation run on a cut copy. + +### 2. A controls run + +A new results folder, `evals/artifact/results/-dev-controls/`, holds a builds folder laid out like an artifact run, so the PRD-002 rederive stage takes it unchanged: + +- the six subjects' `none`, `quick`, and `published` builds, copied from v0.8; +- the cut copy in place of each `principles` build. + +**Reads.** The run reuses the PRD-002 baseline's reader chains for `none`, `quick`, and `published`, pass for pass. Only the cut copy is read again. + +**Grading.** The grader grades all four explanations fresh, blind under shuffled labels, exactly as in the baseline. So the only differences from the baseline's pass k are: + +- the cut copy; +- its fresh read; +- a fresh grading call. + +The reader and grader prompts are word for word those of the baseline run. The grading code has changed since in two ways that touch only the calling: a reply that doesn't fit is asked again once, and the raw marks are kept. + +**Passes and models.** Three passes. The reader, grader, cutter, and checker use the `claude` CLI's default model, as the baseline did. + +**The leak check, after the run.** A cut the checker passed may still leave the block in a passage it missed. Such a passage would let the reader cite the block and the grader credit it, which would look like a blind test. So for each subject whose cut block is not caught (Goal 3): + +- a fresh `claude -p` checker sees the reader's chain and the cut copy; +- it says whether a part the reader cited, as it stands in the copy, carries the cut block, and if so quotes that passage word for word. + +A leak counts only when code confirms that the quote appears in the cut copy, ignoring differences in whitespace. If a confirmed leak is found in any pass, that subject leaves the count, and the report lists it with the quote. Otherwise the subject counts as a miss. + +This check can only ever turn a miss into an exclusion, and it asks much the same question the grader answered. So Goal 3 limits what it can do to the verdict. + +### 3. The bar, fixed in the freeze commit + +The bar uses the grader's raw marks, before `stated` on a block is turned into `supported`. + +**Caught.** The cut block is marked unsupported or missing in all three passes. Any other mark in any pass, `supported` or `stated`, means it was not caught. + +**Clean.** An uncut block stays supported in all three passes when two things are true: the intact build was marked supported for it in all three baseline passes, and it does not rest on the cut block. There are 4 such blocks: + +- heat-pump block 1; +- key-exchange blocks 1 and 2; +- two-tides block 1. + +**Counted subjects.** The six subjects in Goal 1, minus any whose cut failed its check and any whose cut leaked. + +**The verdict:** + +- **Trusted:** caught in at least 80% of the counted subjects, rounded up (5 of 6, or 4 of 5), and at most one of the clean-set blocks loses support. +- **Blind:** caught in fewer than that. +- **Inconclusive:** any of these: + - fewer than 5 subjects are counted; + - the catch bar is met, but more than one clean-set block loses support. The cuts or the reads then damaged more than the cut block, so the catches may come from that damage; + - the verdict is trusted only because of leak exclusions: counting every leaked subject as a miss would make it blind. + +The report gives the verdict both ways: with the leak exclusions, and with every leaked subject counted as a miss. + +### 4. A report and a write-up, whatever the result + +A controls report next to `rederive.md` gives, per subject: + +- the cut block's raw mark in each pass, next to the intact build's baseline marks; +- the marks of the uncut blocks; +- the marks of the derived steps, intact and cut. Derived steps that stay supported after their base is cut are direct evidence for making a derived step's credit depend on the blocks it rests on; +- the supported share, intact and cut; +- whether the reader named a gap; +- the leak check's answer, where one ran; +- the four subjects left out, and why. + +`docs/evals.md` gets a subsection under "Rederivation". It gives the verdict under the bar and what it means for PRD-003: + +- **Trusted:** on targets with an eligible block, the test notices a missing block. On those six subjects, the baseline's near-ceiling score is then a fact about the v0.8 builds, not a sign of a blind test. PRD-003 may use the test, but only on targets with an eligible block. +- **Blind or inconclusive:** PRD-003 does not use the test as it is. + +Either way, the subsection notes what the eligibility rule found. In four of the ten dev targets, the question or background supplies every block, or every block the intact build was reliably credited with. Those targets can't show a missing block in any condition, so on them the ceiling comes from the target, not the builds. PRD-002's blocks test allowed blocks from the question on purpose, so the 20 test-split targets may share the problem. + +ROADMAP §15 ("Next") and the "Unreleased" section of `CHANGELOG.md` record the result. "Next" also gets one item: before PRD-003 relies on the test-split targets, classify them by the same eligibility rule. + +### 5. Only if the verdict is blind: fix the test against the controls + +This part gets its own plan after Goal 4's result, because the report's diagnostics say which fix is needed. Its rules are fixed now: + +- **A new scoring version.** The frozen targets, the PRD-002 scoring, and every reported result stay as they are. A change to a target, such as splitting a two-fact block, makes a new versioned target set. That set passes the PRD-002 checker and is frozen before any run. +- **The candidate fixes:** + - a derived step counts only when the blocks it rests on count; + - blocks that hold two facts are split; + - a weaker or constrained reader; + - a stricter grader rule for outside knowledge. +- **Tuned only against the cut copies,** where the right answer is known. Never against the four conditions. +- **Confirmed on held-out controls.** A second set of cut copies is made the same way from another condition's builds and frozen before the fixed test sees it. It must pass the same bar. +- **Then one fresh four-condition dev run** with the fixed test, judged by the PRD-002 separation rule and written up whatever it shows. + +If the verdict is inconclusive, the PRD reports it and stops. The owner decides what follows. + +## Non-goals + +- **No change to the principles, the skills, or other measured text.** That is PRD-003. +- **No new builds and no run on the test split.** +- **No change to the frozen targets, the PRD-002 scoring, or any reported result.** +- **No code change to the test in Goals 1–4.** Only Goal 5 may change it. +- **No human check here.** It comes next, from `evals/artifact/human-check.md`. + +## Anti-goals + +- Choosing the cut blocks, the bar, or the verdict wording after seeing any control result. +- Tuning anything against the four conditions. +- Counting a cut that leaks, or one that removed more than its block, as a catch. +- Calling the test trusted on a run that missed the bar. + +## Constraints + +- **The repo is public (MIT).** No personal or private-project context. +- **Earlier results stay as they are.** `evals/artifact/results/2026-10-04-dev/`, `2026-10-04-dev-haiku-reader/` and `2026-10-20-dev-rederive/` stay byte-identical: `git diff --exit-code` on them is clean. +- **The targets stay frozen.** `test_targets_are_the_frozen_ones` passes unchanged. +- **Neither the reader nor the grader is told that any build was cut.** The reader never sees the target's steps or result. The grader stays blind under shuffled labels. +- **Every stage saves its output,** and a rerun skips what exists. +- **Tests run offline,** without the `claude` CLI. + +## Decided + +All decided by the owner. + +- **The trust test is a set of explanations with one building block cut out** (2026-10-22). The test is trusted if it reliably catches them, even if the four conditions still come out close. +- **One cut per eligible dev subject,** made from the `principles` builds by a model, confirmed by a separate check and a diff (2026-10-22). +- **Only blocks the explanation alone can supply are cut, which leaves six subjects. The bar is at least 5 caught of the 6** (2026-10-24). This replaces the earlier 8 of 10, after the builder check found four chosen blocks that the reader gets from elsewhere. +- **The known fixes come only if the controls fail** (2026-10-22). They are tuned on the controls, never on the conditions, and are followed by one fresh four-condition run. +- **Small first** (2026-10-22). This PRD, then the human checks, then PRD-003. + +## Calls made by Claude (check these) + +Approving this PRD approves these calls. + +- **The name PRD-002a,** so the Olah-compass PRD stays PRD-003 and existing references still hold. +- **The classification of the 20 dev blocks** in the two tables of Goal 1. The checker re-checks every chosen block before the freeze. +- **Bloom-filter stays,** though its only block is its mechanism, so the cut removes most of the explanation. It still tests whether a reader who knows Bloom filters well gets credit without the explanation. +- **Caught means unsupported or missing in all three passes,** on raw marks. +- **The leak check after the run.** A leaked cut leaves the count instead of counting as a miss, but only on a word-for-word quote that code finds in the cut copy. The verdict is reported both ways. A verdict that is trusted only because of leak exclusions is inconclusive. +- **"Trusted" covers only targets with an eligible block.** PRD-003 may use the test only on those, and the test-split targets get classified first. +- **At least 5 counted subjects are needed for a verdict.** Below that, the result is inconclusive. +- **The clean set of 4 blocks and the inconclusive verdict.** At most one clean-set block may lose support. +- **The eligibility rule reads the baseline's recorded marks,** which turn `stated` on a block into `supported`. The baseline predates the saving of raw marks. +- **The cut copy is graded among the other three conditions,** as in a real run, and not next to its own intact twin, which would let the grader spot the difference. +- **The baseline's reader chains are reused** for the three uncut conditions, so only the cut copy is read again. +- **The cutter and the checkers are fresh `claude -p` calls** with the default model, with at most two retries for a cut. Claude reads every diff before the freeze. +- **The held-out controls in Goal 5** come from another condition's builds. +- **Branch `prd-002a-controls` is stacked on `prd-002-rederivation`,** because PR #2 is not merged. +- **No release is decided here.** + +## Left to the builder + +- The CLI form: a new `eval` suite or a flag on an existing one. +- File names and layout in the controls folder, the prompt wording, and the report layout. +- How the builds folder copies the v0.8 builds, provided the v0.8 folders stay unchanged. + +## Done when + +- The cut copies, checks, diffs, and bar were committed before any control rederivation result. +- The controls run is in a new results folder, with three passes, the leak checks, and its report. +- `docs/evals.md` states the verdict under the bar, including "blind" or "inconclusive" if that is the outcome. ROADMAP and CHANGELOG record it. +- The three earlier results folders are unchanged, and the frozen-targets test passes. +- Tests pass offline, `explainer check` passes, and CI is green once pushed. +- If the verdict is blind, Goal 5 has its own plan and is done when either: + - the fixed test passes the bar on the held-out controls and the fresh four-condition run is written up; or + - the write-up says that no candidate fix passed. diff --git a/docs/superpowers/plans/2026-10-20-rederivation-eval.md b/docs/superpowers/plans/2026-10-20-rederivation-eval.md new file mode 100644 index 0000000..8b983b2 --- /dev/null +++ b/docs/superpowers/plans/2026-10-20-rederivation-eval.md @@ -0,0 +1,440 @@ +# Rederivation Eval Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Add a rederivation stage to the artifact eval, give all 30 subjects a frozen, model-checked target, and report a three-pass dev baseline on the saved v0.8 builds, whatever it shows. + +**Architecture:** +- One new module, `explainer_kit/rederive.py`, holds the targets (validation, hashing, checker), the reader, the grader, scoring, the run, the report, and the grading of a human chain. +- `artifact_eval.py` changes only where the reader's file copies are made: that code is extracted into `copy_for_reader` so both stages share it. +- The CLI gains three `eval` suites (`targets`, `rederive`, `human`), and `eval artifact` runs the rederive stage after it grades. +- The stage reads builds from a `--builds` folder and writes only to `--out`. That is how the baseline reuses the v0.8 builds without touching them. +- Work happens on branch `prd-002-rederivation` from `main`. + +**Tech Stack:** Python 3 / uv, pytest, PyYAML, the `claude` CLI through `explainer_kit.agent.ask`. + +**Spec:** `docs/prds/PRD-002-rederivation-eval.md` + +## Global Constraints + +- **Measured text does not change:** `git diff --stat main -- plugin/skills explainers` is empty at every commit. +- **The builder never sees a target.** In `artifact_eval.py`, `TASK`, `SKILL_TIER` and `build()` are unchanged: `git diff main -- explainer_kit/artifact_eval.py` touches only `read()` and the new `copy_for_reader`. +- **The v0.8 record stays byte-identical:** `git status --porcelain -- evals/artifact/results/2026-10-04-dev evals/artifact/results/2026-10-04-dev-haiku-reader` is empty at every commit. +- **Targets are frozen at the Task 2 commit.** Afterwards, `rederive` blocks in `evals/artifact/questions.yaml` never change. A test (Task 5) fails if a results folder's snapshot differs from the file. +- **The repo is public.** No committed file holds an absolute path, a username, an email address, a key, or private-project context. +- **The reader prompt and the grader prompt contain the same constant, `OUTSIDE_RULE`:** + > Background that this reader already has (see the reader above) may be used without a citation. Every fact about the subject must come from a part of the explanation that the reader names. +- **The separation rule, verbatim:** "Condition A beats condition B when A's supported score is higher than B's in all three passes." +- **A pass** is a fresh reader run over every build, then one grading per subject under its own label shuffle (`random.Random(k)` for pass `k`). +- **Tests run offline.** They stub `agent.ask` and `agent.subprocess.run` and never call the `claude` CLI. +- **The baseline uses the `claude` CLI's default model,** so no `--model` and no `--reader-model`. + +## Review Focus + +1. **A build with no renderings** (a failed build). Expect the reader to be skipped and every step marked missing. The run must not crash, and the report shows it. The test is in Task 3, `test_read_chain_skips_a_build_with_no_renderings`. +2. **A rerun after the targets changed.** Expect the run to refuse, naming the changed subjects, so no folder mixes two target versions. The test is in Task 5, `test_rerun_refuses_changed_targets`. +3. **Absolute paths leaking into saved output** (public repo). Expect the source folder saved relative to `--out`, and no prompt or temp path saved. The test is in Task 5, `test_outputs_hold_no_absolute_paths`. +4. **Grader replies that don't fit:** too few or too many marks, an unknown word, "stated" on a building block, or a missing label. Expect normalisation, not a crash or a skewed score. The test is in Task 4, `test_marks_are_normalised`. +5. **A page whose idea lives in a control.** Expect the grader to see the page's script as well as its visible text, so a reader citing a slider can be checked. The test is in Task 4, `test_grader_text_includes_page_scripts`. + +--- + +### Task 0: Branch + +- [ ] **Step 1:** Create the branch. In the PRD, change `Status: draft, 2026-10-19` to `Status: approved, 2026-10-20`; committing it before any run is what pre-registers the separation rule. Commit the spec and the plan. + +```bash +git switch -c prd-002-rederivation +git add docs/prds/PRD-002-rederivation-eval.md docs/superpowers/plans/2026-10-20-rederivation-eval.md +git commit -m "PRD-002: a rederivation eval, and its plan" +``` + +### Task 1: Targets — format, validation, and the checker + +**Files:** +- Create: `explainer_kit/rederive.py` +- Modify: `explainer_kit/cli.py`. Add `targets` to the `eval` suite choices; `cmd_eval` dispatches it. +- Create: `tests/test_rederive.py` + +**Interfaces:** +- Produces: + - `checks_file() -> Path`, which returns `project_root()/"evals"/"artifact"/"rederive-checks.json"`. + - `target_problems(t: dict) -> list[str]`. + - `target_hash(t: dict) -> str`: the first 12 hex digits of the sha256 of `json.dumps(t, sort_keys=True, ensure_ascii=False)`. + - `check_prompt(subject: dict, reader: str) -> str`. + - `check_targets(only: list[str] | None = None, model: str | None = None) -> Path`. +- The target format in `questions.yaml`, under each subject: + + ```yaml + rederive: + q: "…" + steps: + - {kind: block, text: "…"} + - {kind: derived, text: "…"} + result: "…" + ``` + +- The checks file maps `{subject_id: {"verdict": "pass"|"fail", "tests": {...}, "reason": str, "hash": str, "attempt": int}}`. + +- [ ] **Step 1: Write the failing tests** in `tests/test_rederive.py`. + - `test_target_problems`: + - a valid target (3 steps, at least one `derived`, non-empty `q` and `result`) gives `[]`; + - each of these gives a non-empty list: 2 steps; 7 steps; no `derived` step; kind `"lemma"`; empty `result`; missing `q`. + - `test_target_hash_is_stable_and_sensitive`: the same dict in a different key order hashes the same, and changing one step's text changes the hash. + - `test_check_prompt_shows_the_building_blocks`. For the first dev subject, given a target, the prompt contains: + - every quiz `expect`, the reader line, every step text with its kind, and the result; + - the phrases `"any valid derivation needs"`, `"would not state"`, and `"no new fact about the subject"`. + - `test_check_targets_records_and_resumes`. Monkeypatch `rederive.checks_file` to a tmp file and `ae.questions_file` to a tmp YAML with two subjects that have targets, and a stub `agent.ask` that returns: + + ```json + {"tests": {"correct": true, "blocks": true, "derived_follow": true, "steps": true, "not_stated": true}, "reason": "ok"} + ``` + + for one subject, and `"derived_follow": false` for the other. Assert: + - verdicts `pass` and `fail` (the verdict is computed in code as all five tests true, never read from the reply); + - each `hash` equals `target_hash`, and `attempt == 1`; + - a second call makes **no** `agent.ask` call; + - after editing the failing target and calling again, that subject has `attempt == 2` and only it was asked. +- [ ] **Step 2: Run them to verify they fail.** + Run: `uv run pytest tests/test_rederive.py -q` + Expected: FAIL. The module is not found. +- [ ] **Step 3: Implement the four functions in `explainer_kit/rederive.py`.** + - Module docstring style matches `artifact_eval.py`. + - `CHECKER_SYSTEM = "You are a careful scientist and teacher. You check test questions. Reply with JSON only."` + - The checker prompt asks for five booleans: + - `correct`: the steps and the result are true; + - `blocks`: every `block` step is an idea from the subject's question, from the quiz's expected answers, or background this reader already has; + - `derived_follow`: every `derived` step follows from the steps before it plus background this reader already has, and introduces no new fact about the subject (otherwise only explanations that happen to contain that fact could support it, and length wins); + - `steps`: there are at least three steps, and each is an idea any valid derivation needs, not one route's bookkeeping; + - `not_stated`: a good answer to the subject's question would not state the result. + + It also asks for a one-sentence `reason` that names any failing step. + - `check_targets` walks `ae.load("all", only)[1]`. It skips subjects without `rederive` and subjects whose saved hash matches. It sets `attempt` to the saved attempt plus 1 when the hash changed. It writes the file sorted by id, with a trailing newline. + - `cmd_eval` with `suite == "targets"` calls `check_targets(args.only, args.model)` and prints a count of pass and fail. +- [ ] **Step 4: Run the tests to verify they pass.** + Run: `uv run pytest tests/test_rederive.py -q` + Expected: 4 passed. +- [ ] **Step 5: Commit.** + `git add explainer_kit/rederive.py explainer_kit/cli.py tests/test_rederive.py && git commit -m "rederive: target format, validation, and the checker"` + +### Task 2: Draft, check and freeze all 30 targets + +This task is content work plus live checker calls (`claude -p`), about 30–60 calls. Nothing from Tasks 3–5 may exist in the working tree yet. The targets are written before the reader or grader prompts exist, and stay unchanged afterwards. + +**Files:** +- Modify: `evals/artifact/questions.yaml`. Add a `rederive` block to every subject, and add a header paragraph explaining the block and that the builder and reader never see the steps or the result. +- Create: `evals/artifact/rederive-checks.json` (generated). +- Modify: `tests/test_rederive.py` (one test). + +**How to draft a target.** The worked example below is from outside the 30 subjects and is not added to the file: + +```yaml +rederive: + q: "A code was built for the letter frequencies q of one language and is used for text from another, with frequencies p. How many extra bits per letter does it cost on average, and can that cost ever be negative?" + steps: + - {kind: block, text: "In the best code, a symbol of probability p gets a codeword of about -log2 p bits."} + - {kind: block, text: "Entropy is the average codeword length of the best code: the sum of -p log2 p."} + - {kind: derived, text: "With q's code, each letter costs -log2 q instead of -log2 p, so the average extra cost is the sum of p log2(p/q)."} + - {kind: derived, text: "No code beats the one built for the true source, so the extra cost is never negative, and it is zero only when p = q."} + result: "The sum of p log2(p/q) bits per letter (the KL divergence); never negative, zero only when p = q." +``` + +- A `block` step restates an idea from the subject's `q`, from a quiz `expect`, or from background at the reader's level. If you cannot point to one, the step is not a block, and the target likely rewards length: redraft it. +- Derived steps follow from the steps before them plus reader-level background, and bring in no new fact about the subject. The result follows from those steps. +- The question names a concrete case, quantity, sign, or limit that is not one of the quiz questions. + +- [ ] **Step 1: Write the failing test** `test_every_subject_has_a_checked_target`. For each subject in `ae.load("all")`, with `checks = json.loads(rederive.checks_file().read_text())`: + - if the subject has a `rederive` block: `target_problems(...) == []`, `checks[id]["verdict"] == "pass"`, and `checks[id]["hash"] == target_hash(...)`; + - otherwise: `checks[id]["verdict"] == "fail"` and `checks[id]["attempt"] == 3`. This is the "listed as having none" case. +- [ ] **Step 2: Run it to verify it fails.** + Run: `uv run pytest tests/test_rederive.py::test_every_subject_has_a_checked_target -q` + Expected: FAIL, because the checks file is missing. +- [ ] **Step 3: Draft the 30 `rederive` blocks** in `questions.yaml`, using the guidance above, and run the checker. + Run: `uv run explainer eval targets` + Expected: one verdict per subject in `evals/artifact/rederive-checks.json`. +- [ ] **Step 4: Rewrite failures, at most twice.** + - For each `fail`, rewrite the target using the checker's `reason`, then rerun the checker. The attempt count reaches 2, then 3. + - If a subject still fails at attempt 3, delete its `rederive` block and keep its `fail` entry. + - Ledger each deletion as a ruling. +- [ ] **Step 5: Run the test to verify it passes.** + Run: `uv run pytest tests/test_rederive.py -q` + Expected: all pass. +- [ ] **Step 6: Commit. This is the freeze commit.** + - `git add evals/artifact/questions.yaml evals/artifact/rederive-checks.json tests/test_rederive.py && git commit -m "Freeze the rederivation targets: 30 subjects, model-checked"` + - Ledger the commit hash as `FREEZE`. + +### Task 3: The reader + +**Files:** +- Modify: `explainer_kit/artifact_eval.py`. Extract `copy_for_reader(files: list[Path], shown: Path) -> list[Path]` from `read()`. It empties `shown`, then copies each file into it and returns the copies. `read()` calls it, and its behaviour is unchanged. +- Modify: `explainer_kit/rederive.py` +- Modify: `tests/test_rederive.py` + +**Interfaces:** +- Consumes: `ae.renderings(ws, sid)` and `ae.copy_for_reader`. +- Produces: + - `OUTSIDE_RULE: str`, the exact text in Global Constraints; + - `reader_prompt(files: list[Path], reader: str, question: str) -> str`; + - `chain_reply(text: str) -> dict`, which returns `{"steps": [{"step": str, "part": str, "idea": str}], "result": str, "gap": str}`; + - `chain_text(chain: dict) -> str`; + - `read_chain(subject: dict, cond: str, reader: str, builds: Path, out: Path, k: int, model: str | None) -> dict`, which saves to `out/"rederive"/f"pass-{k}"/"read"//f"{cond}.json"`. + +- [ ] **Step 1: Write the failing tests.** + - `test_reader_prompt_is_blind_to_steps_and_result`: + - it contains the target `q`, `OUTSIDE_RULE`, every file path, `"gap"` and `"part"`; + - it contains no step text and no result text from a real dev target. + - `test_chain_reply_tolerates_shapes`: + - valid JSON round-trips; + - `{"steps": "x"}` gives `steps == []`; + - `"no json"` gives `{"steps": [], "result": "", "gap": ""}`; + - step items that are not dicts are dropped. + - `test_chain_text_numbers_steps`: two steps give lines starting `1.` and `2.`, each with its part and idea, then a `Result:` line. + - `test_read_chain_uses_copies_and_leaves_builds_untouched`. Use a tmp builds folder with `work//a/explanation.md` and a stub `agent.ask` that captures `read=` and returns a chain. Assert: + - every `read` folder lies outside `builds`; + - the saved file exists at the path above; + - a second call makes no `agent.ask` call; + - the sha256 of every file under `builds` is unchanged. + - `test_read_chain_skips_a_build_with_no_renderings`: there is no `agent.ask` call, and the result has `"missing": True` and `steps == []`. + - The existing `tests/test_artifact_eval.py` must still pass. +- [ ] **Step 2: Run them to verify they fail.** + Run: `uv run pytest tests/test_rederive.py -q` + Expected: the new tests FAIL with `AttributeError`. +- [ ] **Step 3: Implement.** + - The reader prompt states, in order: + - the reader line; + - "Read ONLY these files" and the file list; + - the `.html` rule from `ae.reader_prompt`; + - the question; + - "work it out step by step from the explanation"; + - per step: what it establishes (`step`), the part it rests on (`part`: a heading, paragraph, figure or diagram, a page control, an earlier step number, or `background`), and that part's idea in its own words (`idea`); + - `OUTSIDE_RULE`; + - "if the explanation lacks an idea you need, say which in `gap` and stop there"; + - the JSON shape. + - Reuse `ae.READER_SYSTEM`. + - Make copies in `tempfile.mkdtemp()` and remove them afterwards. + - Retry the call once if `steps` comes back empty and `gap` is empty, as `ae.read` does. +- [ ] **Step 4: Run the tests to verify they pass.** + Run: `uv run pytest tests/test_rederive.py tests/test_artifact_eval.py -q` + Expected: all pass. +- [ ] **Step 5: Commit.** + `git add -u && git commit -m "rederive: the reader works the target out and cites the parts it rests on"` + +### Task 4: The grader and the scores + +**Files:** modify `explainer_kit/rederive.py` and `tests/test_rederive.py`. + +**Interfaces:** +- Consumes: `chain_text`, `OUTSIDE_RULE`, `ae.shown_text`, `ae.renderings`, and `ae.GRADER_SYSTEM`. +- Produces: + - `MARKS = ("supported", "unsupported", "missing", "stated")`; + - `grader_text(path: Path) -> str`; + - `grade_prompt(subject: dict, reader: str, entries: dict[str, dict]) -> str`, where `entries` maps a label to `{"text": str, "chain": str}`; + - `marks(raw: object, kinds: list[str]) -> list[str]`; + - `tally(marks: list[str], kinds: list[str], right: bool, stated: bool) -> dict[str, int]`, with keys `supported, made, counted, stated, right, results, results_stated`; + - `pass_scores(tallies: dict[str, list[dict]]) -> dict[str, dict]`. It maps a condition to `{"supported": float, "raw": float, "result": float | None}`, pooled over subjects; + - `separations(passes: list[dict[str, dict]]) -> list[tuple[str, str]]`; + - `grade_subject(subject: dict, key: dict[str, str], reader: str, builds: Path, out: Path, k: int, model: str | None) -> dict`. It saves `out/"rederive"/f"pass-{k}"/"grades"/f"{id}.json"` as `{"key": key, "grades": {cond: {"marks": [...], "result_right": bool, "result_stated": bool}}}`. + +- [ ] **Step 1: Write the failing tests.** + - `test_grader_text_includes_page_scripts`. `

Drag the slider

` gives text containing both `"Drag the slider"` and `"const loss"`. A `.md` file gives the same as `ae.shown_text`. + - `test_grade_prompt_is_blind_and_carries_the_rule`. The prompt contains: + - `OUTSIDE_RULE`; + - each expected step with its kind; + - the result; + - `"=== Explanation A ==="` and the chain text; + - the words `"any order"` and `"stated"`. + + It does not contain `"principles"` or `"skill"`, in any case. + - `test_marks_are_normalised`: + - `marks(["supported", "stated", "bogus"], ["block", "block", "derived", "derived"]) == ["supported", "supported", "missing", "missing"]`; + - `marks("x", ["block"]) == ["missing"]`; + - extra marks are dropped. + - `test_tally`: + - `tally(["supported", "unsupported", "stated", "missing"], ["block", "derived", "derived", "derived"], True, False)` gives `{"supported": 1, "made": 2, "counted": 3, "stated": 1, "right": 1, "results": 1, "results_stated": 0}`; + - with `stated=True`, it gives `right == 0`, `results == 0` and `results_stated == 1`. + - `test_pass_scores_pool_subjects`. For condition `a`, tallies `{"supported": 1, "made": 2, "counted": 2, …, "right": 1, "results": 1}` and `{"supported": 2, "made": 2, "counted": 3, …, "right": 0, "results": 1}` give supported `0.6`, raw `0.8` and result `0.5`. A condition with `results == 0` everywhere gives `result is None`. + - `test_separations_need_every_pass`. + + ```python + separations([ + {"a": {"supported": .7}, "b": {"supported": .5}, "c": {"supported": .6}}, + {"a": {"supported": .7}, "b": {"supported": .5}, "c": {"supported": .4}}, + {"a": {"supported": .6}, "b": {"supported": .6}, "c": {"supported": .3}}, + ]) == [("a", "c")] + ``` + - `test_grade_subject_maps_labels_back_and_fills_missing`. Use a stub grader reply that grades label `A` only: + - the saved grades are keyed by condition; + - the condition whose label is missing gets all `missing` marks and `result_right: False`; + - a rerun makes no call. +- [ ] **Step 2: Run them to verify they fail.** + Run: `uv run pytest tests/test_rederive.py -q` + Expected: the new tests FAIL. +- [ ] **Step 3: Implement.** + - `grader_text`: for `.html`, `ae.shown_text(path)` plus the text of each `` under a `--- page script ---` line; otherwise `ae.shown_text(path)`. Each explanation is truncated at `ae.TEXT_LIMIT`. + - The grader prompt states: + - the subject `q`, the reader, and the target `q`; + - the expected steps, numbered, with kind, and the expected result; + - `OUTSIDE_RULE`; + - "the expected steps are a checklist of ideas: credit a step when the chain establishes the same idea in any order, merged, split, or by an equivalent route"; + - the four marks as the PRD defines them, adding that `stated` applies to derived steps only and that building blocks are meant to be in the explanation; + - `result_right`, and `result_stated` (the explanation states the result outright); + - the reply shape `{"grades": {"