Repository navigation
test(review): run the fanout reviewer-diversity eval and post its probe result #12151
Description
Parent #7584's done-when reads "The fanout eval slice carries a Probe result: comment before the act-on cap merges"; this is that slice. It runs the cases #12139, #12143 and #12144 add, after #12138 gives fanout's report its ### Spec and ### Standards headings. The owner approved the Q160 eval as recommended on 2026-10-15 (poteto-review decision row 99), and its decision rule was fixed before the run, as stated below. The parent's decisions read "act-on cap per heading (setting from the Q160 eval)" (D101), and the eval's rule yields no cap number. The Issue Triage session decided, under the owner's standing direction and within D101 (not an owner decision), that this run also reports, per heading, the largest position of the planted defect, and that a follow-up child sets the default cap from that number: the smallest cap that keeps every planted defect inside "act on", the bar Lauren Tan's PR #100 used ("the smallest version that kept every planted dimension at full panel recall"). The lane cannot run claude plugin eval, so a host with the claude CLI runs this. The eval passes are model usage under the Claude subscriptions or the accounts' promotional API credits, which the owner's standing direction calls "Not cost, so spend it freely" (docs/contracts/loop-lane/README.md:351); each pass still sets a --max-cost-usd ceiling.
Done when
- A one-fixture pilot runs for each arm at
--runs 1before the full run.- When the Codex arms cannot run inside the eval sandbox, they run outside the harness on the same fixture branches, are graded with the same grader prompts, and are labeled lower rigor.
-
claude plugin evalruns everyfanout-diversity-*case at three runs per arm, pluslensonce with no plugin as the floor. Each pass sets a--max-cost-usdceiling. - Before any score counts, each
type: llmgrader passes the/evals:plugin-evaljudge calibration (90% agreement on its labeled samples) with an Opus judge. Only VALID runs count. - A comment on this issue whose first line starts with
Probe result:gives:- the CLI version and the ceiling used;
- per arm: VALID run count, recall (the share of defect-fixture runs naming the planted defect at medium or higher) and high-or-critical false positives per run;
- the defects only one arm caught;
- per arm and per heading (
### Spec,### Standards), the largest position, counted from 1 within that heading, at which the planted defect appears across VALID runs, ornonewhen no fixture plants a defect of that heading's axis; - a verdict for each rule:
- (2) an arm replaces
lensas the default only if its recall beats L by at least 10 points, with a noise-report interval that excludes 0, and its high-or-critical false positives per run exceed L's by no more than 0.5; - (3) add
/codex:reviewto the default sweep if L+X beats L by the same rule; - (4) if M does not beat L, record that same-vendor model diversity adds nothing measurable here, and that the panel's placement moves from fanout's
modelmode to the Codex reviewer; - (5) when recall is within noise, the cheaper arm wins.
- (2) an arm replaces
- The probe result is also committed: a PR adds the
Probe result:verdict line to eachfanout-diversity-*case's description, adds a.changes/review/*.mdfragment withbump: none, and closes this issue. - When a verdict changes the
fanout_diversitydefault, adds Codex to the default sweep, or moves the panel's placement, an issue for that change is filed as a sub-issue of review: spec-first fanout and ratchet move to toolchain #7584, withParent: #7584in its body, and is linked in the comment. - An issue that adds the per-heading act-on cap setting, with its default taken from the per-heading positions above (for the arm the verdict keeps as default), is filed as a sub-issue of review: spec-first fanout and ratchet move to toolchain #7584 with
Parent: #7584in its body, and is linked in the comment. A heading reportednoneis named in it as having no evidence.
Out of scope
- Setting an act-on cap value in this slice.
- Changing fanout's behavior in this slice.
Parent: #7584
Metadata
Metadata
Assignees
Labels
Type
Fields
Work status
Activity
kyle-sexton commentedon Nov 2, 2026
Carried from #12139 by the Issue Triage session: #12169 (c3bf4c1) left one item of #12139's arm criterion open, and it belongs with this host run because a
claude plugin evalrun has no Codex login.mx,lx) of thefanout-diversity-*cases runnable: add the Codex plugin directory to those cases' plugins and give the run a Codex login, or record inplugins/review/evals/fanout-diversity.mdthat the host run uses another harness for them (test(review): add fanout reviewer-diversity eval wiring and four reused-scaffold cases #12169 "Remaining work").github-actions commentedon Nov 2, 2026
melodic-automation-lanes commentedon Nov 2, 2026
The work-resolve-issue run that
agent-readystarted waited in its slot past a change to its lane workflow files on main, so it stopped before any model turn. The run's report-outcome removed and addedagent-readyback to start a fresh run on main's workflow files. If this run ends the same way, it re-fires the issue again, waiting longer each time.7 remaining items
github-actions commentedon Nov 2, 2026
github-actions commentedon Nov 2, 2026
melodic-automation-lanes commentedon Nov 2, 2026
The work-resolve-issue lane stopped on this issue (
host-only): the run reports the work needs a host run the lane cannot make.It added
host-runand removedagent-ready: an agent run on a host builds it, and it waits on no decision from a person.In the run's words:
What the run says it did not do:
claude plugin evalruns, grader judge calibration, the Probe result comment, the committed verdict lines and .changes/review fragment, and the follow-up sub-issues of ＃7584: all need the host eval runRun: https://github.com/melodic-software/claude-code-plugins/actions/runs/41408649137
To hand it back to the lane, remove
host-runand addagent-readyagain.github-actions commentedon Nov 2, 2026
Intake triage did not triage this issue after 2 dispatches, so the sweep no longer dispatches it and labeled it needs-triage for a maintainer. Triage it by hand, and read the runs for why intake did not:
github-actions commentedon Nov 2, 2026
Proposed work class: work-class: scoped
Basis: automated intake triage; the lanes App applied this class.
Size: Medium
Complexity: Medium
Uncertainty: Knowable
Flags: none
Blocked by: none
Understood: this slice runs the
fanout-diversity-*evals on a host with theclaudeCLI. It then posts aProbe result:comment with recall, false positives, per-heading defect positions and rule verdicts. It also commits the verdict lines in a PR and files the follow-up issues. The host run is a done-when item that an unattended lane run can't do, so admission routes it tohost-run. Nothing is missing from the issue text. The eval passes draw on prepaid or subscription usage, so I've raised no cost flag. I found no likely duplicates.Automated triage, applied by the lanes App.
kyle-sexton commentedon Nov 2, 2026
kyle-sexton commentedon Nov 2, 2026
Probe result: L and M tie at 24/24 recall;
lensstays the default; the Codex arms are not measurable under codex 1.0.6.Run. Claude Code 2.1.316,
claude plugin evalover the 24 L and M cases, three runs per case, Opus subject and Opus judge,--scaffold --allow-tools Bash Write Workflow,TMPDIRunder/run/user/<uid>(#12422: under/tmpor/var/tmpthe run's Bash cannot write its workspace). Ceilings: L pass 150 USD (spent 73.63, including the no-plugin floor), M pass 100 USD (spent 45.78). One-fixture pilot at--runs 1for each arm first.Judge calibration.
calibrate-judge.py buildskips file-focus graders, so each labeled sample was written to.work/eval/findings.mdby a scaffold and the unchanged grader judged it: 120 labels (the original 48 plus 72 borderline samples now committed) x 3 runs, Opus judge, 360/360 agreement, every one of the 24 graders 100%.lens)model: opus, sonnet)### Spec1,### Standards1; M### Spec1,### Standards1. It landed under Spec in 33 of 48 runs and under Standards in 15, always at report rank 1. No fixture lacks a heading, so neither heading readsnone.Verdicts.
lensstays the default./codex:reviewincluded, setsdisable-model-invocation: true, so the model cannot run fanout's Codex surface (owner decision, 2026-11-02). fix(review): dispatch fanout's Codex surface through a path the model may run #12495 makes it runnable.Grader fixes found in the run (committed in the PR): the arm check required the layer label
(docs/conventions/review.yaml), but fanout's spec (SKILL.md:66) fixes onlyfanout_diversity: <value> (<layer>)and runs often write(repository); 7 of 36 M runs that ranmodelmode with two tiers failed on that wording, and were regraded VALID offline with the corrected pattern. Without--allow-tools Workflow, every arm silently ran fanout's main-thread fallback.Follow-ups under #7584: #12503 (act-on cap, default 1 per heading), #12504 (panel placement to the Codex reviewer, blocked by #12495), #12505 (harder fixtures and the two ground-truth axis labels).
test(review): lead the fanout-diversity verdict with its marker and s…
kyle-sexton commentedon Nov 2, 2026
Addendum to the probe result: what the decoys show below CRITICAL.
The false-positive figure above counts only CRITICAL findings. Every IMPORTANT or CRITICAL finding on the 4 decoy fixtures (24 reports) was then checked against the fixture's code. One agent labeled them; the labels had no second review.
not_defectsclaimawaitremoves theloadProfileframe from async stack traces (return-promise). No request gets a different response.Correction to the cap evidence: every fixture plants one defect in a small diff, so the defect landing at position 1 is close to guaranteed and does not support a cap of 1. #12503 is now blocked by #12505, which adds multi-defect fixtures.
kyle-sexton commentedon Nov 2, 2026
Done-when met or decided: one-fixture pilots for each arm, full L and M passes at three runs per case with the no-plugin floor and ceilings, Opus judge calibration (360/360), the
Probe result:comment (#12151 (comment), addendum 6589390007), verdict lines andbump: nonefragment in #12507, and follow-ups #12503, #12504 and #12505 under #7584. The X arms were recorded as not measurable by owner decision (2026-11-02). #12514 has since made Codex dispatchable, and #12495 carries their rerun; its result updates rule (3).kyle-sexton commentedon Nov 3, 2026
Probe result: rule (3) is not met. Codex adds no measurable recall at ceiling, so
/codex:reviewstays out of the default sweep. Lower rigor: the X arms ran outsideclaude plugin eval.Run. Claude Code 2.1.316, codex plugin 1.0.6, after #12514. One
claude -pper run (Opus subject, three runs per fixture per arm, 72 runs), on the same fixtures and scaffolds as L and M, with the host's own Codex login: a login copied into a run's home would be readable by the model under test. Ceiling 250 USD for the subject runs, 125.67 spent; Codex's own usage is not in that figure. The unchangedrecall.mdandfalse-positive.mdgraded each report throughclaude plugin evalwith an Opus judge: byte-identical to the L and M arms' (cmp), the same 24 graders calibrated at 360/360.A run counts as VALID only when the arm check matched and the trace holds the script's
codex: ran. The two checks agreed on all 72.sort-columnduplication finding fromcode-reviewerandarchitecture-guardian, and 2 are frommodel:tiers.stock-race, one M+X run, from the adversarial review): no regression test covers two concurrent reservations. True, since the fixture has no tests, but a coverage preference with no failing input (the addendum's labels; one agent, no second review).price-centsL+X index 0, printedcodex: skipped (Codex CLI not logged in), and the arm check correctly refused it. It was the first run after the host freeze and the driver restart at 23:37Z; why its login check failed is UNVERIFIED. It was not rerun, so that cell has n=2.Verdicts.