Repository navigation
Description
Parent #7584's done-when reads "The fanout eval slice carries a Probe result: comment before the act-on cap merges", and its decisions say the reviewer-diversity eval "(12 diffs, 4 arms, 3 runs) settles the lens default, panel placement and the act-on cap" (Q134, Q160, D101). The owner approved the Q160 eval as recommended on 2026-10-15 (poteto-review decision row 99). This slice adds the arm wiring, the graders and the first four of the 12 fixtures; two later slices of #7584 add the four new defect fixtures and the four decoys in the same shape, and a host-run slice runs them, because the issue lane cannot run claude plugin eval. The design below is the approved Q160 text. On main, /review:fanout run-everything spreads by fanout_diversity: lens, the default, runs one reviewer agent per concern, and model sends one prompt per fan-out model tier (plugins/review/skills/fanout/SKILL.md "Arguments", plugins/review/reference/config.md:13). The only cross-vendor reviewer is the Codex orchestrator surface.
Done when
-
plugins/review/evals/gainsfanout-diversity-*cases over four fixtures that reuse the scaffolds ofguard-removal-null-user,guard-removal-empty-ratings,guard-removal-short-titleandqg-security-search-sqli, with the prompt changed to/review:fanout run-everything. -
plugins/review/evals/fanout-diversity.mdstates, for the later slices to follow: the case directory patternfanout-diversity-<fixture>-<arm>(arm one ofl,m,mx,lx, one case directory per arm, since a case carries one prompt, one plugins list and one committedreview.yaml); the ground-truth file's name and fields; the three grader file names; and a template for each grader, in which only the planted-defect text taken from the ground-truth file varies between fixtures (type: llmgraders takecriteriaandfocus, so the defect is written into the criteria). The arm-check regex grader is byte-identical across fixtures of one arm. - Each fixture has a ground-truth file that names its one planted defect and that defect's axis (
specwhen the change fails its stated purpose, elsestandards), ornonefor a decoy. - Every fixture can run under each of the four arms:
lens(L),model(M),modelplus Codex (M+X) andlensplus Codex (L+X).- Each arm's scaffold commits
docs/conventions/review.yamlwith that arm'sfanout_diversityon the base branch, so the reviewed diff never edits it. - The M arms list the
multi-agentplugin besidereviewin the case's plugins list. - The X arms name the Codex surface in the prompt.
- Each arm's scaffold commits
- Each case has three graders:
- a regex grader on the report's
fanout_diversity:line and## Surfacessection, which checks that the run used its arm (fanout_diversity: modelfor M arms, the Codex surface listed for X arms), so a run that fell back counts as invalid; - a
type: llmrecall grader on the persisted findings file, which passes when the file names the planted defect at medium severity or higher, judged against the ground-truth file; - a
type: llmfalse-positive grader, which counts high or critical findings the ground truth marks wrong.
- a regex grader on the report's
- Each grader has a
samples/<grader>.jsonwith at least one pass and one fail entry, in the shape ofguard-removal-null-user/samples/. - Each case uses
schema_version: "1.1"incase.yamland setsruns: 3inprompt.mdfrontmatter, asguard-removal-null-userdoes. - The diff edits no file under
plugins/review/skills/and no file underplugins/review/agents/. - A
.changes/review/*.mdfragment withbump: nonedescribes the cases. - The PR's CI is green.
Out of scope
- The new defect fixtures and the decoys (later slices of review: spec-first fanout and ratchet move to toolchain #7584).
- Running the eval.
- Changing the
fanout_diversitydefault, adding Codex to the default sweep, or setting an act-on cap.
Parent: #7584
Activity
github-actions commentedon Nov 1, 2026
melodic-automation-lanes commentedon Nov 2, 2026
The lanes App's pull request #12169 merged with
Refs:, so this issue stays open and nothing starts the lane on it again. work-unblock removedstatus: readyandagent-ready, so the hourly triage sweep sends it back to intake triage. Admission then parks it atstatus: confirmedfor a check of what that pull request left, until a**Decision recorded.**comment says what is left or someone closes it.kyle-sexton commentedon Nov 2, 2026
Delivered: closing as completed. Checked by the Issue Triage session against origin/main: #12169 (c3bf4c1) adds the 16
fanout-diversity-<fixture>-<arm>cases over the four reused scaffolds,plugins/review/evals/fanout-diversity.md, one ground-truth file per fixture, three graders per case with samples, and abump: nonereview fragment; it edits nothing underplugins/review/skills/orplugins/review/agents/, and the scaffolds are 100755 on main. One item of the arm criterion stays open: the Codex arms (mx,lx) cannot run underclaude plugin evalwithout a Codex login, so that item moved to the host-run eval slice #12151.