Repository navigation
test(review): add the four decoy fanout reviewer-diversity eval cases #12144
Description
Parent #7584's reviewer-diversity eval (Q134, Q160, D101; approved as recommended by the owner on 2026-10-15, poteto-review decision row 99) runs 12 fixtures under four arms. #12139 adds the arm wiring, the graders and four reused-scaffold fixtures, and describes the case layout under plugins/review/evals/. This slice adds the four decoys from the approved Q160 design, in that layout: correct code that looks wrong, which measures each arm's false positives.
Done when
-
plugins/review/evals/gains fourfanout-diversity-*decoy fixtures, each a correct change that looks wrong (for example, a null check removed because a type rules the value out). - Each decoy's ground-truth file reads
nonefor the planted defect and names the finding a reviewer would wrongly raise, in the shape test(review): add the fanout reviewer-diversity eval wiring and its four reused-scaffold cases #12139 uses. - Each fixture has one case directory per arm and the three graders, each grader file following its template in
plugins/review/evals/fanout-diversity.mdand differing from test(review): add the fanout reviewer-diversity eval wiring and its four reused-scaffold cases #12139's only in the decoy's text, and each grader has asamples/<grader>.jsonwith at least one pass and one fail entry. - Each case uses
schema_version: "1.1"incase.yamland setsruns: 3, as test(review): add the fanout reviewer-diversity eval wiring and its four reused-scaffold cases #12139's cases do. - The diff edits no file under
plugins/review/skills/and no file underplugins/review/agents/. - A
.changes/review/*.mdfragment withbump: nonedescribes the cases. - The PR's CI is green.
Out of scope
- The new defect fixtures (a sibling slice), and running the eval.
Parent: #7584
Activity
github-actions commentedon Nov 2, 2026
github-actions commentedon Nov 2, 2026
melodic-automation-lanes commentedon Nov 2, 2026
The work-resolve-issue run that
agent-readystarted waited in its slot past a change to its lane workflow files on main, so it stopped before any model turn. The run's report-outcome removed and addedagent-readyback to start a fresh run on main's workflow files. If this run ends the same way, it re-fires the issue again, waiting longer each time.github-actions commentedon Nov 2, 2026
github-actions commentedon Nov 2, 2026