test(review): add fanout-diversity fixtures that separate the arms

Parent: #7584

Problem

In #12151 both lens and model named every planted defect in every run (24/24 each), so the suite is at ceiling and cannot tell the arms apart: M minus L +0.01, 95% interval -0.03 to +0.05. The ground-truth axis also disagrees with where fanout files two defects: csv-handle-leak and search-sqli are labeled standards but land under ### Spec in most runs.

Done when

At least four new defect fixtures where a pilot shows at least one arm misses the planted defect in some runs (subtler defects, larger diffs, or defects spread across files).
The csv-handle-leak and search-sqli ground-truth axis values are checked against the commit-message claim rule in plugins/review/evals/fanout-diversity.md and corrected where wrong.
At least two fixtures plant two or more defects under one heading, so the per-heading positions can set feat(review): add a per-heading act-on cap to fanout's report #12503's act-on cap.
Decoy runs report IMPORTANT+ findings by label (false, already on main, preference, about the changed lines). Baseline from test(review): run the fanout reviewer-diversity eval and post its probe result #12151: L 9/12 runs and M 7/12 had one; 0 of 31 were false, 19 were preferences.
The L and M arms are rerun over the new fixtures and the result is posted on review: spec-first fanout and ratchet move to toolchain #7584.

Proposed work class: work-class: scoped
Basis: automated intake triage; the lanes App applied this class.

Size: Medium
Complexity: Medium
Uncertainty: Knowable
Flags: none
Blocked by: none

I understand this as a test-fixture task under #7584. The #12151 suite is at ceiling (lens and model both 24/24), so it can't separate the arms. The work is in four parts:

Add at least four harder defect fixtures.
Re-check the axis labels for csv-handle-leak and search-sqli against the commit-message claim rule.
Add at least two fixtures with two or more defects under one heading.
Report decoy IMPORTANT+ findings by label, then rerun the L and M arms and post the result on review: spec-first fanout and ratchet move to toolchain #7584.

The done-when items are specific, so I'm asking nothing further. The pilot and rerun need model runs. #12503 is related, since the multi-defect fixtures feed its act-on cap. I found no duplicate.

_{Automated triage, applied by the lanes App.}

The triage sweep dispatched intake triage for this issue (attempt 1 of 2).

No description provided.

The work-resolve-issue lane stopped on this issue (host-only): the run reports the work needs a host run the lane cannot make.
It added host-run and removed agent-ready: an agent run on a host builds it, and it waits on no decision from a person.

In the run's words:

host-only: four of the five done-when items need claude plugin eval runs of the fanout-diversity arms: a pilot to pick fixtures where an arm misses, decoy runs reported by label, and the L and M rerun posted on ＃7584. This lane's Bash runs only its named wrappers and cannot start an eval run or comment on ＃7584. Writing the new fixtures without a pilot would ship cases the issue's own criterion may reject, so no fixture was written.

What the run says it did not do:

At least four new defect fixtures picked by a pilot where an arm misses: the pilot is a claude plugin eval host run this lane cannot start
At least two fixtures that plant two or more defects under one heading, for the act-on cap in ＃12503: these come out of the same pilot-gated fixture set
Decoy runs reporting IMPORTANT+ findings by label (false, already on main, preference, about the changed lines): needs eval runs
Rerunning the L and M arms over the new fixtures and posting the result on ＃7584: needs a host eval run and an issue comment, and this lane can do neither
Tidy, review and the pull-request prep were skipped because there is no diff

Run: https://github.com/melodic-software/claude-code-plugins/actions/runs/41540441285

To hand it back to the lane, remove host-run and add agent-ready again.

Claim. Backlog HITL host-run worker is taking this now.

docs(review): say the two-defect fixtures share an axis, not a heading

Claim. Backlog HITL host-run worker is taking this now.

Navigation Menu

Repository navigation

test(review): add fanout-diversity fixtures that separate the arms #12505

Problem

Done when

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

test(review): add fanout-diversity fixtures that separate the arms #12505

Description

Problem

Done when

Activity

github-actions commented on Nov 2, 2026

github-actions commented on Nov 2, 2026

github-actions commented on Nov 2, 2026

github-actions commented on Nov 2, 2026

melodic-automation-lanes commented on Nov 2, 2026

kyle-sexton commented on Nov 3, 2026

kyle-sexton commented on Nov 3, 2026

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions