Skip to content

test(review): add the fanout reviewer-diversity eval wiring and its four reused-scaffold cases #12139

Closed

Description

@kyle-sexton

Parent #7584's done-when reads "The fanout eval slice carries a Probe result: comment before the act-on cap merges", and its decisions say the reviewer-diversity eval "(12 diffs, 4 arms, 3 runs) settles the lens default, panel placement and the act-on cap" (Q134, Q160, D101). The owner approved the Q160 eval as recommended on 2026-10-15 (poteto-review decision row 99). This slice adds the arm wiring, the graders and the first four of the 12 fixtures; two later slices of #7584 add the four new defect fixtures and the four decoys in the same shape, and a host-run slice runs them, because the issue lane cannot run claude plugin eval. The design below is the approved Q160 text. On main, /review:fanout run-everything spreads by fanout_diversity: lens, the default, runs one reviewer agent per concern, and model sends one prompt per fan-out model tier (plugins/review/skills/fanout/SKILL.md "Arguments", plugins/review/reference/config.md:13). The only cross-vendor reviewer is the Codex orchestrator surface.

Done when

  • plugins/review/evals/ gains fanout-diversity-* cases over four fixtures that reuse the scaffolds of guard-removal-null-user, guard-removal-empty-ratings, guard-removal-short-title and qg-security-search-sqli, with the prompt changed to /review:fanout run-everything.
  • plugins/review/evals/fanout-diversity.md states, for the later slices to follow: the case directory pattern fanout-diversity-<fixture>-<arm> (arm one of l, m, mx, lx, one case directory per arm, since a case carries one prompt, one plugins list and one committed review.yaml); the ground-truth file's name and fields; the three grader file names; and a template for each grader, in which only the planted-defect text taken from the ground-truth file varies between fixtures (type: llm graders take criteria and focus, so the defect is written into the criteria). The arm-check regex grader is byte-identical across fixtures of one arm.
  • Each fixture has a ground-truth file that names its one planted defect and that defect's axis (spec when the change fails its stated purpose, else standards), or none for a decoy.
  • Every fixture can run under each of the four arms: lens (L), model (M), model plus Codex (M+X) and lens plus Codex (L+X).
    • Each arm's scaffold commits docs/conventions/review.yaml with that arm's fanout_diversity on the base branch, so the reviewed diff never edits it.
    • The M arms list the multi-agent plugin beside review in the case's plugins list.
    • The X arms name the Codex surface in the prompt.
  • Each case has three graders:
    • a regex grader on the report's fanout_diversity: line and ## Surfaces section, which checks that the run used its arm (fanout_diversity: model for M arms, the Codex surface listed for X arms), so a run that fell back counts as invalid;
    • a type: llm recall grader on the persisted findings file, which passes when the file names the planted defect at medium severity or higher, judged against the ground-truth file;
    • a type: llm false-positive grader, which counts high or critical findings the ground truth marks wrong.
  • Each grader has a samples/<grader>.json with at least one pass and one fail entry, in the shape of guard-removal-null-user/samples/.
  • Each case uses schema_version: "1.1" in case.yaml and sets runs: 3 in prompt.md frontmatter, as guard-removal-null-user does.
  • The diff edits no file under plugins/review/skills/ and no file under plugins/review/agents/.
  • A .changes/review/*.md fragment with bump: none describes the cases.
  • The PR's CI is green.

Out of scope

Parent: #7584

Activity

  1. added
    priority: mediumReal value, no hard deadline; normal backlog flow.
    status: readyTriaged, unblocked, and fully specified; eligible to pick up.
    agent-readyFully specified and briefed; eligible for autonomous pickup from the frontier.
    on Nov 1, 2026
  2. added theissue type on Nov 1, 2026
  3. added
    work-class: scopedA briefed fix or small feature; blast radius bounded by the brief, tests exist.
    on Nov 1, 2026
  4. github-actions commented on Nov 1, 2026

    @github-actions
    Contributor
    No description provided.
  5. melodic-automation-lanes commented on Nov 2, 2026

    @melodic-automation-lanes
    Contributor

    The lanes App's pull request #12169 merged with Refs:, so this issue stays open and nothing starts the lane on it again. work-unblock removed status: ready and agent-ready, so the hourly triage sweep sends it back to intake triage. Admission then parks it at status: confirmed for a check of what that pull request left, until a **Decision recorded.** comment says what is left or someone closes it.

  6. removed
    status: readyTriaged, unblocked, and fully specified; eligible to pick up.
    agent-readyFully specified and briefed; eligible for autonomous pickup from the frontier.
    on Nov 2, 2026
  7. kyle-sexton commented on Nov 2, 2026

    @kyle-sexton
    MemberAuthor

    Delivered: closing as completed. Checked by the Issue Triage session against origin/main: #12169 (c3bf4c1) adds the 16 fanout-diversity-<fixture>-<arm> cases over the four reused scaffolds, plugins/review/evals/fanout-diversity.md, one ground-truth file per fixture, three graders per case with samples, and a bump: none review fragment; it edits nothing under plugins/review/skills/ or plugins/review/agents/, and the scaffolds are 100755 on main. One item of the arm criterion stays open: the Codex arms (mx, lx) cannot run under claude plugin eval without a Codex login, so that item moved to the host-run eval slice #12151.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    priority: mediumReal value, no hard deadline; normal backlog flow.work-class: scopedA briefed fix or small feature; blast radius bounded by the brief, tests exist.

    Type

    Fields

    No fields configured for Task.

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions