Skip to content

test(songwriting): fix three eval-suite defects in the next measured pass #8431

Open

Description

@kyle-sexton

Refs: #8427

Summary

Review of #8427 found three defects in the songwriting eval suite. The suite was frozen before the after-version runs (plugins/playbooks/skills/skill-authoring/reference/rewrite-a-plugin.md, step 2: "change nothing in it after the first after-version run"), and the PR's decision table reports scores from those runs, so the fixes wait for the next measured pass on this plugin. Each was verified on head 92362dac.

Findings

  1. dg-variations-arrow-blocks cannot run the rubric the skill requires. prompt.md allows only Skill, Read, Glob, Grep. skills/diagnose/SKILL.md:60-62 makes the rubric passes non-skippable and has the countable passes run as commands (datamuse.sh syllables, a word-frequency count). Without Bash the case can pass on formatting while the mandatory checks never run. Fix: grant the tools the rubric needs, with fixtures or mocks for the network call, then re-measure. Thread
  2. mp-chorus-stress-count / raw-count-not-balance passes a reply that endorses the raw count. Its first alternative \braw[- ]syllables?\b matches any mention. Checked: "The raw syllable counts, 8/9/9/8, show the chorus is nicely balanced" matches. Fix: require a negation on that alternative, and add a fail sample that endorses the raw count. Thread
  3. rh-window-menu-no-winner / eight-plus-candidates counts lines that are not candidates. Table header rows and bold heading lines count toward 8. Checked: a table with a header row and 7 candidate rows matches. Fix: count candidate rows only, and add a fail sample with 6-7 candidates plus headings. Thread

The PR body's "Ready-step review findings left in place" lists the earlier grader findings for the same pass.

Acceptance

  • Each grader fix accepts every reply that meets the case's expected outcome and rejects a reply written to fail, proved by samples.
  • The fixed suite is frozen again and the affected skills are re-measured before and after under the rewrite protocol.

Activity

  1. added
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Oct 19, 2026
  2. github-actions commented on Oct 19, 2026

    @github-actions
    Contributor

    Proposed work class: work-class: scoped
    Basis: automated intake triage; the lanes App applied this class.

    Size: Medium
    Complexity: Medium
    Uncertainty: Knowable
    Flags: none
    Blocked by: none

    Understood: the remaining work is a one-line path fix in diagnose (before and after copies), then re-freezing the songwriting eval suite and re-measuring five cases on both models. The recorded decision names the steps, and the re-measure is a host run.

    Decision recorded.

    This says what the 2026-10-31 host run and #11569 left. Agents decided this on evidence under standing direction §1. The issue stays host-run: what is left is a measured run.

    [quote truncated; the full decision is at the link]
    #8431 (comment)

    The decision settles the open questions, so nothing further is asked. The host-run step is routed by admission, not held for a person.

    Automated triage, applied by the lanes App.

  3. added
    priority: lowNice-to-have, cosmetic, or speculative; opportunistic.
    and removed
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Oct 19, 2026
  4. kyle-sexton commented on Oct 19, 2026

    @kyle-sexton
    MemberAuthor

    Owner decision (2026-10-19): change the case, not the skill.

    mt-loneliness-labeled-menu expects a menu even when the user asks for one metaphor. Both the before and after versions of metaphor honor "just give me one" and skip the menu, and that is the right behavior. Rewrite the case to expect one metaphor with its family-distance label when the user asks for one. Keep a separate case where the user asks for options, which expects the menu.

    Then re-measure metaphor in the next measured pass. It currently ships its before text (#8427).

  5. github-actions commented on Oct 26, 2026

    @github-actions
    Contributor

    The triage sweep dispatched intake triage for this issue (attempt 1 of 2).

  6. added
    status: readyTriaged, unblocked, and fully specified; eligible to pick up.
    work-class: scopedA briefed fix or small feature; blast radius bounded by the brief, tests exist.
    status: needs-decisionAwaiting a human or maintainer judgment call.
    and removed
    status: readyTriaged, unblocked, and fully specified; eligible to pick up.
    on Oct 26, 2026
  7. melodic-automation-lanes commented on Oct 26, 2026

    @melodic-automation-lanes
    Contributor

    Not admitted to the issue lane: it carries host-run. Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).

  8. melodic-automation-lanes commented on Oct 26, 2026

    @melodic-automation-lanes
    Contributor

    The latest triage verdict holds nothing: admission moved the issue to status: confirmed; its comment says why.

  9. kyle-sexton commented on Oct 29, 2026

    @kyle-sexton
    MemberAuthor

    Re-triage. This is not an owner stop. The three grader defects are still on main, and the owner decided the fourth change on 2026-10-19 (comment).

    Checked on main:

    • plugins/songwriting/evals/dg-variations-arrow-blocks/prompt.md allows [Skill, Read, Glob, Grep] only.
    • plugins/songwriting/evals/mp-chorus-stress-count/case.yaml:16 still opens with the bare \braw[- ]syllables?\b alternative.
    • plugins/songwriting/evals/mt-loneliness-labeled-menu still expects a menu.

    Acceptance criteria:

    • plugins/songwriting/evals/dg-variations-arrow-blocks grants the tools the diagnose rubric runs (skills/diagnose/SKILL.md:60-62), with a fixture or mock in place of the network call.
    • plugins/songwriting/evals/mp-chorus-stress-count grader raw-count-not-balance requires a negation on the raw-count alternative, and a new fail sample that endorses the raw 8/9/9/8 count is rejected.
    • plugins/songwriting/evals/rh-window-menu-no-winner grader eight-plus-candidates counts candidate rows only, and a new fail sample with 6-7 candidates plus a header row and bold headings is rejected.
    • plugins/songwriting/evals/mt-loneliness-labeled-menu expects one metaphor with its family-distance label when the user asks for one, and a separate case where the user asks for options expects the menu.
    • /evals:validate passes on the suite, and each changed grader accepts every pass sample and rejects every fail sample offline.

    Checks a person runs: freeze the fixed suite again and re-measure the affected skills (diagnose, metaphor and the others the cases cover) before and after, under the rewrite protocol. Fix 1 makes the suite Bash-granting, so this needs a host with a ready sandbox.

  10. 29 remaining items

  11. added
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    priority: mediumReal value, no hard deadline; normal backlog flow.
    and removed
    priority: lowNice-to-have, cosmetic, or speculative; opportunistic.
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Oct 31, 2026
  12. added
    status: readyTriaged, unblocked, and fully specified; eligible to pick up.
    status: confirmedA maintainer confirmed the item is real and in scope; ready to be worked.
    and removed
    status: readyTriaged, unblocked, and fully specified; eligible to pick up.
    on Oct 31, 2026
  13. kyle-sexton commented on Nov 1, 2026

    @kyle-sexton
    MemberAuthor

    Host run (2026-11-01): metaphor before and after, five cases, Opus and Sonnet. Verdict: keep the before text. Neither model's compare shows the after text non-inferior at the 5-point margin, and the two metaphor cases alone are too few to call. Nothing ships, and no pull request was opened.

    Setup

    • Suite frozen at main 379109c2fc, which holds test(songwriting): accept raw-count dismissals the grader failed #11569 and fix(songwriting): give diagnose the full datamuse.sh path #11648. plugins/songwriting/evals is tree ebd24d91488f. Both copies carry the same five cases (dg-variations-arrow-blocks, mp-chorus-stress-count, rh-window-menu-no-winner, mt-loneliness-labeled-menu, mt-loneliness-one-labeled), with an identical evals digest.
    • Before: main's songwriting plugin. After: the same, with the body of skills/metaphor/SKILL.md replaced by the after text (845 words as posted).
      • The after copy keeps main's description. Slash-invoked cases measure no triggering, and skill-criteria.md step 6 ships only measured parts, so the after text's shorter description still needs its own probe measurement.
      • diff -rq of the two copies shows skills/metaphor/SKILL.md as the only differing file. Both copies sit outside any git repository.
    • Scope: before is main, so diagnose, meter-prosody and rhyme carry identical text in both copies. Their rows are a noise check. They are not the comparison the decision sets out, of the pre-feat(songwriting): concise skill text measured by evals, plus outcome cases and probes #8427 text (1f61e714d9^, with the item 1 path fix applied to the before copy) against the current text, re-run under item 2. This run measures metaphor only.
    • validate-cases.py: exit 0, one WARN (Bash needs --allow-tools).
    • CLI 2.1.316, subscription login. Every leg used the same command, each in its own systemd-run --user --scope -p MemoryMax=8G unit, one at a time:
      claude plugin eval <copy> --model <sonnet|opus> --runs 3 -j 2 --trust-plugin --keep-temp --scaffold --allow-tools Bash --judge-model opus --threshold 0.8 --no-publish
    • TMPDIR was a 31 to 34 character directory on the home filesystem, one per leg. That is above the 24-character bound plugin-eval records. The session started check passed in all 150 traces. Every with-arm dg run in the four counted legs ran Bash, including datamuse.sh ... syllables.
    • Order: Sonnet before, Sonnet after, Opus after, Opus before (A-B-B-A), each pair back to back. All graders are regex or tool_used, so no judge calibration applies.
    • run-validity.py and noise-report.py were taken from main 379109c2fc, which includes fix(evals): warn on denials every run of both arms shares and pass 0-turn forked sessions in run-validity #11787 and fix(evals): run invocation probes without subagents and name the run line in run-validity #11901.
    Leg Copy Model Start (UTC-5) Wall clock costUsd (list) Validity
    L1 before sonnet 04:36 13m05s 6.50 VALID (warnings: skill fired, ceiling)
    L2 after sonnet 04:50 14m36s 6.67 VALID (warning: skill fired)
    L3 after opus 05:05 15m26s 11.89 VALID (warning: skill fired)
    L4 before opus 05:20 13m29s 10.42 INVALID: 1 permission denial (see below)
    L4b before opus 05:34 16m00s 11.80 VALID (warnings: skill fired, ceiling)

    L4 was not counted. Its INVALID line reads: check permission denials: FAIL (1 denial in 1 run), mp-chorus-stress-count with-arm run 1. A Bash sed read of the plugin's context/pat-pattison/research/response-filter.md was "blocked by a deny rule". That run scored 1.000. L4b reran the whole leg with the same flags.

    The skill fired warning means no case has a tool_used: Skill grader. Every prompt slash-invokes its skill.

    Scores (mean with-arm score over 3 runs, VALID legs only)

    Sonnet (L1 before, L2 after)

    Case Skill text Before After Change
    mt-loneliness-labeled-menu metaphor, changed 1.000 1.000 0.000
    mt-loneliness-one-labeled metaphor, changed 0.889 0.889 0.000
    dg-variations-arrow-blocks diagnose, identical 0.933 0.667 -0.267
    mp-chorus-stress-count meter-prosody, identical 0.833 0.750 -0.083
    rh-window-menu-no-winner rhyme, identical 0.917 1.000 +0.083

    Opus (L4b before, L3 after)

    Case Skill text Before After Change
    mt-loneliness-labeled-menu metaphor, changed 0.667 0.833 +0.167
    mt-loneliness-one-labeled metaphor, changed 0.889 0.778 -0.111
    dg-variations-arrow-blocks diagnose, identical 0.800 0.400 -0.400
    mp-chorus-stress-count meter-prosody, identical 0.917 0.833 -0.083
    rh-window-menu-no-winner rhyme, identical 0.833 0.833 0.000

    family-distance-label in mt-loneliness-one-labeled, the grader behind #8427's Sonnet loss, passed 2 of 3 runs on Sonnet in both versions. On Opus it passed 2 of 3 before and 1 of 3 after.

    Noise report, after against before, all five cases

    noise-report.py <after.json> --threshold 0.8 --interval-method normal --grader-agreement --baseline <before.json> --margin 0.05

    Sonnet (L2 against L1):

    with-arm mean: 0.86, 95% interval 0.73 to 0.99 (normal, 5 cases)
    without-arm mean: 0.42, 95% interval 0.14 to 0.70 (normal, 5 cases)
    cost: 6.67 USD for the whole suite (list-price estimate)
    delta (with minus without), paired over 5 cases: +0.44, 95% interval +0.09 to +0.79
    verdict: the interval excludes 0, so the difference is larger than the noise at 5 cases
    with-arm pass count at threshold 0.80: 3 of 5, 95% interval 0.17 to 1.00 (normal)
    without-arm pass count at threshold 0.80: 1 of 5, 95% interval 0.00 to 0.55 (normal)
    no llm grader in this result, so there is no judge agreement to read
    with-arm change (this minus baseline), paired over 5 cases: -0.05, 95% interval -0.17 to +0.06
    without-arm change (this minus baseline), paired over 5 cases: -0.08, 95% interval -0.18 to +0.02
    compare verdict: not shown non-inferior at margin 0.05 (the interval's lower end -0.17 is below -0.05)
    

    Opus (L3 against L4b):

    with-arm mean: 0.74, 95% interval 0.57 to 0.90 (normal, 5 cases)
    without-arm mean: 0.42, 95% interval 0.17 to 0.66 (normal, 5 cases)
    cost: 11.89 USD for the whole suite (list-price estimate)
    delta (with minus without), paired over 5 cases: +0.32, 95% interval +0.04 to +0.60
    verdict: the interval excludes 0, so the difference is larger than the noise at 5 cases
    with-arm pass count at threshold 0.80: 3 of 5, 95% interval 0.17 to 1.00 (normal)
    without-arm pass count at threshold 0.80: 0 of 5, 95% interval 0.00 to 0.00 (normal)
    no llm grader in this result, so there is no judge agreement to read
    with-arm change (this minus baseline), paired over 5 cases: -0.09, 95% interval -0.27 to +0.10
    without-arm change (this minus baseline), paired over 5 cases: -0.08, 95% interval -0.17 to +0.01
    case drop: case dg-variations-arrow-blocks with-arm fell from 0.80 to 0.40, a third or more; name its cause and prove it before shipping
    compare verdict: not shown non-inferior at margin 0.05 (the interval's lower end -0.27 is below -0.05), and 1 case(s) dropped by a third or more
    

    Over the two metaphor cases alone, both models gave compare verdict: n too small to call (cases scored in both results: 2; 3 or more are needed). The with-arm means were 0.94 to 0.94 on Sonnet and 0.78 to 0.81 on Opus.

    Step-6 verdict: keep the before text

    skill-criteria.md step 6 requires non-inferiority at a 5-point margin and no case dropping by a third without a measured cause. Neither holds:

    1. Both suite compares fail the margin. Most of the spread comes from cases whose skill text is byte-identical between the copies (dg -0.267 and -0.400, mp -0.083 on both models). This run cannot separate those swings from the metaphor change, and step 6 gives no rule for dropping them.
    2. The Opus dg case drop has no proven cause.
      • All three after runs came back with two variations, against an expected outcome of three or four. Each run spawned one or more "skeptic" subagents that cut the rest. L4b run 2 also used a skeptic and scored 0.600.
      • Diagnose's text is identical in both copies, so the metaphor change is an unlikely cause. Run-to-run variance is the likely one. That stays a hypothesis: step 6 counts a cause only once restoring the text moves the number, and here no text differs to restore.
    3. The metaphor cases alone are too few to call. On their point estimates, Sonnet is unchanged on both. Opus gains on the menu case (+0.167) and loses on the one-metaphor case (-0.111). Part of the menu-case gain is a grader effect: grader findings 4 and 5 below are both before-version failures.

    This is "not shown non-inferior", not a measured loss. Per the decision, metaphor keeps its before text. Metaphor's result does not hold this issue open.

    Grader findings (spot-read against the kept traces)

    1. dg-variations-arrow-blocks / ran-syllable-count rejects real script runs.

      • The grader failed 9 of 12 with-arm runs: L1 run 2, all of L2 and L3, and L4b runs 1 and 2.
      • All 12 runs called datamuse.sh with syllables in a Bash command. I read the commands of all 9 failing runs. Each one set the path in a variable and then called it (S=<plugin>/context/pat-pattison/scripts/datamuse.sh; for w in ...; do ... "$S" syllables), which input_match does not cover.
      • The grader therefore rejects runs that meet the expected outcome. That lowers dg's scores by 0.2 per failed run and adds most of dg's spread between legs.
    2. mp-chorus-stress-count / raw-count-not-balance still rejects valid dismissals after test(songwriting): accept raw-count dismissals the grader failed #11569. L1 runs 0 and 1 failed on "this method measures line length in stressed syllables, and by that count the chorus is 3/4/4/3, so I wouldn't call it 'nicely balanced'" and "Syllable counts are still the wrong measure for 'balanced'". With-arm passes were 1/3, 0/3, 1/3 and 2/3 across L1, L2, L3 and L4b.

    3. mt-loneliness-one-labeled / family-distance-label misses the word order "medium-distance". L3 run 1 failed on "It's a medium-distance pairing", because the pattern requires distance before the label. The other misses I read (L1 run 1, L2 run 1, L3 run 0) contain no distance wording, so those failures are correct.

    4. mt-loneliness-labeled-menu / finder-question-lists requires the literal word "characteristics".

      • L4b runs 1 and 2 failed on paraphrased finder lists: "First I listed what this loneliness is like, then what else is like that. Both lists are below" under the heading "What else is like that?", and "Step 2: What else feels that way?".
      • The expected outcome asks for both lists ("what characteristics it has, what else has them"), and both replies give them.
    5. mt-loneliness-labeled-menu / no-single-winner matches a choosing guide. L4b run 1 failed on the heading "How I'd choose:", which introduces picks by feeling ("For quiet ache: chore: initialize marketplace scaffold #1 or fix: correct plugin-system inaccuracies and harden references #2"). It crowns no option.

    The verdict above does not depend on these. They are noise sources a re-measure should remove first.

    What remains

    • Widen ran-syllable-count, raw-count-not-balance, family-distance-label, finder-question-lists and no-single-winner to accept the forms quoted above, each with a pass sample for the quoted reply and an existing fail sample still rejected.
    • The decision's item 2 comparison for diagnose, meter-prosody and rhyme (pre-feat(songwriting): concise skill text measured by evals, plus outcome cases and probes #8427 text against current text) was not run here. Its rows above are same-text noise.
    • A metaphor re-measure that can call non-inferiority needs more metaphor cases than these two.
  14. kyle-sexton commented on Nov 1, 2026

    @kyle-sexton
    MemberAuthor

    Claimed by host lane 4 (melo-lap-001 WSL), 2026-11-01T11:40Z: the grader fixes from the 2026-11-01 host run's findings (one PR, offline-validated), then the item 2 comparison for diagnose, meter-prosody and rhyme as a paid host run after that PR merges.

  15. kyle-sexton commented on Nov 1, 2026

    @kyle-sexton
    MemberAuthor

    Item 2 measured pass (lane 4): cannot call, so this issue stays open.

    claude plugin eval, suite frozen at main 1a0da52 (holds #11941). "Before" is the three SKILL.md files at 1f61e714d9^; "after" is main. 3 runs per arm on Sonnet 5.5 and Opus 5.5. All four legs VALID (run-validity.py --runs 3 --scaffolded) with 0 denials.

    Case (skill) Sonnet before Sonnet after Opus before Opus after
    dg-variations-arrow-blocks (diagnose) 1.000 1.000 0.933 0.867
    mp-chorus-stress-count (meter-prosody) 0.833 1.000 0.833 0.833
    rh-window-menu-no-winner (rhyme) 1.000 1.000 0.917 0.917
    • noise-report.py --baseline <before> --margin 0.05: "not shown non-inferior at margin 0.05" on both models. Sonnet after minus before is +0.06 [-0.05, +0.16]; Opus is -0.02 [-0.07, +0.02]. Neither model has a case drop. The without arm (no skill text) moved only within noise, so conditions held.
    • Not a pass under the decision, and not a measured loss.

    What the next pass needs:

    1. mp-chorus-stress-count / raw-count-not-balance gives false negatives in both versions. Six with-arm misses state that the raw 8/9/9/8 count does not show balance, in wording the pattern does not cover. With those corrected, mp scores 1.000 in every leg. Widen the pattern, with samples, before the next freeze.
    2. More cases per skill. With one case each, the interval cannot show non-inferiority at a 5-point margin.
    3. Borderline graders worth a look: rh / tier-labels-span (a closeness word in place of "family") and rh / no-single-winner (two picks for one mood, the shape test(songwriting): widen five eval graders that failed valid replies #11941 exempted for metaphor).
  16. kyle-sexton commented on Nov 2, 2026

    @kyle-sexton
    MemberAuthor

    Suite readied for the next item-2 pass: #12153 merged (head d58acd3). raw-count-not-balance is now an llm grader; no-single-winner (rh-window) and the register and identity graders were fixed through six review rounds and one pr-verify round; diagnose, meter-prosody and rhyme now have two slash-invoked cases each, and the new diagnose case runs with Bash and the offline datamuse stand-in.

    Next (lane 4): calibrate the three llm graders on their samples, run the before-version smoke pass on the three new cases, freeze the suite at main, then the before/after legs on Sonnet and Opus under rewrite-a-plugin.md.

  17. kyle-sexton commented on Nov 2, 2026

    @kyle-sexton
    MemberAuthor

    Item 2 measured pass (lane 4, second round): Sonnet non-inferior, Opus cannot be called, so this issue stays open.

    claude plugin eval, suite frozen at main edef3f69 (holds #12153, #12194, #12209). "Before" is the three SKILL.md files at 1f61e714d9^; "after" is main. Six cases (tags diagnose, meter-prosody, rhyme), 3 runs, with-arm only (--ablation none), --scaffold --allow-tools Bash, Opus judge, legs A-B-B-A. All four legs VALID (run-validity.py --runs 3 --scaffolded). The only warnings were denied reads of the session's own tool-result files, which are not counted, and no skill fired, which no grader checks. Every trace shows the pinned model.

    Case Sonnet before Sonnet after Opus before Opus after
    dg-audit-identity-rhyme 0.667 0.833 0.917 0.833
    dg-variations-arrow-blocks 1.000 1.000 0.933 1.000
    mp-chorus-stress-count 1.000 1.000 1.000 1.000
    mp-sung-line-template 1.000 1.000 0.778 1.000
    rh-release-identity-menu 0.333 0.667 0.833 1.000
    rh-window-menu-no-winner 1.000 1.000 0.917 0.750
    Overall (scored) 0.833 0.917 0.896 0.931
    Cost (list-price estimate) 6.45 USD 6.31 USD 10.92 USD 9.52 USD

    Two-version noise report (noise-report.py --interval-method normal --grader-agreement --margin 0.05, paired over 6 cases, scored numbers; the verdict rule is step 6 of skill-criteria.md):

    • Sonnet: +0.08, 95% interval -0.03 to +0.19: non-inferior at margin 0.05.
    • Opus: +0.03, 95% interval -0.08 to +0.15: not shown non-inferior, because the lower end is below -0.05.
    • No case drops by a third or more on either model.
    • Every llm grader's judge votes were unanimous.

    Why Opus cannot be called. Every failing with-plugin run was read in full. The scored Opus shortfall comes from regex graders that check a meaning by phrase shape and fail replies that meet the case's stated outcome:

    • tier-labels-span gave 2 false negatives (Opus after, rh-window). One reply wrote the subtractive tier as "Drops the d". The other wrote assonance and consonance as "Same vowels, different consonants".
    • mosaic-entry gave 2 false negatives (Opus after and before, rh-window): both replies call their multi-word entries "two-word".
    • identity-flagged gave 1 false negative (Opus before, rh-release): "Both are identities".
    • Sonnet had 5 more: identity-flagged ×2, eight-plus-candidates ×2 and register-gate-skipped ×1. It also had 2 false passes on no-single-winner ("Pick: police.", "Police, but only just.").
    • Rescored by hand, Opus is 0.924 -> 0.972 (+0.048, normal interval -0.035 to +0.131). That is context only, not the verdict: only failing runs were audited, so a correction could only raise a score, and passing runs were not checked for false passes.

    Real behavior findings, which the after text does not change:

    • Sonnet names police as the lesser pick in every rh-release run, before and after.
    • Both models often skip the datamuse.sh syllables call the audit path asks for. Opus after skipped it in 2 of 3 dg-audit runs; their Bash calls only read files.

    Metaphor is not in this pass. The 2026-11-01 host run kept the before text.

    Next (lane 4):

    1. Move the remaining semantic regex graders in the rhyme menu cases (tier-labels-span, mosaic-entry, identity-flagged) to llm graders. Do it on the branch that already carries test(songwriting): grade no-single-winner with an llm judge in the menu cases #12195 (no-single-winner) and test(songwriting): grade eight-plus-candidates with an llm judge in the rhyme menu cases #12299 (eight-plus-candidates), with labeled samples that include the real replies from these legs.
    2. Calibrate the judges once, after every grader change.
    3. Re-measure only the Opus pair on that suite. Sonnet has passed.
  18. kyle-sexton commented on Nov 3, 2026

    @kyle-sexton
    MemberAuthor

    Item 2 Opus re-measure (lane 4, third round): cannot call, so this issue stays open.

    claude plugin eval, Opus subject and judge, frozen main a9dc1f04, six cases, 3 runs, with-arm only. SKILL.md files are unchanged since edef3f69; the suite digest was identical in both copies. "Before" is the three SKILL.md files at 1f61e714d9^.

    Leg Verdict (run-validity.py --runs 3 --scaffolded) Why
    after VALID, mean 0.887 dg-audit 0.917, dg-variations 0.933, mp-chorus 1.0, mp-sung 0.556, rh-release 0.917, rh-window 1.0
    before INVALID mp-sung hit max turns; denied cd <plugin>/research; sed; grep
    before2 INVALID same denial shape
    before3 INVALID denied F=<own tool-results>; grep $F

    With no VALID before leg there is no paired verdict. Context only, not a result: against the three before legs the change reads +0.05, -0.05 and -0.04. mp-sung's drop (0.556 against 1.0) looks like variance in the Skill hand-off; that is unproven. The #12376 graders went 3/3 in every leg.

    Verdicts are as judged at run time; the sandboxes are deleted, so no leg can be re-judged now. The before3 denial is run-validity failing a harmless read of the session's own tool results, not a case defect. Lane 4's notes at the time count two legs failing that way; only before3's command is on record, so that count is unconfirmed. The gate defect is reproduced and filed as #12583, and this work now waits on it.

    Next: after the gate fix, a fresh before and after pair launched together, with a restore re-run if mp-sung still drops. Spend: 51.34 USD, per the run's own cost reports.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    host-runNeeds an unattended run on a host; the host runner takes it, not a person or the issue lane.priority: mediumReal value, no hard deadline; normal backlog flow.status: confirmedA maintainer confirmed the item is real and in scope; ready to be worked.work-class: scopedA briefed fix or small feature; blast radius bounded by the brief, tests exist.

    Type

    Fields

    No fields configured for Task.

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions