Repository navigation
test(songwriting): grade no-single-winner with an llm judge in the menu cases #12195
Description
Observed
The no-single-winner regex grader in the songwriting menu cases (rh-window-menu-no-winner, rh-release-identity-menu, mt-loneliness-labeled-menu) checks a semantic outcome, "the reply names no single winner", by phrase shape. Each attempt to catch a bare single pick (How I'd choose: motel.) has opened a new false positive or false negative in review:
- test(songwriting): ready the eval suite for the next #8431 pass #12153: widening the marker to
[*_]*failed guide prose ("how I'd choose between them: ..."); droppingbyfailed an italic*by feel*guide. - test(songwriting): keep by exempt in the metaphor menu grader #12194: the bare-pick alternative failed guides that open with a filler word (
How I'd choose: honestly, it depends ...,first, ...,simple, ...) and guides with emphasized option numbers (bitterness, **#5**; ...), and its option-number exemption letHow I'd choose: motel, #2.through.
The rhyme cases carry the bare-pick alternative from #12153 and so still fail those guide shapes; #12194 removed it from the metaphor case and leaves the bare single pick uncaught there.
Fix wanted
Replace no-single-winner in all three cases with an llm grader whose rubric is the expected outcome (the reply offers a menu or a choosing guide keyed to feelings and crowns no single candidate; a guide that maps several moods to options passes; a lone pick, with or without its number, fails). Keep every current regex sample as a labeled sample, add the shapes listed above, calibrate the judge on them, and then remove the rhyme cases' bare-pick alternative. Do it after #8431 item 2's measured pass, which freezes the rhyme graders.
Activity
test(songwriting): drop the metaphor bare-pick alternative, keep the …
github-actions commentedon Nov 2, 2026
Proposed work class: work-class: scoped
Basis: automated intake triage; the lanes App applied this class.
Size: Small
Complexity: Medium
Uncertainty: Knowable
Flags: none
Blocked by: #8431
Understood: replace the regex
no-single-winnergrader in the three songwriting menu cases with an LLM judge. The rubric is that a menu or mood-keyed guide passes and a lone pick, with or without its number, fails. Keep the existing regex samples as labeled samples and add the shapes listed in the issue. Calibrate the judge on them, then remove the rhyme cases' bare-pick alternative. The issue says to do this after #8431 item 2's measured pass, which freezes the rhyme graders, so I listed #8431 as a blocker. No duplicate found.Automated triage, applied by the lanes App.
melodic-automation-lanes commentedon Nov 2, 2026
Not admitted to the issue lane: no trusted-human acceptance criteria: a listed human must put the confirmed
- [ ]criteria in the body or post them as their own comment, or, on an issue a listed human filed, intake's newest refinement comment must hold them unedited. Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).melodic-automation-lanes commentedon Nov 2, 2026
The latest triage verdict holds nothing: admission moved the issue to
status: confirmed; its comment says why.kyle-sexton commentedon Nov 2, 2026
Claimed (lane 4, session 1d59fcd0, melo-lap-001): branch test/12195-no-single-winner-llm holds the calibrated llm-grader fix for this and #12299; PR opening today.