Repository navigation
Open
Labels
host-runNeeds an unattended run on a host; the host runner takes it, not a person or the issue lane.Needs an unattended run on a host; the host runner takes it, not a person or the issue lane.priority: mediumReal value, no hard deadline; normal backlog flow.Real value, no hard deadline; normal backlog flow.status: confirmedA maintainer confirmed the item is real and in scope; ready to be worked.A maintainer confirmed the item is real and in scope; ready to be worked.work-class: scopedA briefed fix or small feature; blast radius bounded by the brief, tests exist.A briefed fix or small feature; blast radius bounded by the brief, tests exist.
Description
Refs: #8427
Summary
Review of #8427 found three defects in the songwriting eval suite. The suite was frozen before the after-version runs (plugins/playbooks/skills/skill-authoring/reference/rewrite-a-plugin.md, step 2: "change nothing in it after the first after-version run"), and the PR's decision table reports scores from those runs, so the fixes wait for the next measured pass on this plugin. Each was verified on head 92362dac.
Findings
dg-variations-arrow-blockscannot run the rubric the skill requires.prompt.mdallows onlySkill, Read, Glob, Grep.skills/diagnose/SKILL.md:60-62makes the rubric passes non-skippable and has the countable passes run as commands (datamuse.sh syllables, a word-frequency count). WithoutBashthe case can pass on formatting while the mandatory checks never run. Fix: grant the tools the rubric needs, with fixtures or mocks for the network call, then re-measure. Threadmp-chorus-stress-count/raw-count-not-balancepasses a reply that endorses the raw count. Its first alternative\braw[- ]syllables?\bmatches any mention. Checked: "The raw syllable counts, 8/9/9/8, show the chorus is nicely balanced" matches. Fix: require a negation on that alternative, and add a fail sample that endorses the raw count. Threadrh-window-menu-no-winner/eight-plus-candidatescounts lines that are not candidates. Table header rows and bold heading lines count toward 8. Checked: a table with a header row and 7 candidate rows matches. Fix: count candidate rows only, and add a fail sample with 6-7 candidates plus headings. Thread
The PR body's "Ready-step review findings left in place" lists the earlier grader findings for the same pass.
Acceptance
- Each grader fix accepts every reply that meets the case's expected outcome and rejects a reply written to fail, proved by samples.
- The fixed suite is frozen again and the affected skills are re-measured before and after under the rewrite protocol.
Metadata
Metadata
Assignees
Labels
host-runNeeds an unattended run on a host; the host runner takes it, not a person or the issue lane.Needs an unattended run on a host; the host runner takes it, not a person or the issue lane.priority: mediumReal value, no hard deadline; normal backlog flow.Real value, no hard deadline; normal backlog flow.status: confirmedA maintainer confirmed the item is real and in scope; ready to be worked.A maintainer confirmed the item is real and in scope; ready to be worked.work-class: scopedA briefed fix or small feature; blast radius bounded by the brief, tests exist.A briefed fix or small feature; blast radius bounded by the brief, tests exist.
Type
Fields
No fields configured for Task.
Activity
github-actions commentedon Oct 19, 2026
Proposed work class: work-class: scoped
Basis: automated intake triage; the lanes App applied this class.
Size: Medium
Complexity: Medium
Uncertainty: Knowable
Flags: none
Blocked by: none
Understood: the remaining work is a one-line path fix in diagnose (before and after copies), then re-freezing the songwriting eval suite and re-measuring five cases on both models. The recorded decision names the steps, and the re-measure is a host run.
The decision settles the open questions, so nothing further is asked. The host-run step is routed by admission, not held for a person.
Automated triage, applied by the lanes App.
kyle-sexton commentedon Oct 19, 2026
Owner decision (2026-10-19): change the case, not the skill.
mt-loneliness-labeled-menuexpects a menu even when the user asks for one metaphor. Both the before and after versions ofmetaphorhonor "just give me one" and skip the menu, and that is the right behavior. Rewrite the case to expect one metaphor with its family-distance label when the user asks for one. Keep a separate case where the user asks for options, which expects the menu.Then re-measure
metaphorin the next measured pass. It currently ships its before text (#8427).github-actions commentedon Oct 26, 2026
The triage sweep dispatched intake triage for this issue (attempt 1 of 2).
melodic-automation-lanes commentedon Oct 26, 2026
Not admitted to the issue lane: it carries host-run. Admission takes the issue once each of those is met, when intake triages it again (
.github/actions/resolve-config/admission-rules.json).melodic-automation-lanes commentedon Oct 26, 2026
The latest triage verdict holds nothing: admission moved the issue to
status: confirmed; its comment says why.kyle-sexton commentedon Oct 29, 2026
Re-triage. This is not an owner stop. The three grader defects are still on main, and the owner decided the fourth change on 2026-10-19 (comment).
Checked on main:
plugins/songwriting/evals/dg-variations-arrow-blocks/prompt.mdallows[Skill, Read, Glob, Grep]only.plugins/songwriting/evals/mp-chorus-stress-count/case.yaml:16still opens with the bare\braw[- ]syllables?\balternative.plugins/songwriting/evals/mt-loneliness-labeled-menustill expects a menu.Acceptance criteria:
plugins/songwriting/evals/dg-variations-arrow-blocksgrants the tools the diagnose rubric runs (skills/diagnose/SKILL.md:60-62), with a fixture or mock in place of the network call.plugins/songwriting/evals/mp-chorus-stress-countgraderraw-count-not-balancerequires a negation on the raw-count alternative, and a new fail sample that endorses the raw 8/9/9/8 count is rejected.plugins/songwriting/evals/rh-window-menu-no-winnergradereight-plus-candidatescounts candidate rows only, and a new fail sample with 6-7 candidates plus a header row and bold headings is rejected.plugins/songwriting/evals/mt-loneliness-labeled-menuexpects one metaphor with its family-distance label when the user asks for one, and a separate case where the user asks for options expects the menu./evals:validatepasses on the suite, and each changed grader accepts every pass sample and rejects every fail sample offline.Checks a person runs: freeze the fixed suite again and re-measure the affected skills (
diagnose,metaphorand the others the cases cover) before and after, under the rewrite protocol. Fix 1 makes the suite Bash-granting, so this needs a host with a ready sandbox.29 remaining items
kyle-sexton commentedon Nov 1, 2026
Host run (2026-11-01): metaphor before and after, five cases, Opus and Sonnet. Verdict: keep the before text. Neither model's compare shows the after text non-inferior at the 5-point margin, and the two metaphor cases alone are too few to call. Nothing ships, and no pull request was opened.
Setup
379109c2fc, which holds test(songwriting): accept raw-count dismissals the grader failed #11569 and fix(songwriting): give diagnose the full datamuse.sh path #11648.plugins/songwriting/evalsis treeebd24d91488f. Both copies carry the same five cases (dg-variations-arrow-blocks,mp-chorus-stress-count,rh-window-menu-no-winner,mt-loneliness-labeled-menu,mt-loneliness-one-labeled), with an identical evals digest.songwritingplugin. After: the same, with the body ofskills/metaphor/SKILL.mdreplaced by the after text (845 words as posted).description. Slash-invoked cases measure no triggering, andskill-criteria.mdstep 6 ships only measured parts, so the after text's shorter description still needs its own probe measurement.diff -rqof the two copies showsskills/metaphor/SKILL.mdas the only differing file. Both copies sit outside any git repository.diagnose,meter-prosodyandrhymecarry identical text in both copies. Their rows are a noise check. They are not the comparison the decision sets out, of the pre-feat(songwriting): concise skill text measured by evals, plus outcome cases and probes #8427 text (1f61e714d9^, with the item 1 path fix applied to the before copy) against the current text, re-run under item 2. This run measures metaphor only.validate-cases.py: exit 0, one WARN (Bashneeds--allow-tools).systemd-run --user --scope -p MemoryMax=8Gunit, one at a time:claude plugin eval <copy> --model <sonnet|opus> --runs 3 -j 2 --trust-plugin --keep-temp --scaffold --allow-tools Bash --judge-model opus --threshold 0.8 --no-publishTMPDIRwas a 31 to 34 character directory on the home filesystem, one per leg. That is above the 24-character boundplugin-evalrecords. Thesession startedcheck passed in all 150 traces. Every with-arm dg run in the four counted legs ran Bash, includingdatamuse.sh ... syllables.regexortool_used, so no judge calibration applies.run-validity.pyandnoise-report.pywere taken from main379109c2fc, which includes fix(evals): warn on denials every run of both arms shares and pass 0-turn forked sessions in run-validity #11787 and fix(evals): run invocation probes without subagents and name the run line in run-validity #11901.L4 was not counted. Its INVALID line reads:
check permission denials: FAIL (1 denial in 1 run),mp-chorus-stress-countwith-arm run 1. A Bashsedread of the plugin'scontext/pat-pattison/research/response-filter.mdwas "blocked by a deny rule". That run scored 1.000. L4b reran the whole leg with the same flags.The
skill firedwarning means no case has atool_used: Skillgrader. Every prompt slash-invokes its skill.Scores (mean with-arm score over 3 runs, VALID legs only)
Sonnet (L1 before, L2 after)
mt-loneliness-labeled-menumt-loneliness-one-labeleddg-variations-arrow-blocksmp-chorus-stress-countrh-window-menu-no-winnerOpus (L4b before, L3 after)
mt-loneliness-labeled-menumt-loneliness-one-labeleddg-variations-arrow-blocksmp-chorus-stress-countrh-window-menu-no-winnerfamily-distance-labelinmt-loneliness-one-labeled, the grader behind #8427's Sonnet loss, passed 2 of 3 runs on Sonnet in both versions. On Opus it passed 2 of 3 before and 1 of 3 after.Noise report, after against before, all five cases
noise-report.py <after.json> --threshold 0.8 --interval-method normal --grader-agreement --baseline <before.json> --margin 0.05Sonnet (L2 against L1):
Opus (L3 against L4b):
Over the two metaphor cases alone, both models gave
compare verdict: n too small to call (cases scored in both results: 2; 3 or more are needed). The with-arm means were 0.94 to 0.94 on Sonnet and 0.78 to 0.81 on Opus.Step-6 verdict: keep the before text
skill-criteria.mdstep 6 requires non-inferiority at a 5-point margin and no case dropping by a third without a measured cause. Neither holds:This is "not shown non-inferior", not a measured loss. Per the decision, metaphor keeps its before text. Metaphor's result does not hold this issue open.
Grader findings (spot-read against the kept traces)
dg-variations-arrow-blocks/ran-syllable-countrejects real script runs.datamuse.shwithsyllablesin a Bash command. I read the commands of all 9 failing runs. Each one set the path in a variable and then called it (S=<plugin>/context/pat-pattison/scripts/datamuse.sh; for w in ...; do ... "$S" syllables), whichinput_matchdoes not cover.mp-chorus-stress-count/raw-count-not-balancestill rejects valid dismissals after test(songwriting): accept raw-count dismissals the grader failed #11569. L1 runs 0 and 1 failed on "this method measures line length in stressed syllables, and by that count the chorus is 3/4/4/3, so I wouldn't call it 'nicely balanced'" and "Syllable counts are still the wrong measure for 'balanced'". With-arm passes were 1/3, 0/3, 1/3 and 2/3 across L1, L2, L3 and L4b.mt-loneliness-one-labeled/family-distance-labelmisses the word order "medium-distance". L3 run 1 failed on "It's a medium-distance pairing", because the pattern requiresdistancebefore the label. The other misses I read (L1 run 1, L2 run 1, L3 run 0) contain no distance wording, so those failures are correct.mt-loneliness-labeled-menu/finder-question-listsrequires the literal word "characteristics".mt-loneliness-labeled-menu/no-single-winnermatches a choosing guide. L4b run 1 failed on the heading "How I'd choose:", which introduces picks by feeling ("For quiet ache: chore: initialize marketplace scaffold #1 or fix: correct plugin-system inaccuracies and harden references #2"). It crowns no option.The verdict above does not depend on these. They are noise sources a re-measure should remove first.
What remains
ran-syllable-count,raw-count-not-balance,family-distance-label,finder-question-listsandno-single-winnerto accept the forms quoted above, each with a pass sample for the quoted reply and an existing fail sample still rejected.kyle-sexton commentedon Nov 1, 2026
Claimed by host lane 4 (melo-lap-001 WSL), 2026-11-01T11:40Z: the grader fixes from the 2026-11-01 host run's findings (one PR, offline-validated), then the item 2 comparison for diagnose, meter-prosody and rhyme as a paid host run after that PR merges.
kyle-sexton commentedon Nov 1, 2026
Item 2 measured pass (lane 4): cannot call, so this issue stays open.
claude plugin eval, suite frozen at main1a0da52(holds #11941). "Before" is the three SKILL.md files at1f61e714d9^; "after" is main. 3 runs per arm on Sonnet 5.5 and Opus 5.5. All four legs VALID (run-validity.py --runs 3 --scaffolded) with 0 denials.noise-report.py --baseline <before> --margin 0.05: "not shown non-inferior at margin 0.05" on both models. Sonnet after minus before is +0.06 [-0.05, +0.16]; Opus is -0.02 [-0.07, +0.02]. Neither model has a case drop. The without arm (no skill text) moved only within noise, so conditions held.What the next pass needs:
mp-chorus-stress-count/raw-count-not-balancegives false negatives in both versions. Six with-arm misses state that the raw 8/9/9/8 count does not show balance, in wording the pattern does not cover. With those corrected, mp scores 1.000 in every leg. Widen the pattern, with samples, before the next freeze.rh/tier-labels-span(a closeness word in place of "family") andrh/no-single-winner(two picks for one mood, the shape test(songwriting): widen five eval graders that failed valid replies #11941 exempted for metaphor).test(songwriting): answer the pr-verify fail note on the suite
kyle-sexton commentedon Nov 2, 2026
Suite readied for the next item-2 pass: #12153 merged (head d58acd3).
raw-count-not-balanceis now an llm grader;no-single-winner(rh-window) and the register and identity graders were fixed through six review rounds and one pr-verify round; diagnose, meter-prosody and rhyme now have two slash-invoked cases each, and the new diagnose case runs with Bash and the offline datamuse stand-in.Next (lane 4): calibrate the three llm graders on their samples, run the before-version smoke pass on the three new cases, freeze the suite at main, then the before/after legs on Sonnet and Opus under rewrite-a-plugin.md.
kyle-sexton commentedon Nov 2, 2026
Item 2 measured pass (lane 4, second round): Sonnet non-inferior, Opus cannot be called, so this issue stays open.
claude plugin eval, suite frozen at mainedef3f69(holds #12153, #12194, #12209). "Before" is the three SKILL.md files at1f61e714d9^; "after" is main. Six cases (tagsdiagnose,meter-prosody,rhyme), 3 runs, with-arm only (--ablation none),--scaffold --allow-tools Bash, Opus judge, legs A-B-B-A. All four legs VALID (run-validity.py --runs 3 --scaffolded). The only warnings were denied reads of the session's own tool-result files, which are not counted, and no skill fired, which no grader checks. Every trace shows the pinned model.Two-version noise report (
noise-report.py --interval-method normal --grader-agreement --margin 0.05, paired over 6 cases, scored numbers; the verdict rule is step 6 ofskill-criteria.md):Why Opus cannot be called. Every failing with-plugin run was read in full. The scored Opus shortfall comes from regex graders that check a meaning by phrase shape and fail replies that meet the case's stated outcome:
tier-labels-spangave 2 false negatives (Opus after, rh-window). One reply wrote the subtractive tier as "Drops thed". The other wrote assonance and consonance as "Same vowels, different consonants".mosaic-entrygave 2 false negatives (Opus after and before, rh-window): both replies call their multi-word entries "two-word".identity-flaggedgave 1 false negative (Opus before, rh-release): "Both are identities".identity-flagged×2,eight-plus-candidates×2 andregister-gate-skipped×1. It also had 2 false passes onno-single-winner("Pick: police.", "Police, but only just.").Real behavior findings, which the after text does not change:
datamuse.sh syllablescall the audit path asks for. Opus after skipped it in 2 of 3 dg-audit runs; their Bash calls only read files.Metaphor is not in this pass. The 2026-11-01 host run kept the before text.
Next (lane 4):
tier-labels-span,mosaic-entry,identity-flagged) to llm graders. Do it on the branch that already carries test(songwriting): grade no-single-winner with an llm judge in the menu cases #12195 (no-single-winner) and test(songwriting): grade eight-plus-candidates with an llm judge in the rhyme menu cases #12299 (eight-plus-candidates), with labeled samples that include the real replies from these legs.kyle-sexton commentedon Nov 3, 2026
Item 2 Opus re-measure (lane 4, third round): cannot call, so this issue stays open.
claude plugin eval, Opus subject and judge, frozen maina9dc1f04, six cases, 3 runs, with-arm only. SKILL.md files are unchanged sinceedef3f69; the suite digest was identical in both copies. "Before" is the three SKILL.md files at1f61e714d9^.run-validity.py --runs 3 --scaffolded)cd <plugin>/research; sed; grepF=<own tool-results>; grep $FWith no VALID before leg there is no paired verdict. Context only, not a result: against the three before legs the change reads +0.05, -0.05 and -0.04. mp-sung's drop (0.556 against 1.0) looks like variance in the Skill hand-off; that is unproven. The #12376 graders went 3/3 in every leg.
Verdicts are as judged at run time; the sandboxes are deleted, so no leg can be re-judged now. The before3 denial is run-validity failing a harmless read of the session's own tool results, not a case defect. Lane 4's notes at the time count two legs failing that way; only before3's command is on record, so that count is unconfirmed. The gate defect is reproduced and filed as #12583, and this work now waits on it.
Next: after the gate fix, a fresh before and after pair launched together, with a restore re-run if mp-sung still drops. Spend: 51.34 USD, per the run's own cost reports.