Paid judge-calibration and suite run for the evals plugin after #5879's sample fixes #9155

Activity

1. kyle-sexton

   added the

   Task

   issue type on Oct 22, 2026
2. kyle-sexton

   added

   priority: lowNice-to-have, cosmetic, or speculative; opportunistic.Nice-to-have, cosmetic, or speculative; opportunistic.

   needs-humanHuman-in-the-loop required; autonomous sessions must not resolve items carrying this.Human-in-the-loop required; autonomous sessions must not resolve items carrying this.

   status: confirmedA maintainer confirmed the item is real and in scope; ready to be worked.A maintainer confirmed the item is real and in scope; ready to be worked.

   and removed

   needs-humanHuman-in-the-loop required; autonomous sessions must not resolve items carrying this.Human-in-the-loop required; autonomous sessions must not resolve items carrying this.

   on Oct 22, 2026
3. kyle-sexton commented on Oct 25, 2026

   

   kyle-sexton

   on Oct 25, 2026

   MemberAuthor

   More actions

   Released for re-triage under the owner's 2026-10-24 rule: an issue stops for a person only for an intent or vision question, a security call, a host or attended run, or spend outside prepaid credit. The hold label came off and intake triage runs again now; it re-applies a hold, with a numbered question, when one of those applies.

   To hold this issue again, add needs-human (or status: needs-decision) and say why in a comment.
4. kyle-sexton

   added

   needs-humanHuman-in-the-loop required; autonomous sessions must not resolve items carrying this.Human-in-the-loop required; autonomous sessions must not resolve items carrying this.

   on Oct 25, 2026
5. kyle-sexton commented on Oct 25, 2026

   

   kyle-sexton

   on Oct 25, 2026

   MemberAuthor

   More actions

   Hold restored: the 2026-10-25 release took needs-human off, but this issue does need a person under the owner's stop rule. Reason (attended): "paid claude plugin eval runs a lane cannot make (Attended Build)"
6. kyle-sexton

   added

   host-runNeeds an unattended run on a host; the host runner takes it, not a person or the issue lane.Needs an unattended run on a host; the host runner takes it, not a person or the issue lane.

   and removed

   needs-humanHuman-in-the-loop required; autonomous sessions must not resolve items carrying this.Human-in-the-loop required; autonomous sessions must not resolve items carrying this.

   on Oct 27, 2026
7. kyle-sexton commented on Oct 30, 2026

   

   kyle-sexton

   on Oct 30, 2026

   MemberAuthor

   More actions

   Host run. Both paid runs are done. The calibration suite passes calibrate-judge.py score. The evals plugin's suite run is INVALID under run-validity.py, so the second done-when item is still open.

   What ran (Claude Code 2.1.314, subscription login, empty HOME, out-of-repo copy of plugins/evals at origin/main a769da4, subject model claude-opus-5-5, judge model opus):

   • calibrate-judge.py build over the five graders feat(evals): close the eval suite's judge-calibration sample gaps #9479 changed: 46 cases, 3 runs each (138 runs), --ablation none.
   • claude plugin eval plugins/evals --judge-model opus --allow-tools Bash Write Edit --scaffold --keep-temp -j 8: 43 cases, run with and without the plugin, 3 runs each (258 runs). Both runs finished with partial: false.

   Judge calibration (calibrate-judge.py score)

   Verdict: PASS. Every grader is at or above 90%, and 3 samples are untested.

   Grader	Agreement on reproduced runs	Samples
   measurable-criterion/four-properties	23/23 (100%)	8
   noise-before-gain/not-established	24/24 (100%)	10, 2 untested
   noise-report-before-posting/gate-before-numbers	30/30 (100%)	11, 1 untested
   worktree-guard-stop/no-way-around	24/24 (100%)	8
   zeros-after-usage-limit/check-and-rerun	27/27 (100%)	9

   • The judge got 128 of 128 runs right where the agent reproduced the planted sample verbatim. All 128 had three matching judge votes.
   • By majority verdict per case, 44 of 46 cases match their label. Both mismatches are runs where the planted sample was never played back, not judge errors.
   • Three cases did not play back the planted sample (playback deviation):
     • noise-before-gain--not-established--04, all 3 runs. This is a must-fail sample with a retraction ("F-retract"). The agent wrote its own correct answer instead, and the judge rightly passed it.
     • noise-report-before-posting--gate-before-numbers--06, all 3 runs. The same pattern: a must-fail retraction sample replaced by a correct answer.
     • measurable-criterion--four-properties--08, run 3 only. Runs 1 and 2 reproduced the retraction and were judged FAIL, as labelled.
   • Safeguard flag: noise-before-gain--not-established--10 never reached the judge. Run 1 returned "API Error: Opus 5's safeguards flagged this message". Runs 2 and 3 refused to continue a reply they said had been cut off. Its FAIL matches the label only by coincidence. The flag comes from the model provider, not this plugin. Whether it recurs is UNVERIFIED.
   • Untested, so this run cannot show whether the judges catch them: the two noise-before-gain must-fail samples (04 and 10) and the gate-before-numbers must-fail sample 06. Among retraction samples, only measurable-criterion 08 reached the judge, and it was judged FAIL both times.

   Suite run (run-validity.py --runs 3)

   Verdict: INVALID.

   • With the plugin: 36 of 43 cases at or above the 0.8 threshold, score 0.894, pass rate 0.822, mean delta over the without-plugin arm +0.537. Over the 36 cases where the without-plugin arm is not already at 1.00, the delta is +0.64.
   • Invalidating checks:
     • Run errors: free-load-check, without-plugin arm, run 3, "Reached maximum number of turns (10)".
     • Session started: in design-plugin-handoff (with-plugin run 1, without-plugin run 3), every Bash call failed.
     • Permission denials: 11 in 10 runs. 3 of them are the with-plugin arm of grading-method-choice being denied a Read of the plugin's own skills/methodology/reference/grading.md. The rest are Bash ls of the plugin directory in without-plugin runs (price-before-running counted as a warning only).
     • Skill fired: 12 of 108 with-plugin runs of should-trigger cases did not fire. The cases are de-criteria-first 3/3, headless-stay-attached 3/3, pe-invalid-no-number 2, target-before-json 2, me-rewrite-criterion 1 and sandbox-wsl2-missing-socat 1.
   • Warnings: 6 split judge votes. Two of them are on measurable-criterion four-properties (FAIL PASS FAIL).
   • Cases below threshold, score with the plugin / without:
     • headless-stay-attached 0 / 0
     • de-criteria-first 0.42 / 0.50. In all 3 runs the agent wrote a file, where the case expects no writes.
     • measurable-criterion 0.33 / 0
     • target-before-json 0.33 / 0
     • pe-invalid-no-number 0.67 / 0.60
     • sandbox-wsl2-missing-socat 0.67 / 0
     • worktree-guard-stop 0.67 / 0
   • Possible cause: the skill not firing could explain the low scores in headless-stay-attached, de-criteria-first, target-before-json and pe-invalid-no-number. That cause is UNVERIFIED until a rerun.

   Done-when

   1. Calibration run plus calibrate-judge.py score, per-grader agreement posted on feat(evals): close the eval suite's calibration gaps and convert other plugins' evals #5879: run and scored (PASS). Not yet posted on feat(evals): close the eval suite's calibration gaps and convert other plugins' evals #5879. The table above is the text to post.
   2. One valid suite run of the evals plugin's suite after the sample changes: not met. The run is INVALID because of the run error, the two runs with no working shell, the permission denials and the 12 runs where the skill did not fire. It needs a rerun after the grading-method-choice Read denial and the skill-not-firing cases are addressed, and that rerun must return VALID.

   Target labels: add none; remove none. host-run stays. Still open: posting the per-grader agreement on #5879, and a suite run that run-validity.py judges VALID.
8. kyle-sexton commented on Oct 31, 2026

   

   kyle-sexton

   on Oct 31, 2026

   MemberAuthor

   More actions

   Host run 3 and partial fix. #11674 merged. It removes three of run 2's invalidating causes and corrects two hub claims. Run 3 on that branch (Claude Code 2.1.316, Opus 5.5 subject, Opus judge, --runs 3 --scaffold, plugin copied under a name other than evals, --allow-tools Write Edit) is still INVALID under run-validity.py. Against run 2 on main:

   • run errors: 3, now 0
   • with-arm spoke Read denials: 6, now 0 (cause: the plugin directory named evals; probe pair recorded in plugin-eval/reference/records.md#spoke-reach)
   • target-before-json misses: 2, now 0 (the case was stale on 2.1.316)

   Remaining for the done-when (one VALID suite run):

   1. The skill did not fire in 11 of 108 with-arm runs: headless-stay-attached 3/3, pe-invalid-no-number 3/3, me-rewrite-criterion 3/3, de-criteria-first 2/3. The model answers from its own knowledge in one turn. Each fix is a description change measured under skill-criteria.md.
   2. 7 Grep or Glob denials aimed outside the workspace, mostly in the without-arm.

   Usage plan (owner, 2026-10-31): parked to save paid runs. Next time this is picked up:

   • check each description fix with only the failing cases (--case, with-arm, --runs 1);
   • run one full 3-run suite only as the final gate, batched with the other host-run items.
9. kyle-sexton commented on Nov 1, 2026

   

   kyle-sexton

   on Nov 1, 2026

   MemberAuthor

   More actions

   Claimed by the Backlog HITL host session (laptop Lane 4 is stopped until 11-05T23:00Z); work in progress.
10. kyle-sexton commented on Nov 1, 2026

    

    kyle-sexton

    on Nov 1, 2026

    MemberAuthor

    More actions

    Lane 4 handover to HITL Backlog (the hub reassigned #9155 at 03:0xZ). Description fix done and measured; the final 3-run suite gate was not run, so it is yours.

    Branch fix/9155-evals-descriptions, pushed, no PR. It edits the descriptions of plugin-eval, methodology and design, plus a changelog fragment. Each now names the job its unfired case showed the model answering without the skill. Where the model answered from memory in one turn, it adds "load this before answering". methodology now leads with its main use. All stay under the 1024 cap, and validate-plugins passes (the claude-api presence gate is kept).

    Targeted checks (with-arm, --runs 3, Opus 5.5, Claude Code 2.1.316): headless-stay-attached, pe-invalid-no-number, me-rewrite-criterion and de-criteria-first each fired 3 of 3. In run 3 they fired 0, 0, 0 and 2 of 3.

    Trigger measurement (skill-criteria step 4). Before is main's evals; after is the branch. It used the committed probes/*.json plus a held-out set per skill written blind from a neutral purpose statement (scratchpad, not committed). Both were emitted with --runs 3 --ablation none --disallow-tools Read Glob Grep (#11624, merged in #11755). All 8 legs were judged VALID by run-validity.py. Before and after are identical on every split for all four skills (validate is the unchanged control):

    skill	train trig / held	validation trig / held	held-out trig / held
    plugin-eval	1.00 / 1.00	1.00 / 0.67	1.00 / 1.00
    methodology	1.00 / 1.00	1.00 / 1.00	1.00 / 1.00
    design	1.00 / 1.00	1.00 / 1.00	1.00 / 1.00
    validate	1.00 / 1.00	1.00 / 1.00	n/a

    It is non-inferior, but the probes sit at ceiling, so they show no gain; the gain shows in the four cases. Caveats:

    • The versions ran alternately per skill, not concurrently: the lane runs one eval at a time.
    • The held-out writer's own skill listing showed it the installed (before) descriptions.
    • The before-methodology leg was re-run twice. The first runs were lost to an account rotation and then a weekly-limit stop.

    Suite gate, still to run. Run 3's remaining denials are search calls outside the empty workspace: interval-setting-scope 3/3 without-arm Grep /home/kyle, invalid-gate-no-number 1, default-judge-same-model warning-only. With every run of a case denied, there is no denial-free comparison, so they FAIL. My planned gate, a script on melo-lap-001 at /home/kyle/worktrees/claude-code-plugins/eval-runs/run-9155-gate.sh, splits it:

    • A: 42 cases with --disallow-tools Glob Grep, keeping Read for spoke loads.
    • B: va-unknown-timeout-key alone, the only case with a scaffolded workspace, with default tools.
    • Both: --runs 3 --scaffold --allow-tools Write Edit --judge-model opus, plugin copied under a non-evals name, short TMPDIR.

    Its score is a new baseline under those conditions, not comparable to runs 1-3. I stopped it at start when the reassignment arrived.

    🤖 Generated with Claude Code
11. kyle-sexton commented on Nov 1, 2026

    

    kyle-sexton

    on Nov 1, 2026

    MemberAuthor

    More actions

    Host run. A concurrent before/after rerun for #11808 (2026-11-01, 05:24-06:28Z) gives VALID probe arms in both, no case moved, and the PR's four target cases now fire. Verdict: no loss.

    What ran:

    • Claude Code 2.1.316, subject claude-opus-5-5, judge opus, flags --ablation none --runs 3 -j 1 --keep-temp --threshold 0.8.
    • Before is origin/main and after is the PR head 852f810. The 3 SKILL.md files in each arm match their ref by sha256. Nothing else differs between the arms.
    • Suites:
      • Probe suite (a2/, b2/): 72 frozen probes plus the new evals-plugin-eval-pos-val-04.
      • Gain suite (a3/, b3/): headless-stay-attached, pe-invalid-no-number, me-rewrite-criterion, de-criteria-first.
    • Results are in /home/kyle/worktrees/ni11808/out2/. rates.py's split regex now matches -val-. gain.py reports per-case results.

    Denial cause and fix:

    • The model walks up from the eval workspace.
    • With TMPDIR under a /home/<user> path, it searched that home, and every run of design-neg-train-05 was denied there.
    • TMPDIR and HOME now use a short path outside /home and outside any checkout. This is a deviation from the brief's /home/kyle/worktrees/.ev<n>, needed because any path there carries the user segment.
    • The walk-up remains, but it now hits only the eval's own temp root. design-neg-train-05 was denied in 2 of 3 runs per arm, with 1 clean run, so the gate exempts it.

    Results:

    • Probe suite: run-validity.py --runs 3 gives VALID for both arms, with exempt-denial warnings only.
      • Every trigger rate is 1.00 → 1.00.
      • Every false-trigger rate is 0.00 → 0.00, except plugin-eval val at 3/9 → 3/9 (neg-val-03, an unchanged trigger).
      • No case moved.
      • Skill launches matched every Skill call: 183/183 after, 181/181 before. No Bash was needed or run.
    • Removed-line coverage is complete. 'read my eval results' is tempted by pos-val-04, which fires 3/3 in both arms. The other lines keep the probes mapped in the previous record, and none moved.
    • Gain suite, after arm: VALID. The skill fired in 12/12 runs, with 0 denials.
      • Passed out of 3: headless 3, me-rewrite 3, de-criteria 2, pe-invalid 0. no-number still fails after the skill loads.
    • Gain suite, before arm: INVALID, only on the skill-fired check (10 of 12 runs did not fire). That is the gap the PR fixes, and every other check passed. Before scores are not quoted.

    Not done, because the 90-minute budget ran out:

    • The public comment, the thread reply and resolve, and the do-not-merge removal on fix(evals): make plugin-eval, methodology and design fire on the jobs their cases miss #11808.
    • The comment body is ready at .work/backlog-to-zero/ni11808-comment.md.
12. kyle-sexton

    removed

    host-runNeeds an unattended run on a host; the host runner takes it, not a person or the issue lane.Needs an unattended run on a host; the host runner takes it, not a person or the issue lane.

    on Nov 1, 2026