Repository navigation
Run the eval suites deferred by the upstream-pointer sweep PRs #7296
Description
Description
The upstream-pointer sweep PRs (#7061-#7274) merged with their eval runs deferred: they mostly replace restated upstream Claude Code docs text with pointer records. Each PR body lists the skills whose eval run was owed. This issue collects those lists so the runs are not lost once the PRs close.
Run first: PRs that also changed behavior or output
- fix(source-control): point restated upstream docs at live sections and match the proxy's GraphQL refusal #7241 source-control:
babysit_gh.pyrecognizes the cloud proxy's GraphQL refusal wording, which changes how the unattended babysit loop falls back to REST. Evals:check, plus a babysit-prs run against a PR in a cloud session. - docs(work-items): point restated upstream docs at live sections #7239 work-items:
work/scripts/preflight.shprints corrected warning text (one test assertion updated). Evals:attend-queue,setup,ship,work. - docs(harness-ops): point restated upstream docs at live sections #7245 harness-ops: the install-state audit table has new swept rows (uploads, dev-mods, usage-data), a jobs kept row, and a reworded backups note. Evals:
audit-install-state,audit-native-overlap,audit-performance,audit-skill-visibility,check,machine-profile,prerequisites. - docs(instruction-placement): point restated upstream docs at live sections #7254 instruction-placement: the migrate shim check reads the AGENTS.md plugin option under both its old and new ID. Evals:
migrate,audit,delta, plus architecturemap-containers,map-context,map-flow,map-landscape,record-decision. - docs(context-budget): point restated upstream docs at live sections #7253 context-budget: the audit's emitted known-uncovered note text changed. Evals:
audit,check.
Pointer-only PRs
- refactor(discipline): point subagent specifics at the live subagents page #7061 discipline:
use-your-skills,sweep-all(sweep-all eval case 8 was updated with the worktree-containment wording) - docs(autonomy): point restated upstream docs at live sections #7133 autonomy:
setup(evals 6, 7, 31),check - docs(harness-config): point restated upstream docs at live sections #7136 harness-config:
audit-instructions,audit-pass,draft-auto-mode-rules,unhobble,audit - docs(code-tidying): point restated upstream docs at live sections #7250 code-tidying:
audit-dead-code,batch-simplify,audit,check - docs(discovery): point restated upstream docs at live sections #7251 discovery:
check,research,trace-intent - docs(disk-hygiene): point restated upstream docs at live sections #7252 disk-hygiene:
audit,check,clean,setup - docs(formatters): point restated upstream docs at live sections #7255 formatters:
checkin actionlint, bash-format, biome-format, eol-normalizer, go-format, markdown-format, powershell-format, ruff-format, typos-format - docs(context-guard): point restated upstream docs at live sections #7258 context-guard:
check - docs(docs-hygiene): point restated upstream docs at live sections #7259 docs-hygiene:
audit-encapsulation,audit-progressive-disclosure,rename-references - docs(planning): point restated upstream docs at live sections #7260 planning:
interview,plan,design,prd,draft-goal-condition,improve, multi-agentcheckandsetup - docs(guardrails): point restated upstream docs at live sections #7261 guardrails: desktop-notification
check, machine-healthauditandsetup, repo-fleet-hygieneapplyandaudit, repo-hygieneclean - docs(harness-memory): point restated upstream docs at live sections #7262 harness-memory:
audit,stateless - docs(misc): point restated upstream docs at live sections #7263 misc:
clarify, context7checkandlookup,audit-tools,build-cli, developer-experiencesetup,generate-file-name-gate, dometrainsetup, event-stormingmethodology,simulation, firecrawlupdate,reach,advise,manage,object-writing - docs(skill-quality): point restated upstream docs at live sections #7264 skill-quality:
check,setup - docs(review): point restated upstream docs at live sections #7265 review:
scan, bugssetup,write,explain-change,security-review - docs(testing): point restated upstream docs at live sections #7266 testing: code-metrics
principles, mutation-testingaudit,check,plan,write - docs(plugin-quality): point restated upstream docs at live sections #7267 plugin-quality: evals
methodology,audit-posture, plugin-qualityaudit - docs(performance): point restated upstream docs at live sections #7268 performance:
teach,book-distill,docpage-digest,map-corpus, knowledgesetup,go-faster - docs(session-flow): point restated upstream docs at live sections #7270 session-flow:
audit-friction,check,clean-stop,continue-in-background,handoff,reanchor,reconcile,running-retro,show-options - docs: point restated upstream docs at live sections in the visual plugins #7271 visual:
learn-style,diagnose, computer-usesetup, mirosetup,use, playwrightcheck,demo-video,present,visualize - docs(playbooks): point restated upstream docs at live sections #7274 playbooks:
boris,repo-sweep,update - docs(lib): point shared hook, managed-settings and channels comments at live sections #7242 lib and docs: point restated upstream docs at live sections #7272 repo-docs: no skill evals owed.
Run each with /evals:plugin-eval against the merged main. A regression goes in its own bug issue that links this one.
Affected area
Plugin skills across the marketplace (see the list).
Activity
github-actions commentedon Oct 12, 2026
Thanks for filing this. I understand this as a tracking issue collecting the eval runs deferred by the upstream-pointer sweep PRs (#7061-#7274), split into PRs that also changed behavior/output (run first) and pointer-only PRs. It names the specific eval suites owed per plugin and notes that any regression found should go in its own linked bug issue. The description is complete and the scope is clear, so nothing appears missing.
I didn't find an exact duplicate. #7292 is related (also from the same sweep) but covers restated-fact cleanup rather than deferred eval runs, so it's a separate concern.
Automated triage. A maintainer reviews it.
kyle-sexton commentedon Oct 19, 2026
This was generated by AI during triage.
Hold: each listed eval run spends eval budget, and no spend or ceiling has been approved for these runs. The owner has to approve that before an agent runs them.
kyle-sexton commentedon Oct 27, 2026
This was generated by AI.
Parked for capacity, not for the owner. The hold said no spend was approved. That is answered: subscription spend does not stop work (owner Q91), and evals run on the subscription login, only while usage allows (owner, 2026-10-25). The runs go through the plugin eval dispatch lane, so this waits on #9704 (validate, gate and summarize dispatched eval runs), now a blocked-by edge, and on subscription headroom.
Under the owner's 2026-10-26 stop rule, a person is needed only for a change in intent or vision, an implementation finding that changes the intent or primary design, or a high or critical security issue; host, attended and eval runs are agent work. So
needs-humancomes off, and the issue moves tohost-runwithstatus: confirmed, which keeps the issue lane from taking it when #9704 closes: #9704 adds an eval dispatch workflow, not eval capability for the issue lane.melodic-automation-lanes commentedon Oct 28, 2026
The work-resolve-issue lane stopped on this issue (
host-only): the run reports the work needs a host or attended run the lane cannot make.It added
host-runand removedagent-ready: a maintainer session on a host builds it, and it waits on no decision from a person.In the run's words:
What the run says it did not do:
Run: https://github.com/melodic-software/claude-code-plugins/actions/runs/40636250186
To hand it back to the lane, remove
host-runand addagent-readyagain.kyle-sexton commentedon Oct 31, 2026
Host run. Ran
claude plugin evalsuites for the owed skills on the WSL host (plugin arm against no-plugin arm, 3 runs each): the 90-minute box ended with 25 of the 59 skills that have cases fully VALID, and 70 of the 129 owed skills have no case this command can run.What ran
69e05e53675a20fcd1ffe7755f188d077417bbeb, copied outside any git repository, because the CLI refuses a plugin inside a repository with many linked worktrees (plugins/evals/skills/plugin-eval/SKILL.md, "Worktree isolation").claude plugin eval <copy> [--case <glob>] --runs 3 -j 2 --scaffold --trust-plugin --keep-temp --no-publish --allow-tools Bash Write Edit WebFetch --judge-model opus --json ..., each in its own cappedsystemd-runscope, with an emptyHOMEand the userCLAUDE_CONFIG_DIR. No ceiling flag: the owner's rule recorded on this issue on 2026-10-27 is that subscription spend does not stop work.plugins/evals/skills/plugin-eval/scripts/run-validity.py --runs 3 --scaffolded, the gate/evals:plugin-evalrequires. This row's brief pointed at the test: measure the Bash-gated plugin eval suites from #7009 on a host with a ready sandbox #7116 definition (partial: false, no run error, no skipped paid graders); the gate is stricter, and also fails on permission denials and other checks. Scores below come only from cases whose file passes the gate./plugin:skillprompt, its tags, or a required Skill grader). Unprefixed names in the issue were resolved from each sweep PR's body and file list:checkin docs(code-tidying): point restated upstream docs at live sections #7250 istoolchain:check, and itsauditiscodebase-health:auditorai-slop:audit(the PR touches both, its body says only "audit", neither has a case, and this record counts the first);improvein docs(planning): point restated upstream docs at live sections #7260 isimprovement:improve;check,plan,writein docs(testing): point restated upstream docs at live sections #7266 are testing's;audit-postureismcp-tools(both read from the file lists of docs(testing): point restated upstream docs at live sections #7266 and docs(plugin-quality): point restated upstream docs at live sections #7267);show-optionsis nowsession-flow:recommend-skills(feat(session-flow)!: rename show-options to recommend-skills #11149).Results, VALID cases only
With and Without are means over every VALID case. Delta is the mean over cases whose without-arm is under 1.00, because the gate leaves ceiling cases out of the delta, so it can differ from With minus Without. The #7116 cases ran on 2.1.314 at an earlier main. The last two rows are partial: 4 and 2 cases are missing a VALID result.
INVALID, no score reported
context-budget:audit:verdict: INVALID (3 permission denials in the traces), all inau-unmeasured-no-figurewith-arm runs. The denied Bash calls write under the plugin data directory and run the skill's node script.source-control:babysit-prs:verdict: INVALID (2 permission denials in the traces). Both denied calls liststate/babysit-prs/active-workers/under the plugin data directory.performance:go-faster:verdict: INVALID (the run is partial (interrupted); 2 run errors; 7 arms hold a row count other than --runs 3; 1 permission denial in the traces). The 7-case group took over 29 minutes, about 3 minutes a run, and the box's timeout (exit 124) stopped it during the plugin arm; no without-arm run finished.context-budget:auditandbabysit-prsare INVALID on denials alone. Both meet the test: measure the Bash-gated plugin eval suites from #7009 on a host with a ready sandbox #7116 definition: partial false, 0 run errors in 30 and 6 runs, no skipped paid graders.su-check-report. Whether that is a defect in those skills or an eval sandbox limit is UNVERIFIED.Findings
security-review-kill-switch-fail-open: a ceiling case, so it has no delta. In a VALID rerun here and the VALID test: measure the Bash-gated plugin eval suites from #7009 on a host with a ready sandbox #7116 run, 0 of 6 plugin-arm runs pass and 6 of 6 no-plugin runs pass. In the two runs I read, the review finds the fail-open bug; in one it rates it IMPORTANT, and both fail the case's CRITICAL check. The cause is not established. Not filed as a regression: no pre-PR run exists to compare, so the sweep is not shown to be the cause.wk-confirm-before-claim: a ceiling case; the plugin arm passes 2 of 3 runs. The failing run failedno-claim-before-yesby making a Bash call the grader counts as a claim or item write; that run went throughreclaim,list-frontierand a checklist file before any yes. Same caveat: no baseline.run-validity.pyat69e05e5, these test: measure the Bash-gated plugin eval suites from #7009 on a host with a ready sandbox #7116 result files are INVALID:session-flow(28 permission denials),testing-browser(2 denials),testing-k36(skill did not fire in 3 with-arm runs),performance-3-pr(9 denials),performance-3-r1(row count). The test: measure the Bash-gated plugin eval suites from #7009 on a host with a ready sandbox #7116 records report them under their own definition. Those scores are not reused here, which is why session-flow reads as not run below.harness-ops:audit-native-overlapis VALID with a warning:an-apply-defer-row-refusedhad a denial that left its score unchanged.Not done
evals/evals.json. The live page https://code.claude.com/docs/en/plugin-evals says of that format "neither tool reads the other's case files", so "Run each with/evals:plugin-eval" cannot be met as written for them. No skill-creator run was started. The skills: actionlint:check, autonomy:setup, autonomy:check, bash-format:check, biome-format:check, bugs:setup, code-metrics:principles, codebase-health:audit, computer-use:setup, computer-use:diagnose, context-guard:check, desktop-notification:check, disk-hygiene:audit, disk-hygiene:check, disk-hygiene:clean, disk-hygiene:setup, docs-hygiene:audit-encapsulation, docs-hygiene:audit-progressive-disclosure, docs-hygiene:rename-references, docs-naming:generate-file-name-gate, dometrain:setup, eol-normalizer:check, event-storming:methodology, event-storming:simulation, firecrawl:update, github:advise, go-format:check, harness-config:audit-instructions, harness-config:audit-pass, harness-config:draft-auto-mode-rules, harness-config:unhobble, harness-config:audit, harness-memory:audit, harness-memory:stateless, harness-ops:check, harness-ops:prerequisites, improvement:improve, instruction-placement:migrate, instruction-placement:audit, instruction-placement:delta, kindle-dedrm:manage, knowledge:setup, machine-health:setup, markdown-format:check, mcp-tools:audit-posture, miro:setup, multi-agent:check, mutation-testing:audit, playbooks:boris, playbooks:repo-sweep, playbooks:update, playgrounds:use, playwright:check, playwright:demo-video, plugin-quality:audit, powershell-format:check, repo-fleet-hygiene:apply, repo-fleet-hygiene:audit, repo-hygiene:clean, review:explain-change, ruff-format:check, skill-quality:check, skill-quality:setup, source-control:check, testing:check, testing:plan, typos-format:check, visualization:present, visualization:visualize, work-items:setup.performance ac*in its own box, then session-flow, bugs and the rest of that list, from the same queue shape. Rerun each candidate case at the parent of its PR's merge commit (kill-switch at docs(review): point restated upstream docs at live sections #726566ef3807e07d9cac27c34c161b5e92a7d51cdd1e, wk at docs(work-items): point restated upstream docs at live sections #7239879533934870644813e93d0b982dc13e00dd9590): that decides regression or pre-existing, and a regression gets its own bug issue.Acceptance, per the issue's checklist
The issue states no separate criteria; each checkbox is read as "the named evals ran".
checkhas no case;babysit-prsINVALID; cloud run not possible heresetuphas no casecheckandprerequisiteshave no casemigrate,audit,deltahave no casecheckVALID;auditINVALIDNo code changed and no PR was opened.
Target labels: add none; remove none.
kyle-sexton commentedon Nov 1, 2026
Claimed by the Backlog HITL host session; work in progress.
kyle-sexton commentedon Nov 1, 2026
Host run. A second round of
claude plugin evalsuites for the skills owed here: it confirmed the kill-switch case fix, ran session-flow, bugs, toolchaincheckand planningplan, and fixed one case grader in PR #11892.This record replaces the 2026-10-31 record only for the rows it reruns. Everything else in that record stands.
What ran
d33ec9d16e75c10b3128f4655dff59c246d06ac1. The plugins were exported withgit archiveto a directory outside any repository.systemd-runscope, with an emptyHOME, the userCLAUDE_CONFIG_DIRand a short TMPDIR. The command wasclaude plugin eval <plugin> --case <glob> --runs 3 -j 2 --scaffold --trust-plugin --keep-temp --no-publish --allow-tools Bash Write Edit WebFetch --judge-model opus --json ..., the same flags as the first round.plugins/evals/skills/plugin-eval/scripts/run-validity.py --runs 3 --scaffolded, run from the same export. Every file passed "session started": no trace shows a sandbox bridge-socket failure. Bash ran in the with-arm runs of the cases that need it: 7 to 16 calls per run in toolchainck-*, session-flowcb-*andho-*.Results, VALID files
INVALID files, scores not counted
session-flow:handoff(ho-*, 4 cases): 1 denial. In one with-arm run, Glob searched the plugin's own source directory forsave_point.py. Unscored for reference: with 0.88, without 0.35.ho-todowrite-renders-listfailstodowrite-offeredin 3 of 3 plugin-arm runs.session-flow:running-retro(rr-*, 2 cases): 11 denials. 7 are with-arm Bash and Glob calls that read the session transcript under the sandbox config directory. 4 are Write or Edit calls to.claude/rules/build.md, 1 with-arm and 3 without-arm.planning:plan(plan-*, 6 cases): 3 denials, all inplan-brief-assumption-carried. Each is a with-arm Bashcporsedthat reads the skill'stemplates/checklist.mdfrom the plugin directory. Unscored for reference:plan-review-replans-major-deviationscores 0.50 with the plugin (0/3) and 0.42 without.Finding fixed
af-unknown-token-stops: all 3 plugin-arm replies stop correctly and say that--apply-fixes"isn't an argument this skill accepts". Therejects-bad-tokenregex missed that wording. PR #11892 widens the regex and adds samples. Replayed on the kept replies, the plugin arm now passes 3 of 3 and the no-plugin arm 0 of 3. The PR is open and ready; it is not merged.Regression candidates from the first round
wk-confirm-before-claim: it cannot be decided as specified. The case does not exist at the parent of docs(work-items): point restated upstream docs at live sections #7239 (0728cf8), and the work skill changed three more times after that PR. A 2/3 against 3/3 difference over 3 runs also cannot separate the two. docs(work-items): point restated upstream docs at live sections #7239 changed only preflight warning text, so no bug issue is filed.Still owed
design,prdandinterview(36 cases); knowledgebook-distill,docpage-digestandmap-corpus; adhdclarify; songwritingobject-writing; animationlearn-style; code-tidyingaudit-dead-codeandbatch-simplify; educationteach; evalsmethodology; developer-experience. The queue reached its start deadline beforeplanning design-*.performance:go-faster(ac*): it timed out after 29 minutes last round, so it needs its own run window.kyle-sexton commentedon Nov 1, 2026
Host run, round 3. More of the owed suites, run against main at
ae8c90ccc517c8d1e350402ff2c9f30ac0544156, with the cause recorded for each round 2 INVALID suite and one case grader fixed in PR #11900, merged 2026-11-01T12:54Z.What ran (round 3)
HOMEand short TMPDIR as round 2.pgrepon the CLI eval process. New suites stopped starting at 08:15Z.run-validity.py --runs 3 --scaffolded. No run hit an "OAuth token revoked" error or a rate limit. Bash calls in each trace were counted per run. Every with-arm run of every suite below made 1 to 6 Bash calls, and no trace shows a sandbox bridge-socket failure.methodology. The #11808 rerun scoredme-rewrite-criterion3 of 3 with the plugin under--ablation none, which has no no-plugin arm, so no delta exists yet.Results, VALID files (round 3)
prd-handoff-scope-clearis a ceiling case: 1.00 without, 0.83 with (1 plugin-arm run at 0.5)The
planning:interviewsuite (interview-*, which matched 27 cases) is INVALID. It ran 87 minutes and the box's timeout (exit 124) stopped it before it finished. It also had 2 run errors, 1 denial, and 6 with-arm runs in which the skill did not fire. Unscored for reference: with 0.82, without 0.35. In 2 cases every run in both arms scored 0:interview-context-unfamiliar-term-upstreamandinterview-coverage-unattended-unexamined. Those cases are worth reading before the next run, which should split the suite into two globs so each fits a window.INVALID files, scores not counted (round 3)
knowledge:distill(distill-*, 5 cases): 32 denials. The skill reads and writes its plugin data directory under the eval'sconfig/plugins/data/, which the eval denies, and writes the distilled skill under.claude/skills/, which is denied in don't-ask mode in both arms. This has the same structural cause as running-retro below. Unscored for reference: with 0.87, without 0.75.animation:learn-style(ls-*, 2 cases): 2 denials, both in the no-plugin arm. Each was a read of a large tool output that the CLI had saved under the eval's deniedconfig/projects/.../tool-results/. The plugin arm had no denials. Unscored for reference: with 0.98, without 0.89.Cause of the 3 INVALID suites (round 2)
Read from the kept sandboxes'
config/settings.jsonand traces. No case defect makes any of them INVALID, and our TMPDIR and HOME setup does not cause them. Each run's sandbox puts the plugin root inallowRead, and the root's parent, the eval'sconfig/directory and everyevals/directory indenyRead. The plugin root is also indenyWrite.plan: this is a skill-text gap that only the eval exposes. In the same runs, read-only Bash on plugin files passed:cat,sed -n, andbash <plugin>/scripts/check-plan-outcome.sh. Only commands that write a file and name a plugin file as their source were denied:cp <plugin>/skills/plan/templates/checklist.md .work/...andsed ... <plugin file> > .work/.... The denial reason wassubcommandResults. The skill's checklist says "Copy into<memory_dir>/...". Reading the template and writing the copy with the Write tool would avoid it. No case change can, and no grant should.handoff: in 1 of 3 with-arm runs, Glob searched the whole plugin root for**/scripts/save_point.pyand was denied in don't-ask mode (decision_reason_type: mode). The plugin root holds the hiddenevals/directory.SKILL.mdnamessave_point.pywithout a path; the path appears only inreference/save-point.md:207. The case grants Glob already, so nothing should be loosened. This is an eval-only refusal caused by a wording gap: giving the${CLAUDE_PLUGIN_ROOT}/scripts/path in SKILL.md would stop the search.running-retro: two causes.<CLAUDE_CONFIG_DIR>/projects/(skills/running-retro/SKILL.md:73,216). The eval puts itsconfig/directory indenyRead. This is structural: the suite cannot pass the gate while the skill reads a live transcript..claude/rules/build.mdwere denied in don't-ask mode in both arms, while Edits to CLAUDE.md went through. A related case defect is fixed: theno-instruction-file-writtengrader matched Write only, so a run whose Edit of the rule file was denied still passed every grader. PR #11900, now merged, adds a matching Edit grader. Unscored observation: in round 2, with-arm run 3 wrote the learning into CLAUDE.md, failingclaude-md-unchanged.Still owed after round 3
interviewneeds a complete run in two halves, andplanandhandoffneed the skill-text changes below.clarify, songwritingobject-writing, code-tidyingbatch-simplify, educationteach, evalsmethodologywith a no-plugin arm, developer-experience (audit-tools,build-cli,setup), and performancego-faster, which needs its own run window. Run evalsmethodologyfrom a copy whose directory is not namedevals:plugins/evals/skills/plugin-eval/reference/records.mdrecords that anevalsplugin in a directory namedevalsgets its root put underdenyRead.distilland session-flowrunning-retrocannot reach VALID while the eval denies itsconfig/directory and.claude/writes. planningplanand session-flowhandoffneed the skill-text changes above first.Target labels: add none; remove none.
kyle-sexton commentedon Nov 1, 2026
Host run, round 4. I ran more of the owed suites on main at
94928a1b35f5444f26446e48371098216a171aba, with the round 2-3 flags and setup. The account reached its weekly limit at about 15:50Z, so most suites in this round errored and are not counted.What ran
git archiveexport, in a capped scope, with an emptyHOME, a short TMPDIR, andGH_TOKEN/GITHUB_TOKEN/DOCKER_CONFIGunset.run-validity.py --runs 3 --scaffolded.Results
cl-prose-no-identifiersgoes from 0.00 to 0.67Not counted
batch-simplify(bs-), educationteach(te-), evalsmethodology(me-), developer-experience (all 16 cases) and performancego-faster(ac) had 30, 36, 6, 84 and 21 errored runs. Those runs began after the account's weekly limit was reached. They need a rerun on an account with headroom.interviewmatched 0 cases in both halves.--case 'interview-[a-i]*'selected nothing, so this CLI's case glob does not take bracket classes. Split the next run by explicit prefixes instead (for exampleinterview-c*,interview-e*, ...).Still owed after round 4
evals), developer-experience, go-faster (in its own window), and interview split by prefix globs.object-writing: read the one denial before the rerun.distilland session-flowrunning-retrocannot reach VALID under the eval sandbox. planningplanand session-flowhandoffneed the skill-text changes first. The 70 skills with no eval case and the fix(source-control): point restated upstream docs at live sections and match the proxy's GraphQL refusal #7241 cloud run are still owed.Target labels: add none; remove none.
kyle-sexton commentedon Nov 2, 2026
Host run, round 5. I reran the suites round 4 lost to the usage limit: batch-simplify, teach, methodology, developer-experience, interview (split by prefix) and go-faster, one suite at a time.
What ran
git archive origin/mainexport taken at its start: batch-simplify atdda1002d63c83a48b2441c56df56a19d412ed4e8, teach atae240892bebca0d2f48a55ab80d5f64b94d8c0f1, methodology and developer-experience at7539d186d9cdcccd5bcf4d7b63580543c49f8bd6, interview fromb280b2de7d4c649d0613ada9a9659910883b9fafto4676c092cd65d0c0c5dd4da45a29b177a31239c6(each prefix's SHA is in its row), go-faster at2bf110b3d0a7c952ae829d272bcb6ee045ec2a65.--runs 3 -j 2 --scaffold --trust-plugin --keep-temp --no-publish --allow-tools Bash Write Edit WebFetch --judge-model opus. Each suite ran in a capped systemd scope, with an emptyHOME,CLAUDE_CONFIG_DIRset to the subscription config, an on-disk TMPDIR under/home/kyle/worktrees, andGH_TOKEN/GITHUB_TOKEN/DOCKER_CONFIGunset. methodology ran from a copy whose directory is not namedevals.run-validity.py --runs 3 --scaffoldedon each result. No run in any suite errored for the usage limit; the only run errors are the four max-turns runs below.Results
code-tidying
batch-simplify(bs-*, 9 cases): INVALIDOverall with 0.99. No run errors, no denials, 54 of 54 traces. INVALID only because
bs-no-task-tools-fallbackhas askill-invokedgrader (tool_used: Skill) and none of its 3 with-arm runs made a Skill call. Its prompt is the slash command/code-tidying:batch-simplify ..., and the with-arm trace reads the skill's owncontext/reference.md, so the skill text loaded without a Skill tool call. The grader is with-only and unscored. Unscored delta over the 7 cases below ceiling: +0.30.education
teach(te-*, 8 cases): INVALIDNo run errors, 48 of 48 traces. INVALID on 1 denial:
te-root-declared-substitutionwith-arm run 2, a Bash call that ran the skill's ownscripts/list-workspaces.sh. A with-arm denial aimed at the plugin directory is a FAIL even though that run scored 1.00, as its two siblings did. Unscored mean delta +0.33.evals
methodology(me-*, 1 case): VALIDAll checks PASS; the skill fired in all 3 with-arm runs. Delta +0.67.
developer-experience (all, 16 cases): INVALID
No run errors, 84 of 84 traces. Two FAILs:
history_filecases (s1-branch-start-build, s4-migrator-build, s7-act-after-yes, t4-apply-migrate) have 0 without-arm rows. Their prompt.md says a path target runs ahistory_filecase single-arm, so this is by design, but the validator counts it as a FAIL.rmof the old.claude/developer-experience.mdin all 3 runs (each still scored 1.00). t4-check-old-location (3 runs) and n1-missing-conventions-nudge (1 run, a warning) were denied a Grep of the session's own saved tool-result file under the run's config directory.Also: 2 split judge votes (s3-script-cleanup and s6-tool-inventory, grader
verdicts-cite-files). Unscored delta over the 9 cases below ceiling with a baseline: +0.49.planning
interview(interview-*, 40 cases, 15 prefix invocations)On main there are 40
interview-*cases, not 35. One invocation per first letter (c d e f g h i l n o p q r s u) covers all 40.The 11 VALID prefixes (d e f g h i n o p q u) hold 26 cases; the 4 INVALID prefixes (c l r s) hold 14. Most VALID prefixes warn that the skill fired in 0 with-arm runs; those cases have no skill-fired grader, so the check does not apply to them. Why the 4 failed:
interview-coded-terms-round-answerableandinterview-context-unfamiliar-term-upstreamhave a Skill grader and made no Skill call in any of their 6 with-arm runs. Both prompts open with/planning:interview, and the one with-arm trace read for each mentionsskills/interview(7 and 11 times), the same shape asbs-no-task-tools-fallback. No denials or run errors.cdinto the plugin'scontext/directory to readproblem-framing.md.interview-round-topic-groups-indexwith-arm runs 2 and 3 hit the case's max turns (12).interview-scope-cut-moscowwith-arm runs 1 and 3 hit max turns (12).performance
go-faster(ac*, 7 cases): INVALIDTook 39 minutes. Overall with 0.96. No run errors, no denials, 21 of 21 traces. INVALID only on row count: every
ac*case is ahistory_filecase, which its case.yaml says runs single-arm from a path target, so no case has a without-arm and there is no delta. 1 split judge vote (ac3, gradersection-order).Not counted
Still owed after round 5
history_filecases: all ran clean apart from row count, because they are single-arm by design. Either rerun them with--ablation none --runs 3and validate that pass, or changerun-validity.pyto accept a designed single-armhistory_filecase. Until then they have no counted score and no delta.bs-no-task-tools-fallback,interview-coded-terms-round-answerableandinterview-context-unfamiliar-term-upstreamload the skill through the prompt's slash command and never call the Skill tool, so theirtool_used: Skillgrader fails the skill-fired check every run. Either drop that grader or grade the skill load another way, then rerun those 3 cases.rm(t4-apply-migrate) and Grep of the session's own tool-result file (t4-check-old-location) are denied, and rerun.scripts/list-workspaces.sh,context/problem-framing.md).round-topic-groups-indexandscope-cut-moscow: 2 of 3 with-arm runs hit the 12-turn cap in each; raisemax_turnsor narrow the prompt, then rerun.object-writing(read the one denial before a rerun). knowledgedistilland session-flowrunning-retrocannot reach VALID under the eval sandbox. planningplanand session-flowhandoffneed the skill-text changes first. The 70 skills with no eval case and the fix(source-control): point restated upstream docs at live sections and match the proxy's GraphQL refusal #7241 cloud run are still owed.Target labels: add none; remove none.
fix(evals): drop file-less redirects before reading a denied Bash cal…
docs(planning): state the brainstorm grader drop by mechanism in its …
kyle-sexton commentedon Nov 2, 2026
Host run, round 6. I fixed the eval cases and the
run-validity.pyrules behind round 5's leftovers. I then reran seven cases one at a time (five I changed, plus the two whose denials I read) from a freshgit archiveof the fix branch, and re-validated the round 5 and #7148 results with the patched script at no eval cost.What ran
host-run/7296-r6-eval-case-fixes, fromorigin/mainat4189cc4dcd:88642f2728: the case fixes and the single-arm row-count rule.8094a7b436: drops file-less redirects inbash_paths.32c2713d30: rewords the planning changelog fragment.ba09ccccad(head, pushed): mergesorigin/mainate697879162, because the stale-base-overlap check foundreading-results.mdchanged on both sides.fd062ff95a. That is the pre-amend commit; its case files are identical to88642f2728, which only renamed the changelog fragments.88642f2728.8094a7b436.--runs 3 -j 2 --scaffold --trust-plugin --keep-temp --no-publish --allow-tools Bash Write Edit WebFetch --judge-model opus, plus--case <exact name>.HOME,CLAUDE_CONFIG_DIRset to the subscription config, and a short on-disk TMPDIR under/home/kyle/worktrees.GH_TOKEN,GITHUB_TOKENandDOCKER_CONFIGwere unset.run-validity.py --runs 3 --scaffoldedat the headba09ccccad. I re-ran it over all seven round 6 results, and the verdicts match the ones taken as each case ended.check-affected.sh --testspassed onba09ccccad: 14 passed, 2 skipped.test_run_validity.pypassed all 110 tests, and ruff check is clean.Fixes (branch)
Slash-command Skill graders. I dropped the
tool_used: Skillgrader and its sample from six cases whose prompt slash-invokes the skill under test. A slash-invoked skill loads without a Skill call, so that grader failed every run. The grader was with-only and unscored, so no score changes.expected_outcomeno longer says the Skill tool is called.Of these, only bs-no-task-tools-fallback and the two interview cases were rerun. The brainstorm cases were not; their scores cannot change, since the grader was unscored. This adds no new grader, so R38 (fail samples for new graders) does not apply.
validate-cases.pyreports no FAIL on the planning or code-tidying suite; every warning there was there before.max_turns. interview-round-topic-groups-index and interview-scope-cut-moscow now have
max_turns30, up from 12, andtimeout_seconds900, up from 600.Denials, read.
tool-results/file (t4-check-old-location, n1-missing-conventions-nudge) is already set aside as a warning by main'srun-validity.py(fix(evals): warn on a run's own tool-results reads in run-validity #12096). It needs no fix.cdinto the plugin'scontext/and thensed, instead of using Read, once in 3 runs. The deny rule blocked it. This was a one-off with no case fix; the rerun had no denial.Single-arm
history_filecases (run-validity.py). I changed the script and did not rerun the cases.context.history_file, and whose result has nowithoutkey at all, now passes the row count as a warning ("single-arm by design, with no delta"). An empty or short without-arm still fails, and so does a missing without-arm on a case with nohistory_file.suite.ablation: with-without, yet these cases came back with nowithoutkey. So the CLI runs them single-arm whatever the flag. A--ablation nonererun would have produced the same with-arm rows at about 60 minutes of eval.reading-results.mdrecords the observed shape with its basis and a recheck trigger.Redirect operands (test(source-control): add eval cases for babysit-prs, babysit-loop and setup #7148 worker's finding (a)).
bash_pathsread2>&1,2>/dev/nulland>/dev/nullas cwd-relative paths. It now drops a redirect that names no file; a redirect to a real file still names a path. The new test rows failed before the fix (1 failure) and pass after it.Not done: test(source-control): add eval cases for babysit-prs, babysit-loop and setup #7148 worker's finding (b), which would set aside a Write or Edit denied under the workspace's
.claude/as a warning. fix(evals): warn on a run's own tool-results reads in run-validity #12096 (784d63162c) decided to keep that shape a FAIL, and a test asserts it (test_other_denials_near_tool_results_still_fail). Its reasoning: a denied protected-path write means the case is measured only up to the write, and the case-authoring checklist says to treat such a case that way or record it as unmeasured. The test(source-control): add eval cases for babysit-prs, babysit-loop and setup #7148 result has no such denial anyway (see below), so leaving it out changes no verdict. Options are under "Needs a decision".Results
max_turns.answerable-without-follow-up) and the regex (context-claims-referenced) both fail. The with-arm answer does open each question with a Context block citingfile:line. So the case's graders may be stricter than the skill's output; that needs a read before its 0.00 is trusted.scripts/list-workspaces.shwith acatorlsof${CLAUDE_PLUGIN_DATA}/workspace-root. SKILL.md:126 names that file as a rung, and the eval sandbox denies plugin-data paths.Round 5 results re-validated with this branch's script (no new runs)
.e7296r5/gofaster, round 5's main at2bf110b3d0): VALID (warnings: 7 cases single-arm by design with no delta, skill fired, 1 split vote). The counted with-arm scores are round 5's: overall 0.96. These cases have no delta by design.7539d186d9): the row count now passes as a warning. The suite is still INVALID on the 3 t4-apply-migratermdenials (see Found); the tool-results Grep denials are warnings only.skill-invokedgrader. That result was made with the grader this branch drops; see the rerun row above.#7148 re-validation
run-validity.py --runs 3 --scaffoldedon/home/kyle/worktrees/.ev7148r4/out/aggregate-result.json, with the traces still present under the run's tmpfs directory (30 of 30 resolve):ls -la <plugin-data>/.../active-workers/ 2>&1denials in runs 1 and 2 are now set aside as plugin-data warnings.lswithpython3 <plugin dir>/skills/babysit-prs/scripts/manage_babysit_lease.py release --help. That names a plugin-dir path, so it stays a FAIL under the current rules. It has the same chained shape as te-root-declared-substitution..claude/appears in this result, so finding (b) would not change its verdict.Done-when
history_filecases: met by the script change. go-faster is VALID. developer-experience is held only by t4-apply-migrate.Found
rm .claude/developer-experience.mdwas denied in all 3 runs with the generic dontAsk message, yet theold-file-removedgrader passed: atool_usedgrader matches the call, not its result..claudeas a protected directory and says dontAsk denies protected-path writes (https://code.claude.com/docs/en/permission-modes#protected-paths). The page names writes only; whether a Bashrmcounts is not stated there, so that cause is UNVERIFIED./toolchain:ratchetwith atoolchain:ratchetSkill grader. I left them for the coordinator rather than widen this branch to another plugin.host-run/12043-eval-skill-load; this branch does not touch it.For the coordinator (not owner escalations: no intent, security or cost question)
${CLAUDE_PLUGIN_DATA}/workspace-rootin its own call. This is a skill change and needs eval evidence of no loss.run-validity.pyto warn on a denied Bash call whose only plugin-dir part the same run later ran un-denied, while its denied part is plugin data..claude/writes. You asked me to set a denied Write or Edit under the workspace's.claude/aside as a warning (test(source-control): add eval cases for babysit-prs, babysit-loop and setup #7148 finding (b)). I did not apply it, because fix(evals): warn on a run's own tool-results reads in run-validity #12096 (784d63162c) decided to keep that exact shape a FAIL and pinned it with a test. Do you want fix(evals): warn on a run's own tool-results reads in run-validity #12096 overridden?784d63162ccommit message; permission-modes Protected paths.Target labels: add none; remove none.