[kyle-sexton]

Refs: #9155

Summary

The /evals:plugin-eval, /evals:methodology and /evals:design descriptions now name the jobs their eval cases showed the model answering without loading the skill. In run 3 on #9155 these cases missed the skill in 11 of 108 with-arm runs.

Fix

• plugin-eval: names planning headless claude -p eval runs, and judging whether a delta, arm score or validity verdict can be shared.
• methodology: leads with writing or rewriting a success criterion; the claude-api presence note moves to the end and is kept.
• design: names writing test cases for an LLM feature, with criteria before any case.
• Merged main to clear the stale-base overlap on plugin-eval/SKILL.md.

Verification

Claude Code 2.1.316, subscription login, Opus 5.5 subject, opus judge, out-of-repo copy of plugins/evals at 4753936, --allow-tools Write Edit --scaffold --keep-temp -j 2.

• Per-case check, with-arm only (--ablation none --runs 3): the skill fired in 12 of 12 runs across headless-stay-attached, pe-invalid-no-number, me-rewrite-criterion and de-criteria-first. Run 3 had 11 misses in these four cases.

• Full suite, 43 cases, both arms, 3 runs (258 runs): run-validity.py --runs 3 --scaffolded returns INVALID for two reasons:

  • 7 counted permission denials, mostly Grep calls in the without-plugin arm reaching above the sandbox to the parent directory: interval-setting-scope 3 of 3 runs, plus wrap-bare-skill 1 run. No description affects these.
  • The skill did not fire in 1 of 111 with-arm runs: de-skill-target-suite run 2. Run 3 had 11 misses.
  • These checks pass: run errors, row count, scaffold, session started (no sandbox failure), and models.

• scripts/check-affected.sh --tests is green on the merged head. push-branch.sh then merged main again (857f30c), and that merge brought a 37-line plugin-eval/SKILL.md body change and a run-validity.py change from main. The runs above measured 4753936, so they do not cover that later main content.

• 852f810 puts the methodology claude-api routing clause back at the front, with both halves of its split (review thread). On that head, with-arm only at 3 runs, me-rewrite-criterion, measurable-criterion and route-hillclimb-existing-suite each scored 3 of 3, and the skill fired 6 of 6 times in the two cases that have a fired grader.

Remaining on #9155

• Hold (do-not-merge): the non-inferiority comparison that plugins/playbooks/skills/skill-authoring/reference/skill-criteria.md "Rewriting an existing skill" asks for has not run. It needs:
  • a frozen probe set and a held-out set;
  • concurrent before and after runs that /evals:plugin-eval judges VALID;
  • the two-version noise report, covering all three descriptions on the final head.
• A VALID full-suite run. The run here is INVALID on 7 without-plugin Grep denials that reach above the sandbox: interval-setting-scope 3 of 3 runs, plus single runs in wrap-bare-skill and default-judge-same-model.
• The skill did not fire in 1 with-arm run, de-skill-target-suite run 2. A rerun should show whether that is noise.
• The newer plugin-eval/SKILL.md body change from main (merged in 857f30c) was not covered by these runs.

Related

• Paid judge-calibration and suite run for the evals plugin after #5879's sample fixes #9155: the host run, which still needs one VALID suite run.
• feat(evals): close the eval suite's calibration gaps and convert other plugins' evals #5879: the per-grader calibration table was posted there on 2026-10-30.

🤖 Generated with Claude Code

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-11-01T03:44:08.499311Z	857f30c	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 857f30cba2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[github-actions]

claude-security-review reviewed 852f8100dc52b71a158568e33e6e369e870b77d8: read 2 of 2 changed file(s); 0 finding(s), 0 as inline comments.

Security review (incremental, 857f30c → 852f810)

I found no security issues.

I read every hunk of both in-scope files in the incremental diff:

• plugins/evals/skills/methodology/SKILL.md: only the frontmatter description changed. The claude-api hillclimb routing clause moved to the front, with a new half saying this skill is for the stage before a suite or its criteria exist. The word "official" was dropped, and the closing routing clause was reworded. user-invocable, disable-model-invocation, argument-hint and the skill body are unchanged. No tool grant, hook, permission or other frontmatter key was touched.
• .changes/evals/fix-9155-evals-descriptions-f3b6135f.md: a one-line changelog edit that describes the same reorder.

Instruction-surface lens: the change reorders and rewords a skill's routing text. It deletes, narrows or softens no standing guardrail, and no rule from a protected class is touched. The skill-body section still says "Knowledge, not a runner" and still routes scaffolding and running to /evals:design and /evals:plugin-eval.

No trust control, plugin manifest, workflow or trusted-path file is in this increment. The PR context file has no embedded directives aimed at the lane.

The PR's own verification runs (described in the PR context) spend subscription usage only. That is not a hold.

critical-findings: 0

[github-actions]

claude-review: 1 inline finding(s) on 857f30cba20547f5f95ffff815f0ffe11cbbc578; the summary is in this lane's pull request comment.

claude-lane: claude-review

[github-actions]

claude-review reviewed 852f8100dc52b71a158568e33e6e369e870b77d8: read 2 of 2 changed file(s); 0 finding(s), 0 as inline comments.

Code review (incremental, 857f30c → 852f810)

No findings. I read every hunk in both files in scope.

• plugins/evals/skills/methodology/SKILL.md: the description now starts with the claude-api routing clause again and states both sides of the split. Use claude-api's hillclimb to sweep model and effort over a measurable suite. Use this skill before that suite or its criteria exist. The rest of the trigger list is unchanged. The /evals:design and /evals:plugin-eval pointers were reworded without changing what they route to, and no trigger phrase was removed. No volatile upstream specific was added, so .claude/rules/skill-bodies-state-current-rules.md is satisfied.
• .changes/evals/fix-9155-evals-descriptions-f3b6135f.md: the ### Fixed line now matches the new description. The routing clause stays first and gains a "preferred for" half. The fragment still uses bump: patch, which fits a description-only change.

Observation (not counted as a finding): the PR body's "Fix" bullet for methodology still says "the claude-api presence note moves to the end and is kept". The 852f810 addendum under "Verification" and the changelog fragment both say the clause is back at the front. Updating that bullet would keep the description consistent.

Verification: the per-case numbers on 852f810 (me-rewrite-criterion, measurable-criterion, route-hillclimb-existing-suite each 3 of 3) come from the author. I did not run them. The PR body also says it is held with do-not-merge until the non-inferiority comparison and a VALID full-suite run are done. This review doesn't change that hold.

No manifest changed, the PR closes no issues (it only refs #9155), and no settled-threads file was provided.

[kyle-sexton]

Non-inferiority result for the description rewrites (head 852f810): no loss.

This addresses the non-inferiority thread. Both arms ran at the same time on 2026-11-01 (05:24-06:28Z): Claude Code 2.1.316, subject claude-opus-5-5, judge opus, --ablation none --runs 3 -j 1 --keep-temp --threshold 0.8. The arms are identical except for the three SKILL.md files: before is origin/main, after is 852f810, checked by sha256. Each eval's TMPDIR and HOME were short and outside /home. The earlier run was INVALID because the model walked up from the eval workspace into a home directory, and this removes that cause.

Invocation probes (73 cases × 3 runs, train, val and held-out): run-validity.py --runs 3 gives VALID for both arms. The only warnings are denials that the gate exempts. evals-design-neg-train-05 is now denied in 2 of 3 runs instead of 3 of 3, with 1 clean run.

skill	split	trigger before → after	false-trigger before → after
design	train / val / held-out	1.00 → 1.00 (all)	0.00 → 0.00 (all)
methodology	train / val / held-out	1.00 → 1.00 (all)	0.00 → 0.00 (all)
plugin-eval	train / val / held-out	1.00 → 1.00 (all)	0.00 → 0.00 train and held-out; val 3/9 → 3/9

No case moved. The val false-trigger is plugin-eval-neg-val-03, which fires in both arms on the unchanged 'can plugin eval measure CLAUDE.md or rules' trigger.

Each removed or changed description line has a probe that tempts it, and none moved:

• plugin-eval 'what is the delta': tempted by methodology-ho-05, methodology-neg-val-03 and design-neg-train-01.
• plugin-eval 'read my eval results': tempted by the new plugin-eval-pos-val-04 ("My eval results just came back. Can you go through them with me and tell me what they mean?"). It fires 3/3 in both arms, so there is no loss on that line and no gain either.
• methodology success-criterion lines: tempted by methodology-pos-train-03, methodology-ho-03 and plugin-eval-neg-train-03.
• methodology claude-api clause: tempted by methodology-neg-train-04 and methodology-pos-val-03.
• methodology "official" and "and scoring": tempted by methodology-pos-val-01 and methodology-neg-train-02.
• design additions: tempted by design-neg-train-06 and design-ho-06.

The four outcome cases the PR targets: the after arm is VALID. Its skill fired in 12 of 12 runs, and the traces show 12 of 12 launches with no denials.

case	after arm: passed, skill fired
headless-stay-attached	3/3, 3/3
me-rewrite-criterion	3/3, 3/3
de-criteria-first	2/3, 3/3 (one run missed asks-criteria-question)
pe-invalid-no-number	0/3, 3/3 (no-number still fails)

The before arm of this suite is INVALID because the skill did not fire in 10 of its 12 runs, which is the gap this PR closes. Every other check passed, including no denials. Because it is INVALID, no before score is quoted. This suite shows that the skills now fire. It is not a validated with-before score delta. pe-invalid-no-number and one de-criteria-first run still fail on answer content after the skill loads, which is skill-body work outside this description change.

No case grants Bash, so no Bash was needed. Every Skill call in the traces was followed by a launch of the same skill.