From 677c9e941296ce70060279d2a300abe243751ba8 Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Sat, 31 Oct 2026 19:41:43 -0400 Subject: [PATCH 1/4] fix(evals): make plugin-eval, methodology and design fire on the jobs their cases miss Run 3 of the evals suite left four should-trigger cases unfired, which makes the suite INVALID. Each description now names the missed job, with "load this before answering" where the model answered from memory in one turn; methodology leads with its main use. Targeted with-arm checks (--runs 3, Opus 5.5, Claude Code 2.1.316): headless-stay-attached, pe-invalid-no-number, me-rewrite-criterion and de-criteria-first each fired 3 of 3. Refs: #9155 Co-Authored-By: Claude Opus 5.5 --- .changes/evals/fix-evals-descriptions-fire-9155.md | 7 +++++++ plugins/evals/skills/design/SKILL.md | 2 +- plugins/evals/skills/methodology/SKILL.md | 2 +- plugins/evals/skills/plugin-eval/SKILL.md | 2 +- 4 files changed, 10 insertions(+), 3 deletions(-) create mode 100644 .changes/evals/fix-evals-descriptions-fire-9155.md diff --git a/.changes/evals/fix-evals-descriptions-fire-9155.md b/.changes/evals/fix-evals-descriptions-fire-9155.md new file mode 100644 index 00000000000..7533af47ed5 --- /dev/null +++ b/.changes/evals/fix-evals-descriptions-fire-9155.md @@ -0,0 +1,7 @@ +--- +bump: patch +--- + +### Fixed + +- `/evals:plugin-eval`, `/evals:methodology` and `/evals:design` descriptions now name the jobs their eval cases showed the model answering without the skill: planning headless eval runs, judging whether an eval delta or validity verdict can be shared, writing a success criterion, and writing test cases for an LLM feature. `methodology` leads with its main use; its claude-api note moved to the end. diff --git a/plugins/evals/skills/design/SKILL.md b/plugins/evals/skills/design/SKILL.md index 945dabccd74..a3979547ae3 100644 --- a/plugins/evals/skills/design/SKILL.md +++ b/plugins/evals/skills/design/SKILL.md @@ -1,5 +1,5 @@ --- -description: "Design an evaluation suite for an LLM-based application or a Claude Code skill: interview for measurable success criteria, pick a grading method per criterion, and scaffold a criteria doc plus eval cases into the consumer repo. Use when: 'design evals', 'create an eval suite', 'scaffold evals', 'write evals for my skill', 'define success criteria for this app', 'set up LLM testing', 'build a test set for my prompt', 'evals for an LLM feature built on the Messages API', 'which eval route each part of a repo takes'. Not for eval-design theory questions (use /evals:methodology), not for statically validating an existing suite (use /evals:validate) or a skill-creator JSON eval file, and not for running or scoring a suite, which is /evals:plugin-eval and the CLI it guides." +description: "Design an evaluation suite or test cases for an LLM-based application or a Claude Code skill: interview for measurable success criteria before any case is written, pick a grading method per criterion, and scaffold a criteria doc plus eval cases into the consumer repo. Use when: writing test cases for an LLM feature or prompt, 'design evals', 'create an eval suite', 'scaffold evals', 'write evals for my skill', 'define success criteria for this app', 'set up LLM testing', 'build a test set for my prompt', 'evals for an LLM feature built on the Messages API', 'which eval route each part of a repo takes'. Not for eval-design theory questions (use /evals:methodology), not for statically validating an existing suite (use /evals:validate) or a skill-creator JSON eval file, and not for running or scoring a suite, which is /evals:plugin-eval and the CLI it guides." argument-hint: "|plugin >" user-invocable: true disable-model-invocation: false diff --git a/plugins/evals/skills/methodology/SKILL.md b/plugins/evals/skills/methodology/SKILL.md index 60b6a397656..0970b399d27 100644 --- a/plugins/evals/skills/methodology/SKILL.md +++ b/plugins/evals/skills/methodology/SKILL.md @@ -1,5 +1,5 @@ --- -description: "When the bundled claude-api skill resolves in this session, prefer its hillclimb for sweeping model and effort against an existing eval suite; this skill when the suite does not exist yet or its criteria are not yet measurable, before handing the suite to that search. Answers LLM-evaluation design questions from Anthropic's official evaluation guidance: success criteria, eval-suite design, grading methods, and the eval route for each part of a repository. Use when: 'define success criteria', 'how do I eval this', 'LLM eval', 'measure prompt quality', 'LLM judge', 'model-graded eval', 'golden answer', 'grading rubric', 'eval grading method', 'exact match vs LLM-graded', 'how many eval cases', 'rewrite this success criterion', 'evals for a service that calls the Messages API', 'evals for the whole repo'. Knowledge, not a runner; for scaffolding a suite use /evals:design, and for running and scoring a plugin's suite against a no-plugin baseline use /evals:plugin-eval." +description: "Answers LLM-evaluation design questions from Anthropic's official evaluation guidance: success criteria (including turning a vague quality goal into a measurable target before any data exists), eval-suite design, grading methods, and the eval route for each part of a repository. Use when: an eval success criterion, metric or target is being written or rewritten, with or without data (load this before answering), 'how do I eval this', 'LLM eval', 'measure prompt quality', 'LLM judge', 'model-graded eval', 'golden answer', 'grading rubric', 'eval grading method', 'exact match vs LLM-graded', 'how many eval cases', 'evals for a service that calls the Messages API', 'evals for the whole repo'. Knowledge, not a runner; for scaffolding a suite use /evals:design, and for running and scoring a plugin's suite against a no-plugin baseline use /evals:plugin-eval. When the bundled claude-api skill resolves, prefer its hillclimb for sweeping model and effort over an existing, measurable suite." argument-hint: "[question or concept]" user-invocable: true disable-model-invocation: false diff --git a/plugins/evals/skills/plugin-eval/SKILL.md b/plugins/evals/skills/plugin-eval/SKILL.md index f93db9ccc17..e9dbaa383a7 100644 --- a/plugins/evals/skills/plugin-eval/SKILL.md +++ b/plugins/evals/skills/plugin-eval/SKILL.md @@ -1,5 +1,5 @@ --- -description: "Guided practice around the `claude plugin eval` CLI, which runs and scores a plugin's eval suite: preflight, no-spend validation, a cost estimate under the ceiling, the run, and reading the with-versus-without delta. Use when: 'run my plugin evals', 'plugin eval', 'evaluate this plugin', 'eval my skill', 'does my skill actually fire', 'what is the delta', 'read my eval results', 'compare two eval runs', 'did my change make the skill better', 'is this gain real', 'aggregate-result.json', 'eval CI gate', 'can this machine run evals', 'how much will this eval cost', 'Bash refuses claude plugin eval', 'plugin eval blocked in a worktree', 'plugin eval refuses: too many worktrees', 'can plugin eval measure CLAUDE.md or rules'. Not for designing success criteria (/evals:design) or the skill-creator JSON eval format." +description: "Guided practice around the `claude plugin eval` CLI, which runs and scores a plugin's eval suite: preflight, no-spend validation, a cost estimate under the ceiling, the run, its validity verdict, and reading the with-versus-without delta. Use when: planning how a headless `claude -p` session launches eval runs, an eval run's delta, arm scores or validity verdict is in question, including whether it can be shared (load this before answering), 'run my plugin evals', 'plugin eval', 'evaluate this plugin', 'eval my skill', 'does my skill actually fire', 'compare two eval runs', 'did my change make the skill better', 'is this gain real', 'aggregate-result.json', 'eval CI gate', 'can this machine run evals', 'how much will this eval cost', 'Bash refuses claude plugin eval', 'plugin eval blocked in a worktree', 'plugin eval refuses: too many worktrees', 'can plugin eval measure CLAUDE.md or rules'. Not for designing success criteria (/evals:design) or the skill-creator JSON eval format." argument-hint: "[preflight|validate|run|read |ci|init] [target]" user-invocable: true disable-model-invocation: false From 96e84a6719db7392764c89f4cba8c1519299c9cb Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Sat, 31 Oct 2026 19:42:35 -0400 Subject: [PATCH 2/4] chore(evals): name the changelog fragment by branch slug Co-Authored-By: Claude Opus 5.5 --- ...tions-fire-9155.md => fix-9155-evals-descriptions-f3b6135f.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename .changes/evals/{fix-evals-descriptions-fire-9155.md => fix-9155-evals-descriptions-f3b6135f.md} (100%) diff --git a/.changes/evals/fix-evals-descriptions-fire-9155.md b/.changes/evals/fix-9155-evals-descriptions-f3b6135f.md similarity index 100% rename from .changes/evals/fix-evals-descriptions-fire-9155.md rename to .changes/evals/fix-9155-evals-descriptions-f3b6135f.md From 475393645e5603e4fac66a5bf89e2f50f6d7dc90 Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Sat, 31 Oct 2026 19:45:47 -0400 Subject: [PATCH 3/4] fix(evals): keep the claude-api presence gate in the methodology description Co-Authored-By: Claude Opus 5.5 --- plugins/evals/skills/methodology/SKILL.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/plugins/evals/skills/methodology/SKILL.md b/plugins/evals/skills/methodology/SKILL.md index 0970b399d27..94391804728 100644 --- a/plugins/evals/skills/methodology/SKILL.md +++ b/plugins/evals/skills/methodology/SKILL.md @@ -1,5 +1,5 @@ --- -description: "Answers LLM-evaluation design questions from Anthropic's official evaluation guidance: success criteria (including turning a vague quality goal into a measurable target before any data exists), eval-suite design, grading methods, and the eval route for each part of a repository. Use when: an eval success criterion, metric or target is being written or rewritten, with or without data (load this before answering), 'how do I eval this', 'LLM eval', 'measure prompt quality', 'LLM judge', 'model-graded eval', 'golden answer', 'grading rubric', 'eval grading method', 'exact match vs LLM-graded', 'how many eval cases', 'evals for a service that calls the Messages API', 'evals for the whole repo'. Knowledge, not a runner; for scaffolding a suite use /evals:design, and for running and scoring a plugin's suite against a no-plugin baseline use /evals:plugin-eval. When the bundled claude-api skill resolves, prefer its hillclimb for sweeping model and effort over an existing, measurable suite." +description: "Answers LLM-evaluation design questions from Anthropic's official evaluation guidance: success criteria (including turning a vague quality goal into a measurable target before any data exists), eval-suite design, grading methods, and the eval route for each part of a repository. Use when: an eval success criterion, metric or target is being written or rewritten, with or without data (load this before answering), 'how do I eval this', 'LLM eval', 'measure prompt quality', 'LLM judge', 'model-graded eval', 'golden answer', 'grading rubric', 'eval grading method', 'exact match vs LLM-graded', 'how many eval cases', 'evals for a service that calls the Messages API', 'evals for the whole repo'. Knowledge, not a runner; for scaffolding a suite use /evals:design, and for running and scoring a plugin's suite against a no-plugin baseline use /evals:plugin-eval. When the bundled claude-api skill resolves in this session, prefer its hillclimb for sweeping model and effort over an existing, measurable suite." argument-hint: "[question or concept]" user-invocable: true disable-model-invocation: false From 852f8100dc52b71a158568e33e6e369e870b77d8 Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Sat, 31 Oct 2026 23:49:58 -0400 Subject: [PATCH 4/4] fix(evals): restore the front-loaded claude-api routing split in methodology The native-references description phrase must come first and name both halves of the routing split. Put the gate clause back at the start, add "this skill before that suite or its criteria exist", and shorten the closing pointers so the description stays under the 1024-character cap. Co-Authored-By: Claude Opus 5.5 --- .changes/evals/fix-9155-evals-descriptions-f3b6135f.md | 2 +- plugins/evals/skills/methodology/SKILL.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.changes/evals/fix-9155-evals-descriptions-f3b6135f.md b/.changes/evals/fix-9155-evals-descriptions-f3b6135f.md index 7533af47ed5..1e5eec9fe0b 100644 --- a/.changes/evals/fix-9155-evals-descriptions-f3b6135f.md +++ b/.changes/evals/fix-9155-evals-descriptions-f3b6135f.md @@ -4,4 +4,4 @@ bump: patch ### Fixed -- `/evals:plugin-eval`, `/evals:methodology` and `/evals:design` descriptions now name the jobs their eval cases showed the model answering without the skill: planning headless eval runs, judging whether an eval delta or validity verdict can be shared, writing a success criterion, and writing test cases for an LLM feature. `methodology` leads with its main use; its claude-api note moved to the end. +- `/evals:plugin-eval`, `/evals:methodology` and `/evals:design` descriptions now name the jobs their eval cases showed the model answering without the skill: planning headless eval runs, judging whether an eval delta or validity verdict can be shared, writing a success criterion, and writing test cases for an LLM feature. `methodology` keeps its claude-api routing clause first and now says what it is preferred for: before a suite or its criteria exist. diff --git a/plugins/evals/skills/methodology/SKILL.md b/plugins/evals/skills/methodology/SKILL.md index 94391804728..ecb53562529 100644 --- a/plugins/evals/skills/methodology/SKILL.md +++ b/plugins/evals/skills/methodology/SKILL.md @@ -1,5 +1,5 @@ --- -description: "Answers LLM-evaluation design questions from Anthropic's official evaluation guidance: success criteria (including turning a vague quality goal into a measurable target before any data exists), eval-suite design, grading methods, and the eval route for each part of a repository. Use when: an eval success criterion, metric or target is being written or rewritten, with or without data (load this before answering), 'how do I eval this', 'LLM eval', 'measure prompt quality', 'LLM judge', 'model-graded eval', 'golden answer', 'grading rubric', 'eval grading method', 'exact match vs LLM-graded', 'how many eval cases', 'evals for a service that calls the Messages API', 'evals for the whole repo'. Knowledge, not a runner; for scaffolding a suite use /evals:design, and for running and scoring a plugin's suite against a no-plugin baseline use /evals:plugin-eval. When the bundled claude-api skill resolves in this session, prefer its hillclimb for sweeping model and effort over an existing, measurable suite." +description: "When the bundled claude-api skill resolves in this session, prefer its hillclimb for sweeping model and effort over a measurable suite; this skill before that suite or its criteria exist. Answers LLM-evaluation design questions from Anthropic's evaluation guidance: success criteria (including turning a vague quality goal into a measurable target before any data exists), eval-suite design, grading methods, and the eval route for each part of a repository. Use when: an eval success criterion, metric or target is being written or rewritten, with or without data (load this before answering), 'how do I eval this', 'LLM eval', 'measure prompt quality', 'LLM judge', 'model-graded eval', 'golden answer', 'grading rubric', 'eval grading method', 'exact match vs LLM-graded', 'how many eval cases', 'evals for a service that calls the Messages API', 'evals for the whole repo'. Knowledge, not a runner; scaffolding a suite is /evals:design, and running a plugin's suite against a no-plugin baseline is /evals:plugin-eval." argument-hint: "[question or concept]" user-invocable: true disable-model-invocation: false