diff --git a/.changes/evals/fix-9155-evals-descriptions-f3b6135f.md b/.changes/evals/fix-9155-evals-descriptions-f3b6135f.md new file mode 100644 index 00000000000..1e5eec9fe0b --- /dev/null +++ b/.changes/evals/fix-9155-evals-descriptions-f3b6135f.md @@ -0,0 +1,7 @@ +--- +bump: patch +--- + +### Fixed + +- `/evals:plugin-eval`, `/evals:methodology` and `/evals:design` descriptions now name the jobs their eval cases showed the model answering without the skill: planning headless eval runs, judging whether an eval delta or validity verdict can be shared, writing a success criterion, and writing test cases for an LLM feature. `methodology` keeps its claude-api routing clause first and now says what it is preferred for: before a suite or its criteria exist. diff --git a/plugins/evals/skills/design/SKILL.md b/plugins/evals/skills/design/SKILL.md index 945dabccd74..a3979547ae3 100644 --- a/plugins/evals/skills/design/SKILL.md +++ b/plugins/evals/skills/design/SKILL.md @@ -1,5 +1,5 @@ --- -description: "Design an evaluation suite for an LLM-based application or a Claude Code skill: interview for measurable success criteria, pick a grading method per criterion, and scaffold a criteria doc plus eval cases into the consumer repo. Use when: 'design evals', 'create an eval suite', 'scaffold evals', 'write evals for my skill', 'define success criteria for this app', 'set up LLM testing', 'build a test set for my prompt', 'evals for an LLM feature built on the Messages API', 'which eval route each part of a repo takes'. Not for eval-design theory questions (use /evals:methodology), not for statically validating an existing suite (use /evals:validate) or a skill-creator JSON eval file, and not for running or scoring a suite, which is /evals:plugin-eval and the CLI it guides." +description: "Design an evaluation suite or test cases for an LLM-based application or a Claude Code skill: interview for measurable success criteria before any case is written, pick a grading method per criterion, and scaffold a criteria doc plus eval cases into the consumer repo. Use when: writing test cases for an LLM feature or prompt, 'design evals', 'create an eval suite', 'scaffold evals', 'write evals for my skill', 'define success criteria for this app', 'set up LLM testing', 'build a test set for my prompt', 'evals for an LLM feature built on the Messages API', 'which eval route each part of a repo takes'. Not for eval-design theory questions (use /evals:methodology), not for statically validating an existing suite (use /evals:validate) or a skill-creator JSON eval file, and not for running or scoring a suite, which is /evals:plugin-eval and the CLI it guides." argument-hint: "|plugin >" user-invocable: true disable-model-invocation: false diff --git a/plugins/evals/skills/methodology/SKILL.md b/plugins/evals/skills/methodology/SKILL.md index 60b6a397656..ecb53562529 100644 --- a/plugins/evals/skills/methodology/SKILL.md +++ b/plugins/evals/skills/methodology/SKILL.md @@ -1,5 +1,5 @@ --- -description: "When the bundled claude-api skill resolves in this session, prefer its hillclimb for sweeping model and effort against an existing eval suite; this skill when the suite does not exist yet or its criteria are not yet measurable, before handing the suite to that search. Answers LLM-evaluation design questions from Anthropic's official evaluation guidance: success criteria, eval-suite design, grading methods, and the eval route for each part of a repository. Use when: 'define success criteria', 'how do I eval this', 'LLM eval', 'measure prompt quality', 'LLM judge', 'model-graded eval', 'golden answer', 'grading rubric', 'eval grading method', 'exact match vs LLM-graded', 'how many eval cases', 'rewrite this success criterion', 'evals for a service that calls the Messages API', 'evals for the whole repo'. Knowledge, not a runner; for scaffolding a suite use /evals:design, and for running and scoring a plugin's suite against a no-plugin baseline use /evals:plugin-eval." +description: "When the bundled claude-api skill resolves in this session, prefer its hillclimb for sweeping model and effort over a measurable suite; this skill before that suite or its criteria exist. Answers LLM-evaluation design questions from Anthropic's evaluation guidance: success criteria (including turning a vague quality goal into a measurable target before any data exists), eval-suite design, grading methods, and the eval route for each part of a repository. Use when: an eval success criterion, metric or target is being written or rewritten, with or without data (load this before answering), 'how do I eval this', 'LLM eval', 'measure prompt quality', 'LLM judge', 'model-graded eval', 'golden answer', 'grading rubric', 'eval grading method', 'exact match vs LLM-graded', 'how many eval cases', 'evals for a service that calls the Messages API', 'evals for the whole repo'. Knowledge, not a runner; scaffolding a suite is /evals:design, and running a plugin's suite against a no-plugin baseline is /evals:plugin-eval." argument-hint: "[question or concept]" user-invocable: true disable-model-invocation: false diff --git a/plugins/evals/skills/plugin-eval/SKILL.md b/plugins/evals/skills/plugin-eval/SKILL.md index b4dd060b152..bfb6d644788 100644 --- a/plugins/evals/skills/plugin-eval/SKILL.md +++ b/plugins/evals/skills/plugin-eval/SKILL.md @@ -1,5 +1,5 @@ --- -description: "Guided practice around the `claude plugin eval` CLI, which runs and scores a plugin's eval suite: preflight, no-spend validation, a cost estimate under the ceiling, the run, and reading the with-versus-without delta. Use when: 'run my plugin evals', 'plugin eval', 'evaluate this plugin', 'eval my skill', 'does my skill actually fire', 'what is the delta', 'read my eval results', 'compare two eval runs', 'did my change make the skill better', 'is this gain real', 'aggregate-result.json', 'eval CI gate', 'can this machine run evals', 'how much will this eval cost', 'Bash refuses claude plugin eval', 'plugin eval blocked in a worktree', 'plugin eval refuses: too many worktrees', 'can plugin eval measure CLAUDE.md or rules'. Not for designing success criteria (/evals:design) or the skill-creator JSON eval format." +description: "Guided practice around the `claude plugin eval` CLI, which runs and scores a plugin's eval suite: preflight, no-spend validation, a cost estimate under the ceiling, the run, its validity verdict, and reading the with-versus-without delta. Use when: planning how a headless `claude -p` session launches eval runs, an eval run's delta, arm scores or validity verdict is in question, including whether it can be shared (load this before answering), 'run my plugin evals', 'plugin eval', 'evaluate this plugin', 'eval my skill', 'does my skill actually fire', 'compare two eval runs', 'did my change make the skill better', 'is this gain real', 'aggregate-result.json', 'eval CI gate', 'can this machine run evals', 'how much will this eval cost', 'Bash refuses claude plugin eval', 'plugin eval blocked in a worktree', 'plugin eval refuses: too many worktrees', 'can plugin eval measure CLAUDE.md or rules'. Not for designing success criteria (/evals:design) or the skill-creator JSON eval format." argument-hint: "[preflight|validate|run|read |ci|init] [target]" user-invocable: true disable-model-invocation: false