glennfaison · Nov 12, 2026 · Nov 12, 2026
@@ -0,0 +1,5 @@
+---
+"github-agent-workflow": minor
+---
+
+`npm run eval -- <skill>/<scenario> --model <model>` runs a model through a skill against the `gh` simulator and judges what it did with the scenario checker. It reports a pass, a deviation from the scenario's golden, or a violation of the skill's contract, with the diff of effects and the skills the model called. It is opt-in (it needs `ANTHROPIC_API_KEY` and costs money), capped by `--max-runs`, `--max-tokens` and `--max-turns`, never part of CI, and keeps transcripts in the git-ignored `.eval/`.
@@ -1 +1,2 @@
 node_modules
+.eval/
@@ -40,6 +40,8 @@ skills/<bucket>/<skill>/SKILL.md     exactly three levels deep
 
 **Every `gh` command a skill uses is listed in `tests/sim/surface.mjs`.** The `gh` simulator (`tests/sim/bin/gh`) answers `gh` from a world file, and a test greps `skills/` and `scripts/` for `gh` calls. A new one fails `npm test` until it is listed as `supported` (and the simulator answers it) or `out-of-scope`. Use only `--json` fields that `gh` has. `tests/lib.test.sh` runs against the same simulator, so there is one mock of `gh`. See `docs/adr/0014-gh-simulator.md`.
 
+**A model can be run through a skill, and the outcome checked.** `npm run eval -- <skill>/<scenario> --model <model>` seeds the simulator from a scenario's `world`, lets a model work the skill, and judges what it left behind with the same checker: a **violation** of the contract, a **deviation** from the scenario's golden (the golden may be wrong), or a pass. It is opt-in: it needs `ANTHROPIC_API_KEY`, it costs money (`--max-runs` and `--max-tokens` cap it, 500000 tokens by default), and it is never part of `npm test` or CI. Transcripts go in `.eval/`, which git ignores. See `docs/adr/0016-model-evals.md`.
+
 ## Inputs and defaults
 
 **Every skill has an `## Inputs` section.** It names the inputs the skill accepts, in two interchangeable forms: **key-value pairs** (`key=value` on a line of its own) and **free text** (whatever the prompt says). It also names the skill's **implicit dependencies**: the tools, skills and repository state it needs that are not inputs. Every pipeline skill lists `runner` and `unattended`, and every skill that reads markers lists `trusted_authors`.
@@ -72,6 +74,7 @@ npm test                      # starts with hooks:check
 npm run sync:check
 npm run check-plugin-version
 npm run check                 # lint, validate, check-plugin-version and test: everything CI runs
+npm run eval -- <skill>/<scenario> --model <model>   # not a check: runs a model, costs money (ANTHROPIC_API_KEY)
 ```
 
 ## Git hooks

@@ -93,6 +93,25 @@ Every skill takes its inputs two ways, mixed freely in one prompt:
 
 `groom_batch` and `issues_per_run` were renamed `triage_batch` and `explore_batch`. The old names still work until the next major version.
 
+## Evaluating a skill against a model
+
+`npm test` checks a skill's text and its scenarios, but never watches a model follow it. To do that, run a model against the `gh` simulator:
+
+```bash
+export ANTHROPIC_API_KEY=...
+npm run eval -- shepherd-github-pr/conflict-without-checks --model <model>
+npm run eval -- shepherd-github-pr --model <model> --max-runs 3 --max-tokens 300000
+```
+
+The model is given the skill, the scenario's inputs, a `Skill` tool and a shell where `gh` is the simulator and `origin` is a local repository. What it leaves behind is judged with the scenario checker, and the report is one of **pass**, **deviation** (it differs from the scenario's golden without breaking a rule, and the golden may be the one that is wrong) or **violation** (it breaks the skill's contract), with the diff of effects and the skills it called.
+
+- **It is opt-in and costs money.** Without `ANTHROPIC_API_KEY` it stops and says so. Every request of a run is billed, and each one carries the whole conversation, so a run of a pipeline skill is typically in the low hundreds of thousands of tokens. `--max-tokens` (default 500000) caps the tokens of the whole invocation, `--max-turns` (default 60) caps one run, and `--max-runs` (default 1) caps how many scenarios a skill name runs. A run that hits a cap is reported as incomplete, not judged. It is never part of `npm test` or CI.
+- **You choose the model:** `--model` or `EVAL_MODEL`. There is no default.
+- **It runs the model's shell commands on your machine**, in a temporary directory with only `PATH` passed on (no API key, no GitHub token). That is not a sandbox against a hostile skill; use a container for one you do not trust.
+- **Try a changed skill without editing the package:** `--skills-dir <copy of skills/>` shows the model that copy while the verdict still comes from the package's contract. A copy that skips a conflicted PR is reported as a violation.
+- **Transcripts** (messages, the `Skill` calls, the `gh` call log, the verdict) are written to `.eval/transcripts/`, which git ignores.
+- Only item skills are run so far. A world field that stands for prose (`openQuestions`, `useCase` and the like) is not visible through `gh`, and the report says when a scenario has one.
+
 ## Security
 
 This pipeline merges code with nobody watching, and anyone can comment on a public issue. So the skills treat issue, PR and comment text as data, never as instructions, and they believe a lease, handoff or failure marker only when a repository owner, member or collaborator, the agent's own account, or a login in `trusted_authors` wrote it. A marker from anyone else is ignored. If the agent runs as a GitHub App, put its bot login in `trusted_authors`. See **Trust** in the protocol, and [ADR 0006](./docs/adr/0006-trust-boundary-for-markers.md).

@@ -76,7 +76,7 @@ A world (and so a scenario's `world:`) may carry `faults: [{match, fail, exit?,
 ## What it does not do
 
 - **It is not GitHub.** Rate limits, pagination, webhooks, merge queues, branch protection and reviews are not modelled. A merge is blocked by the merge state a world gives, nothing else. Some error messages are what `gh` prints and some are close to it (a merge of a closed PR, an unreadable body file).
-- **Some world fields cannot be shown.** `entries`, `kind`, `openQuestions`, `useCase` and `wontfix` stand for facts a model would read from prose (a log entry, a body), not for GitHub state. `seed.mjs` says which ones a world has. A scenario that depends on them needs a runner to write the prose (#61).
+- **Some world fields cannot be shown.** `entries`, `kind`, `openQuestions`, `useCase` and `wontfix` stand for facts a model would read from prose (a log entry, a body), not for GitHub state. `seed.mjs` says which ones a world has. A scenario that depends on them needs a runner to write the prose. `npm run eval` (ADR 0016, #61) does not do that yet: it reports that the scenario has such fields.
 - **A world has no branch names, check names or logs.** A PR is on `feat/<closed issue>-pr-<N>`, which matches the pattern `prs_for_issue` reads, unless it names a `branch`. A check is `check-<i>`. The log of a failing run is one fixed line.
 - **Its field lists are a copy.** They come from one `gh` version and can fall behind.
 - **It is slow next to a shell function.** Each call starts Node (about 75 ms, half of it loading `yaml` through the checker). `tests/lib.test.sh` makes about 390 calls and takes about 45 seconds, where the stub took 10. A compile cache saved little (12 ms a call), so it is not used.

@@ -0,0 +1,37 @@
+# 0016. Run a model through a skill against the simulator, and check the outcome
+
+## Status
+
+Accepted and built (#61). It follows ADR 0012 (scenarios), 0014 (the `gh` simulator) and the `diffWorlds` of #59.
+
+## Context
+
+A scenario says what a skill should do in a world, and `checkScenario` checks that statement against the contract. Nothing checked what a model does. A maintainer who changes a skill could not tell whether a model still follows the contract, short of pointing one at a real repository.
+
+## Decision
+
+`npm run eval -- <skill>/<scenario> --model <model>` (`scripts/eval.mjs`):
+
+1. Seeds a world file from the scenario's `world` (`tests/sim/seed.mjs`), and makes a sandbox: a bare `origin` and a checkout with a branch per open PR, so `git push` works and a push can be seen.
+2. Gives a model the skill's name, the scenario's `inputs` (the target as the skill's own parameter, `unattended=true`, `write_log=false` and `runner=eval` unless the scenario says otherwise), a `Skill` tool and a `bash` tool. `gh` there is the simulator. Every `Skill` call is recorded.
+3. Reads back the last result line of the final answer, the skills called (not the entry skill, and not a `kind: reference` skill that is only read), and `diffWorlds(before, after)`. A head that moved on `origin` is a push. The `agent:lease` comment is left out of the diff, since the checker adds a claim's lease itself.
+4. Judges it two ways. `checkScenario` on the run (its result, effects and `calls` in place of the golden's) gives **violations** of the contract. A comparison with the golden gives **deviations**: another outcome, an effect missing or extra (by op, target, label and marker; not by a sha or the words of a comment), other skills called, a different `rerun`. A run with a violation is a violation, one with only deviations is a deviation, and otherwise it passes.
+
+Constraints, each tested:
+
+- **Opt-in.** It does nothing without `ANTHROPIC_API_KEY` and says so, before anything is made. It is not in `npm test` or CI. The tests use a scripted runner and a fake `fetch`.
+- **Cost-capped.** `--max-tokens` is a budget for the whole invocation, counted from the usage of every request. `--max-turns` caps a run and `--max-runs` the scenarios of a skill name. A run that reaches a cap is `incomplete` and not judged.
+- **The model is an input.** `--model` or `EVAL_MODEL`, with no default.
+- **Transcripts** are written to `.eval/transcripts/`, git-ignored.
+- **The model's shell gets only `PATH`** (plus the simulator's variables and a git identity), so the API key and a GitHub token are not in its environment.
+- **The runner is a seam.** `main` takes a `runner`, so a test (or another harness) can stand in for the model and still use the real sandbox, simulator and judge.
+- **`--skills-dir`** gives the model another copy of the skills while the contract comes from the package, which is how a deliberately wrong skill is tried.
+
+## What it does not do
+
+- **It is not a sandbox.** The model's commands run on the machine, in a temporary directory. Use a container for a skill that is not trusted.
+- **It runs item skills only.** Group and utility scenarios expect dispatches and documents, not one run's effects.
+- **It cannot show prose-only world fields** (ADR 0014). The report notes a scenario that has them, since a deviation may come from that.
+- **One run is one sample.** A model is not deterministic, so a single pass proves little and a single deviation proves little. `--max-runs` does not repeat a scenario.
+- **A fixed prompt.** The model is asked to call the skill and end with its result line. How a real scheduler prompts it is in `docs/RUNNERS.md` and the runner prompts, and is not reproduced here.
+- **Costs are estimates by token, not money.** Prices change, so the report counts tokens.
@@ -8,7 +8,7 @@
     "validate": "node scripts/validate-skills.mjs",
     "lint": "node scripts/lint-files.mjs --all",
     "pretest": "node scripts/check-hooks.mjs",
-    "test": "bash tests/lib.test.sh && node --test tests/helpers.test.mjs tests/lint.test.mjs tests/contracts.test.mjs tests/scenarios.test.mjs tests/diff-worlds.test.mjs tests/gh-sim.test.mjs",
+    "test": "bash tests/lib.test.sh && node --test tests/helpers.test.mjs tests/lint.test.mjs tests/contracts.test.mjs tests/scenarios.test.mjs tests/diff-worlds.test.mjs tests/gh-sim.test.mjs tests/eval.test.mjs",
     "check-plugin-version": "node scripts/sync-plugin-version.mjs --check",
     "check": "npm run lint && npm run validate && npm run check-plugin-version && npm test",
     "hooks:check": "node scripts/check-hooks.mjs",
@@ -18,7 +18,8 @@
     "version": "changeset version && node scripts/sync-plugin-version.mjs",
     "sync": "node scripts/sync.mjs",
     "sync:check": "node scripts/sync.mjs --check",
-    "scenarios:coverage": "node scripts/scenario-coverage.mjs"
+    "scenarios:coverage": "node scripts/scenario-coverage.mjs",
+    "eval": "node scripts/eval.mjs"
   },
   "dependencies": {
     "mattpocock-skills": "1.3.0"