[glennfaison]

Closes #61

Adds npm run eval -- <skill>/<scenario> --model <model>: it runs a model through a skill against the gh simulator and judges what it left behind with the scenario checker.

How it works

1. Seeds a world file from the scenario's world and builds a sandbox: a bare origin plus a checkout with a branch per open PR, so git push works and a push is visible.
2. Gives the model the skill name, the scenario's inputs, a Skill tool and a bash tool (simulator first on PATH). Every Skill call is recorded; the entry skill and kind: reference skills are not counted as calls.
3. Reads back the last result line, the call log and diffWorlds(before, after).
4. checkScenario on that gives violations of the contract. A comparison with the golden (result, effects by op/target/label/marker, calls, rerun) gives deviations. Report is pass, deviation or violation, with the effect diff.

Constraints from the issue

• Opt-in: without ANTHROPIC_API_KEY it exits 2 and says so before making anything. Not in npm test's runs of the API, not in CI.
• Cost-capped: --max-tokens (whole invocation, default 500000), --max-turns (per run), --max-runs. A run that hits a cap is incomplete, not judged.
• Model is an input: --model / EVAL_MODEL, no default.
• Transcripts go to .eval/transcripts/ (git-ignored).
• The model's shell gets only PATH (no API key, no GitHub token).
• --skills-dir <copy> shows the model a modified copy of the skills while the verdict still comes from the package's contract; this is how to try a deliberately wrong skill.

Acceptance criteria

• Runs end to end and prints pass / deviation / violation with the diff. Verified with a scripted runner driving the real sandbox, simulator and judge on shepherd-github-pr/conflict-without-checks.
• A skill that skips a conflicted PR is reported as a violation (the oracle says pr#40 is eligible, but the result is ...). Verified with a scripted runner; see below.
• The call log records which skills were called and checkScenario's calls check uses it.
• Does not run without an API key and says so.
• README, AGENTS.md and ADR 0016 say how to run it and what it costs.

Not verified here

No model was run. There was no API key available, so apiRunner is tested against a fake fetch (tool loop, token accounting, caps, API errors) and the judge against scripted runs. A first real run is the remaining check. In particular, the first two criteria are shown with a stand-in for the model, not an actual one.

Tests

tests/eval.test.mjs (22 tests, added to npm test), including: every golden of every item skill, applied as a run's effects, judges as a pass. npm run check is green.

Limits (ADR 0016)

Item skills only; prose-only world fields are not visible to the model (the report says when a scenario has them); one run is one sample; the model's commands run on the host (use a container for untrusted skills).