diff --git a/.changeset/model-evals.md b/.changeset/model-evals.md new file mode 100644 index 0000000..d4752c7 --- /dev/null +++ b/.changeset/model-evals.md @@ -0,0 +1,5 @@ +--- +"github-agent-workflow": minor +--- + +`npm run eval -- / --model ` runs a model through a skill against the `gh` simulator and judges what it did with the scenario checker. It reports a pass, a deviation from the scenario's golden, or a violation of the skill's contract, with the diff of effects and the skills the model called. It is opt-in (it needs `ANTHROPIC_API_KEY` and costs money), capped by `--max-runs`, `--max-tokens` and `--max-turns`, never part of CI, and keeps transcripts in the git-ignored `.eval/`. diff --git a/.gitignore b/.gitignore index 3c3629e..420a573 100644 --- a/.gitignore +++ b/.gitignore @@ -1 +1,2 @@ node_modules +.eval/ diff --git a/AGENTS.md b/AGENTS.md index 6410dd2..af6bec1 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -40,6 +40,8 @@ skills///SKILL.md exactly three levels deep **Every `gh` command a skill uses is listed in `tests/sim/surface.mjs`.** The `gh` simulator (`tests/sim/bin/gh`) answers `gh` from a world file, and a test greps `skills/` and `scripts/` for `gh` calls. A new one fails `npm test` until it is listed as `supported` (and the simulator answers it) or `out-of-scope`. Use only `--json` fields that `gh` has. `tests/lib.test.sh` runs against the same simulator, so there is one mock of `gh`. See `docs/adr/0014-gh-simulator.md`. +**A model can be run through a skill, and the outcome checked.** `npm run eval -- / --model ` seeds the simulator from a scenario's `world`, lets a model work the skill, and judges what it left behind with the same checker: a **violation** of the contract, a **deviation** from the scenario's golden (the golden may be wrong), or a pass. It is opt-in: it needs `ANTHROPIC_API_KEY`, it costs money (`--max-runs` and `--max-tokens` cap it, 500000 tokens by default), and it is never part of `npm test` or CI. Transcripts go in `.eval/`, which git ignores. See `docs/adr/0016-model-evals.md`. + ## Inputs and defaults **Every skill has an `## Inputs` section.** It names the inputs the skill accepts, in two interchangeable forms: **key-value pairs** (`key=value` on a line of its own) and **free text** (whatever the prompt says). It also names the skill's **implicit dependencies**: the tools, skills and repository state it needs that are not inputs. Every pipeline skill lists `runner` and `unattended`, and every skill that reads markers lists `trusted_authors`. @@ -72,6 +74,7 @@ npm test # starts with hooks:check npm run sync:check npm run check-plugin-version npm run check # lint, validate, check-plugin-version and test: everything CI runs +npm run eval -- / --model # not a check: runs a model, costs money (ANTHROPIC_API_KEY) ``` ## Git hooks diff --git a/README.md b/README.md index 1f32fc6..dc1a95c 100644 --- a/README.md +++ b/README.md @@ -93,6 +93,25 @@ Every skill takes its inputs two ways, mixed freely in one prompt: `groom_batch` and `issues_per_run` were renamed `triage_batch` and `explore_batch`. The old names still work until the next major version. +## Evaluating a skill against a model + +`npm test` checks a skill's text and its scenarios, but never watches a model follow it. To do that, run a model against the `gh` simulator: + +```bash +export ANTHROPIC_API_KEY=... +npm run eval -- shepherd-github-pr/conflict-without-checks --model +npm run eval -- shepherd-github-pr --model --max-runs 3 --max-tokens 300000 +``` + +The model is given the skill, the scenario's inputs, a `Skill` tool and a shell where `gh` is the simulator and `origin` is a local repository. What it leaves behind is judged with the scenario checker, and the report is one of **pass**, **deviation** (it differs from the scenario's golden without breaking a rule, and the golden may be the one that is wrong) or **violation** (it breaks the skill's contract), with the diff of effects and the skills it called. + +- **It is opt-in and costs money.** Without `ANTHROPIC_API_KEY` it stops and says so. Every request of a run is billed, and each one carries the whole conversation, so a run of a pipeline skill is typically in the low hundreds of thousands of tokens. `--max-tokens` (default 500000) caps the tokens of the whole invocation, `--max-turns` (default 60) caps one run, and `--max-runs` (default 1) caps how many scenarios a skill name runs. A run that hits a cap is reported as incomplete, not judged. It is never part of `npm test` or CI. +- **You choose the model:** `--model` or `EVAL_MODEL`. There is no default. +- **It runs the model's shell commands on your machine**, in a temporary directory with only `PATH` passed on (no API key, no GitHub token). That is not a sandbox against a hostile skill; use a container for one you do not trust. +- **Try a changed skill without editing the package:** `--skills-dir ` shows the model that copy while the verdict still comes from the package's contract. A copy that skips a conflicted PR is reported as a violation. +- **Transcripts** (messages, the `Skill` calls, the `gh` call log, the verdict) are written to `.eval/transcripts/`, which git ignores. +- Only item skills are run so far. A world field that stands for prose (`openQuestions`, `useCase` and the like) is not visible through `gh`, and the report says when a scenario has one. + ## Security This pipeline merges code with nobody watching, and anyone can comment on a public issue. So the skills treat issue, PR and comment text as data, never as instructions, and they believe a lease, handoff or failure marker only when a repository owner, member or collaborator, the agent's own account, or a login in `trusted_authors` wrote it. A marker from anyone else is ignored. If the agent runs as a GitHub App, put its bot login in `trusted_authors`. See **Trust** in the protocol, and [ADR 0006](./docs/adr/0006-trust-boundary-for-markers.md). diff --git a/docs/adr/0014-gh-simulator.md b/docs/adr/0014-gh-simulator.md index a7efa36..9db987d 100644 --- a/docs/adr/0014-gh-simulator.md +++ b/docs/adr/0014-gh-simulator.md @@ -76,7 +76,7 @@ A world (and so a scenario's `world:`) may carry `faults: [{match, fail, exit?, ## What it does not do - **It is not GitHub.** Rate limits, pagination, webhooks, merge queues, branch protection and reviews are not modelled. A merge is blocked by the merge state a world gives, nothing else. Some error messages are what `gh` prints and some are close to it (a merge of a closed PR, an unreadable body file). -- **Some world fields cannot be shown.** `entries`, `kind`, `openQuestions`, `useCase` and `wontfix` stand for facts a model would read from prose (a log entry, a body), not for GitHub state. `seed.mjs` says which ones a world has. A scenario that depends on them needs a runner to write the prose (#61). +- **Some world fields cannot be shown.** `entries`, `kind`, `openQuestions`, `useCase` and `wontfix` stand for facts a model would read from prose (a log entry, a body), not for GitHub state. `seed.mjs` says which ones a world has. A scenario that depends on them needs a runner to write the prose. `npm run eval` (ADR 0016, #61) does not do that yet: it reports that the scenario has such fields. - **A world has no branch names, check names or logs.** A PR is on `feat/-pr-`, which matches the pattern `prs_for_issue` reads, unless it names a `branch`. A check is `check-`. The log of a failing run is one fixed line. - **Its field lists are a copy.** They come from one `gh` version and can fall behind. - **It is slow next to a shell function.** Each call starts Node (about 75 ms, half of it loading `yaml` through the checker). `tests/lib.test.sh` makes about 390 calls and takes about 45 seconds, where the stub took 10. A compile cache saved little (12 ms a call), so it is not used. diff --git a/docs/adr/0016-model-evals.md b/docs/adr/0016-model-evals.md new file mode 100644 index 0000000..6162231 --- /dev/null +++ b/docs/adr/0016-model-evals.md @@ -0,0 +1,37 @@ +# 0016. Run a model through a skill against the simulator, and check the outcome + +## Status + +Accepted and built (#61). It follows ADR 0012 (scenarios), 0014 (the `gh` simulator) and the `diffWorlds` of #59. + +## Context + +A scenario says what a skill should do in a world, and `checkScenario` checks that statement against the contract. Nothing checked what a model does. A maintainer who changes a skill could not tell whether a model still follows the contract, short of pointing one at a real repository. + +## Decision + +`npm run eval -- / --model ` (`scripts/eval.mjs`): + +1. Seeds a world file from the scenario's `world` (`tests/sim/seed.mjs`), and makes a sandbox: a bare `origin` and a checkout with a branch per open PR, so `git push` works and a push can be seen. +2. Gives a model the skill's name, the scenario's `inputs` (the target as the skill's own parameter, `unattended=true`, `write_log=false` and `runner=eval` unless the scenario says otherwise), a `Skill` tool and a `bash` tool. `gh` there is the simulator. Every `Skill` call is recorded. +3. Reads back the last result line of the final answer, the skills called (not the entry skill, and not a `kind: reference` skill that is only read), and `diffWorlds(before, after)`. A head that moved on `origin` is a push. The `agent:lease` comment is left out of the diff, since the checker adds a claim's lease itself. +4. Judges it two ways. `checkScenario` on the run (its result, effects and `calls` in place of the golden's) gives **violations** of the contract. A comparison with the golden gives **deviations**: another outcome, an effect missing or extra (by op, target, label and marker; not by a sha or the words of a comment), other skills called, a different `rerun`. A run with a violation is a violation, one with only deviations is a deviation, and otherwise it passes. + +Constraints, each tested: + +- **Opt-in.** It does nothing without `ANTHROPIC_API_KEY` and says so, before anything is made. It is not in `npm test` or CI. The tests use a scripted runner and a fake `fetch`. +- **Cost-capped.** `--max-tokens` is a budget for the whole invocation, counted from the usage of every request. `--max-turns` caps a run and `--max-runs` the scenarios of a skill name. A run that reaches a cap is `incomplete` and not judged. +- **The model is an input.** `--model` or `EVAL_MODEL`, with no default. +- **Transcripts** are written to `.eval/transcripts/`, git-ignored. +- **The model's shell gets only `PATH`** (plus the simulator's variables and a git identity), so the API key and a GitHub token are not in its environment. +- **The runner is a seam.** `main` takes a `runner`, so a test (or another harness) can stand in for the model and still use the real sandbox, simulator and judge. +- **`--skills-dir`** gives the model another copy of the skills while the contract comes from the package, which is how a deliberately wrong skill is tried. + +## What it does not do + +- **It is not a sandbox.** The model's commands run on the machine, in a temporary directory. Use a container for a skill that is not trusted. +- **It runs item skills only.** Group and utility scenarios expect dispatches and documents, not one run's effects. +- **It cannot show prose-only world fields** (ADR 0014). The report notes a scenario that has them, since a deviation may come from that. +- **One run is one sample.** A model is not deterministic, so a single pass proves little and a single deviation proves little. `--max-runs` does not repeat a scenario. +- **A fixed prompt.** The model is asked to call the skill and end with its result line. How a real scheduler prompts it is in `docs/RUNNERS.md` and the runner prompts, and is not reproduced here. +- **Costs are estimates by token, not money.** Prices change, so the report counts tokens. diff --git a/package.json b/package.json index 33d18d9..f86f5ab 100644 --- a/package.json +++ b/package.json @@ -8,7 +8,7 @@ "validate": "node scripts/validate-skills.mjs", "lint": "node scripts/lint-files.mjs --all", "pretest": "node scripts/check-hooks.mjs", - "test": "bash tests/lib.test.sh && node --test tests/helpers.test.mjs tests/lint.test.mjs tests/contracts.test.mjs tests/scenarios.test.mjs tests/diff-worlds.test.mjs tests/gh-sim.test.mjs", + "test": "bash tests/lib.test.sh && node --test tests/helpers.test.mjs tests/lint.test.mjs tests/contracts.test.mjs tests/scenarios.test.mjs tests/diff-worlds.test.mjs tests/gh-sim.test.mjs tests/eval.test.mjs", "check-plugin-version": "node scripts/sync-plugin-version.mjs --check", "check": "npm run lint && npm run validate && npm run check-plugin-version && npm test", "hooks:check": "node scripts/check-hooks.mjs", @@ -18,7 +18,8 @@ "version": "changeset version && node scripts/sync-plugin-version.mjs", "sync": "node scripts/sync.mjs", "sync:check": "node scripts/sync.mjs --check", - "scenarios:coverage": "node scripts/scenario-coverage.mjs" + "scenarios:coverage": "node scripts/scenario-coverage.mjs", + "eval": "node scripts/eval.mjs" }, "dependencies": { "mattpocock-skills": "1.3.0" diff --git a/scripts/eval.mjs b/scripts/eval.mjs new file mode 100644 index 0000000..6e5c020 --- /dev/null +++ b/scripts/eval.mjs @@ -0,0 +1,332 @@ +// Runs a model through a skill against the gh simulator and checks what it did. +// npm run eval -- / --model +// npm run eval -- --model --max-runs 3 +// Opt-in: it calls the Anthropic API (ANTHROPIC_API_KEY) and costs money. It is not part of `npm test` or CI. +// See docs/adr/0016-model-evals.md. +// +// A run: the scenario's `world` is seeded into a world file, a model is given the skill, the scenario's `inputs`, a `Skill` +// tool, a `bash` tool and the simulator first on PATH, and its transcript is recorded. What it left behind is read back as +// the final result line, the skills it called, and `diffWorlds(before, after)`. `checkScenario` judges that against the +// skill's contract (a violation), and it is compared with the scenario's golden (a deviation: the golden may be wrong). + +import { readFileSync, readdirSync, existsSync, mkdirSync, mkdtempSync, writeFileSync, rmSync } from "node:fs"; +import { join, resolve } from "node:path"; +import { tmpdir } from "node:os"; +import { execFileSync, spawnSync } from "node:child_process"; +import { pathToFileURL } from "node:url"; +import { parse } from "yaml"; +import { PKG, skillDirs, contractOf } from "./skill-contracts.mjs"; +import { checkScenario, diffWorlds, expandScenario, matchesReturn, rerunEligibility, ref } from "./scenario.mjs"; +import { seed } from "../tests/sim/seed.mjs"; +import { loadWorld, branchOf } from "../tests/sim/world.mjs"; + +const SIM_BIN = join(PKG, "tests/sim/bin"); +const API = "https://api.anthropic.com/v1/messages"; + +export const DEFAULTS = { max_runs: 1, max_tokens: 500000, max_turns: 60, out: join(PKG, ".eval") }; + +// ---- which scenarios ---------------------------------------------------------------------------------------- + +/** `/` is one scenario, `` is all of that skill's. @returns {{skill, name, file}[]} */ +export function findScenarios(spec, root = join(PKG, "tests/scenarios")) { + const [skill, name] = spec.replace(/\.ya?ml$/, "").split("/"); + const dir = join(root, skill ?? ""); + if (!skill || !existsSync(dir)) throw new Error(`no scenarios for \`${skill}\` in ${root}`); + const files = readdirSync(dir).filter((f) => f.endsWith(".yaml")).sort(); + const picked = name ? files.filter((f) => f === `${name}.yaml`) : files; + if (picked.length === 0) throw new Error(`no scenario \`${name}\` for ${skill}. It has: ${files.map((f) => f.replace(/\.yaml$/, "")).join(", ")}`); + return picked.map((f) => ({ skill, name: f.replace(/\.yaml$/, ""), file: join(dir, f) })); +} + +// ---- the sandbox -------------------------------------------------------------------------------------------- + +/** The environment a command of the model runs in. Nothing of the caller's is passed on but PATH: no API key, no GitHub token. */ +function sandboxEnv(sb) { + return { + PATH: `${SIM_BIN}:${process.env.PATH}`, HOME: sb.home, LANG: "C.UTF-8", GIT_TERMINAL_PROMPT: "0", + GH_SIM_WORLD: sb.worldFile, GH_SIM_LOG: sb.logFile, + GIT_AUTHOR_NAME: "eval", GIT_AUTHOR_EMAIL: "eval@example.invalid", GIT_COMMITTER_NAME: "eval", GIT_COMMITTER_EMAIL: "eval@example.invalid", + }; +} +const git = (cwd, env, ...args) => execFileSync("git", args, { cwd, env, encoding: "utf8", stdio: ["ignore", "pipe", "pipe"] }).trim(); + +/** + * A directory for one run: the world file (a copy the simulator may write to), a bare `origin`, and a checkout of it with one + * branch per open PR. The world's heads are names (`h1`), so the commit each branch really got is kept to tell a push afterwards. + */ +export function makeSandbox(scenarioFile, { root = tmpdir() } = {}) { + const dir = mkdtempSync(join(root, "eval-")); + const sb = { dir, worldFile: join(dir, "world.yaml"), logFile: join(dir, "gh.log"), origin: join(dir, "origin.git"), work: join(dir, "work"), home: join(dir, "home"), shas: {} }; + mkdirSync(sb.home); + const { notShown } = seed(scenarioFile, sb.worldFile); + sb.notShown = notShown; + sb.before = loadWorld(sb.worldFile).world; + const env = sandboxEnv(sb), base = sb.before.default_branch ?? "main"; + git(dir, env, "init", "-q", "--bare", "-b", base, sb.origin); + git(dir, env, "init", "-q", "-b", base, sb.work); + writeFileSync(join(sb.work, "README.md"), "# sandbox\n"); + git(sb.work, env, "add", "-A"); git(sb.work, env, "commit", "-q", "-m", "initial"); + git(sb.work, env, "remote", "add", "origin", sb.origin); + git(sb.work, env, "push", "-q", "origin", base); + for (const pr of sb.before.prs.filter((p) => (p.state ?? "open") === "open" && !p.fork)) { + git(sb.work, env, "checkout", "-q", "-b", branchOf(pr)); + git(sb.work, env, "commit", "-q", "--allow-empty", "-m", `PR #${pr.number}`); + git(sb.work, env, "push", "-q", "origin", branchOf(pr)); + sb.shas[pr.number] = git(sb.work, env, "rev-parse", "HEAD"); + git(sb.work, env, "checkout", "-q", base); + } + return sb; +} + +/** The world the run left, the simulator's call log, and which PR branches were pushed to `origin`. */ +export function collect(sb) { + const after = loadWorld(sb.worldFile).world; + for (const pr of sb.before.prs) { + if (!sb.shas[pr.number]) continue; + let now = null; + try { now = git(sb.origin, sandboxEnv(sb), "rev-parse", "--verify", "-q", `refs/heads/${branchOf(pr)}`); } catch { /* the branch is gone */ } + const a = after.prs.find((p) => p.number === pr.number); + // A head the simulator itself moved (update-branch) is already different. A push through git is only visible here. + if (a && now && now !== sb.shas[pr.number] && a.head === pr.head) a.head = now; + } + let ghLog = []; + try { ghLog = readFileSync(sb.logFile, "utf8").trim().split("\n").filter(Boolean).map((l) => JSON.parse(l)); } catch { /* no gh call */ } + return { before: sb.before, after, ghLog }; +} + +// ---- the tools the model has --------------------------------------------------------------------------------- + +export const TOOLS = [ + { name: "Skill", description: "Load a skill by name. Returns its SKILL.md and the directory its other files (references/, scripts/) are read from.", input_schema: { type: "object", properties: { name: { type: "string" }, args: { type: "string", description: "Inputs for the skill, as key=value pairs, one per line" } }, required: ["name"] } }, + { name: "bash", description: "Run a shell command in the sandbox checkout. `gh` is a simulator of the GitHub CLI and `origin` is a local repository.", input_schema: { type: "object", properties: { command: { type: "string" } }, required: ["command"] } }, +]; + +/** `key=value` pairs, as a skill is given its inputs. */ +export function parseArgs(text = "") { + const out = {}; + for (const m of String(text).matchAll(/([A-Za-z_]\w*)=("[^"]*"|\S+)/g)) out[m[1]] = m[2].replace(/^"|"$/g, ""); + return out; +} + +/** + * `exec(tool, input)` for one run, and the record of what the model called. Skills are found by name in `skillsRoot` + * (any bucket), the way a harness finds them. A skill that is only read for its references (`kind: reference`) is not a call. + */ +export function makeTools(sb, { skillsRoot = join(PKG, "skills"), entry, timeoutMs = 120000 } = {}) { + const calls = [], unknown = []; + const dirs = new Map(readdirSync(skillsRoot, { withFileTypes: true }).filter((b) => b.isDirectory()).flatMap((b) => + readdirSync(join(skillsRoot, b.name), { withFileTypes: true }).filter((d) => d.isDirectory() && existsSync(join(skillsRoot, b.name, d.name, "SKILL.md"))).map((d) => [d.name, join(skillsRoot, b.name, d.name)]))); + const env = sandboxEnv(sb); + function exec(tool, input = {}) { + if (tool === "Skill") { + const dir = dirs.get(input.name); + if (!dir) { unknown.push(input.name); return `error: no skill named \`${input.name}\`. Skills: ${[...dirs.keys()].sort().join(", ")}`; } + const callee = contractOf(input.name); + if (input.name !== entry && callee?.kind !== "reference") { + const inputs = parseArgs(input.args), p = callee?.target?.param, n = p && String(inputs[p] ?? "").match(/(\d+)\D*$/)?.[1]; + calls.push({ skill: input.name, ...(n ? { target: `${callee.target.type}#${n}` } : {}), ...(Object.keys(inputs).length ? { inputs } : {}) }); + } + return `Base directory for this skill: ${dir}\n\n${readFileSync(join(dir, "SKILL.md"), "utf8")}`; + } + if (tool === "bash") { + const r = spawnSync("bash", ["-c", String(input.command ?? "")], { cwd: sb.work, env, encoding: "utf8", timeout: timeoutMs, maxBuffer: 8 << 20 }); + const text = `${r.stdout ?? ""}${r.stderr ?? ""}`; + const cut = text.length > 20000 ? `${text.slice(0, 10000)}\n[... ${text.length - 20000} characters cut ...]\n${text.slice(-10000)}` : text; + return `${cut}${r.error ? `\n[${r.error.code === "ETIMEDOUT" ? `timed out after ${timeoutMs / 1000}s` : r.error.message}]` : ""}\n[exit ${r.status ?? r.signal}]`; + } + return `error: no tool named \`${tool}\``; + } + return { exec, calls, unknown }; +} + +// ---- what the model is asked --------------------------------------------------------------------------------- + +const fmt = (v) => (Array.isArray(v) ? v.join(",") : typeof v === "object" && v !== null ? JSON.stringify(v) : String(v)); + +/** The inputs a run is given: the scenario's, the target as the skill's own parameter, and the eval's defaults for what a scenario leaves out. */ +export function runInputs(sc, contract) { + const inputs = { unattended: true, write_log: false, runner: "eval", ...(sc.inputs ?? {}) }; + const p = contract.target?.param; + if (p && sc.target) inputs[p] = ref(sc.target).n; + return inputs; +} + +export function taskFor(sc, contract) { + const inputs = Object.entries(runInputs(sc, contract)).map(([k, v]) => `${k}=${fmt(v)}`).join("\n"); + return { + system: "You run in a sandbox, with no person to answer questions. `gh` is a simulator of the GitHub CLI that answers from a fixed repository state, and the git remote `origin` is a local repository. Use the Skill tool to load a skill by name and bash to run commands. When you are done, end your final message with the skill's result line, exactly as the skill specifies, on a line of its own.", + prompt: `Call the Skill tool with name \`${sc.skill}\` and follow it.\n\nInputs:\n${inputs}`, + }; +} + +// ---- the model --------------------------------------------------------------------------------------------- + +/** + * The default runner: a tool-use loop on the Messages API. `budget` ({used, max}) is shared by every run of an invocation and counts + * input and output tokens of every request, so it is what the invocation costs. A run stops, unfinished, when it is spent. + * @returns {{stop: "end"|"max-tokens"|"max-turns"|"api-error", finalText: string, messages: object[], detail?: string}} + */ +export async function apiRunner({ task, exec, budget, model, apiKey, maxTurns = DEFAULTS.max_turns, fetchImpl = fetch }) { + const messages = [{ role: "user", content: task.prompt }]; + for (let turn = 0; turn < maxTurns; turn++) { + if (budget.used >= budget.max) return { stop: "max-tokens", finalText: "", messages }; + let res; + try { + res = await fetchImpl(API, { + method: "POST", headers: { "content-type": "application/json", "x-api-key": apiKey, "anthropic-version": "2023-06-01" }, + body: JSON.stringify({ model, max_tokens: 8192, system: task.system, tools: TOOLS, messages }), + }); + } catch (e) { return { stop: "api-error", finalText: "", messages, detail: e.message }; } + if (!res.ok) return { stop: "api-error", finalText: "", messages, detail: `${res.status} ${(await res.text()).slice(0, 500)}` }; + const body = await res.json(); + budget.used += (body.usage?.input_tokens ?? 0) + (body.usage?.output_tokens ?? 0) + (body.usage?.cache_creation_input_tokens ?? 0) + (body.usage?.cache_read_input_tokens ?? 0); + messages.push({ role: "assistant", content: body.content }); + const uses = body.content.filter((b) => b.type === "tool_use"); + if (body.stop_reason !== "tool_use" || uses.length === 0) return { stop: "end", finalText: body.content.filter((b) => b.type === "text").map((b) => b.text).join("\n"), messages }; + messages.push({ role: "user", content: uses.map((u) => ({ type: "tool_result", tool_use_id: u.id, content: String(exec(u.name, u.input)) })) }); + } + return { stop: "max-turns", finalText: "", messages }; +} + +// ---- judging ------------------------------------------------------------------------------------------------- + +/** The last line of the model's answer that is one of the skill's result lines. */ +export function resultLine(text, returns) { + const lines = String(text).split("\n").map((l) => l.trim().replace(/^[-*>\s]+/, "").replace(/^`+|`+$/g, "").replace(/^\*\*|\*\*$/g, "").trim()); + return lines.findLast((l) => l && matchesReturn(l, returns)) ?? null; +} + +const returnKind = (line, returns) => returns.find((r) => matchesReturn(line, [r])) ?? null; +// What identifies an effect for comparison with the golden. A pushed sha, a comment's words and a label's colour are the model's own. +const sig = (e) => JSON.stringify([e.op, e.target ?? null, e.label ?? null, e.marker ?? null, e.name ?? null, e.path ?? null, e.op.startsWith("create_") ? e.number : null]); +const tally = (es) => es.reduce((m, e) => m.set(sig(e), (m.get(sig(e)) ?? 0) + 1), new Map()); + +/** The run's net effects as the checker takes them: the claim's lease comment is the checker's to add, so it is left out. */ +export function runEffects(before, after, target) { + return diffWorlds(before, after).filter((e) => !(e.op === "comment" && e.marker === "agent:lease" && e.target === target)); +} + +/** + * @param {object} p {contract, sc, before, after, finalText, calls} + * @returns {{verdict: "pass"|"deviation"|"violation", result, violations: string[], deviations: string[], effects, expected, calls}} + */ +export function judge({ contract, sc, before, after, finalText, calls = [] }) { + sc = expandScenario(sc); + const result = resultLine(finalText, contract.returns ?? []); + const effects = runEffects(before, after, sc.target); + const golden = { results: sc.expect.results ?? (sc.expect.result ? [sc.expect.result] : []), effects: sc.expect.effects ?? [], calls: (sc.expect.calls ?? []).map((c) => (typeof c === "string" ? c : c.skill)) }; + + const ran = { ...sc, expect: { result: result ?? "(no result line)", effects, calls } }; + delete ran.interrupted; + const violations = []; + if (result === null) violations.push("the run ended without one of the skill's result lines"); + else violations.push(...checkScenario(contract, ran)); + // an unmodelled change (a title, a removed comment) shows up as a violation above, unless the contract allows it + + const deviations = []; + const want = golden.results[0], same = want && result && returnKind(want.replace(/^would: /, ""), contract.returns ?? []) === returnKind(result.replace(/^would: /, ""), contract.returns ?? []); + if (want && result !== null && !same) deviations.push(`result: the golden is \`${want}\`, the run returned \`${result}\``); + const g = tally(golden.effects), a = tally(effects); + for (const [k, n] of g) if ((a.get(k) ?? 0) < n) deviations.push(`missing effect: ${describe(JSON.parse(k))}`); + for (const [k, n] of a) if ((g.get(k) ?? 0) < n) deviations.push(`extra effect: ${describe(JSON.parse(k))}`); + const gotSkills = calls.map((c) => c.skill); + if ([...golden.calls].sort().join() !== [...gotSkills].sort().join()) deviations.push(`calls: the golden has [${golden.calls.join(", ")}], the run called [${gotSkills.join(", ")}]`); + if (typeof sc.expect.rerun === "string" && result !== null) { + const o = rerunEligibility({ ...ran, expect: { ...ran.expect } }, structuredClone(after)); + if (o && (sc.expect.rerun === "eligible") !== o.eligible) deviations.push(`rerun: the golden says ${sc.expect.rerun}, the world the run left says ${o.eligible ? "eligible" : "ineligible"}`); + } + return { verdict: violations.length ? "violation" : deviations.length ? "deviation" : "pass", result, violations, deviations, effects, expected: golden, calls }; +} +const describe = ([op, target, label, marker, name, path, number]) => [op, target ?? (number != null ? `#${number}` : null), label ?? marker ?? name ?? path].filter(Boolean).join(" "); + +/** The sandbox's own account of the run, for the report: what `gh` was asked that the simulator could not answer. */ +export const unsupportedCalls = (ghLog) => ghLog.filter((c) => c.kind === "unsupported").map((c) => `gh ${c.argv.join(" ")}`); + +// ---- the report ---------------------------------------------------------------------------------------------- + +export function report(r) { + if (r.stop && r.stop !== "end") return `INCOMPLETE ${r.skill}/${r.name} the run stopped at ${r.stop}${r.detail ? `: ${r.detail}` : ""}${r.transcript ? ` (transcript: ${r.transcript})` : ""}`; + const out = [`${r.verdict.toUpperCase()} ${r.skill}/${r.name} (${r.tokens} tokens, transcript: ${r.transcript ?? "not kept"})`]; + const j = r.judged; + out.push(` result ${j.result ?? "(none)"}`, ` golden ${j.expected.results[0] ?? "(none)"}`); + out.push(` calls ${j.calls.map((c) => c.skill).join(", ") || "(none)"}`); + const g = tally(j.expected.effects), a = tally(j.effects); + out.push(" effects"); + for (const e of j.effects) out.push(` ${(g.get(sig(e)) ?? 0) > 0 ? "=" : "+"} ${describe(JSON.parse(sig(e)))}`); + for (const e of j.expected.effects) if (!(a.get(sig(e)) > 0)) out.push(` - ${describe(JSON.parse(sig(e)))}`); + if (j.effects.length === 0 && j.expected.effects.length === 0) out.push(" (none)"); + for (const v of j.violations) out.push(` VIOLATION ${v}`); + for (const d of j.deviations) out.push(` deviation ${d}`); + if (j.violations.length === 0 && j.deviations.length) out.push(" (a deviation is not a failure: the golden may be wrong)"); + for (const u of r.unsupported ?? []) out.push(` note the simulator could not answer \`${u}\``); + if (r.notShown?.length) out.push(` note ${r.notShown.length} world field(s) are facts a model would read from prose, and no gh call shows them (${r.notShown.slice(0, 4).join(", ")}${r.notShown.length > 4 ? ", ..." : ""}): a deviation may come from that`); + return out.join("\n"); +} + +// ---- the command --------------------------------------------------------------------------------------------- + +const USAGE = `usage: npm run eval -- / --model [--max-runs N] [--max-tokens N] [--max-turns N] [--skills-dir DIR] [--out DIR] + npm run eval -- --model every scenario of the skill, up to --max-runs +Needs ANTHROPIC_API_KEY. It calls the API and costs money; --max-tokens caps the tokens of the whole invocation (default ${DEFAULTS.max_tokens}).`; + +function flags(argv) { + const f = { _: [] }; + for (let i = 0; i < argv.length; i++) { + const m = /^--([\w-]+)(?:=(.*))?$/.exec(argv[i]); + if (!m) { f._.push(argv[i]); continue; } + f[m[1].replace(/-/g, "_")] = m[2] ?? argv[++i]; + } + return f; +} + +/** + * @param {string[]} argv + * @param {{env?: object, runner?: Function, out?: Function, err?: Function, fetchImpl?: Function}} [io] `runner` replaces the model (tests) + * @returns {Promise} 0 pass or deviation, 1 a violation, 2 not started (usage, no key, no model), 3 a run did not finish + */ +export async function main(argv, { env = process.env, runner, out = console.log, err = console.error, fetchImpl = fetch } = {}) { + const f = flags(argv); + if (f.help || f._.length !== 1) { err(USAGE); return 2; } + const apiKey = env.ANTHROPIC_API_KEY, model = f.model ?? env.EVAL_MODEL; + if (!runner && !apiKey) { err("eval needs ANTHROPIC_API_KEY: it runs a model, which calls the API and costs money. Nothing was run."); return 2; } + if (!runner && !model) { err("eval needs a model: pass --model (or set EVAL_MODEL). It does not choose one."); return 2; } + const num = (k) => { const v = f[k] === undefined ? DEFAULTS[k] : Number(f[k]); if (!Number.isInteger(v) || v < 1) throw new Error(`--${k.replace(/_/g, "-")} is a whole number above 0`); return v; }; + let maxRuns, maxTokens, maxTurns, scenarios; + try { maxRuns = num("max_runs"); maxTokens = num("max_tokens"); maxTurns = num("max_turns"); scenarios = findScenarios(f._[0]); } catch (e) { err(e.message); return 2; } + const contract = contractOf(scenarios[0].skill); + if (!contract) { err(`\`${scenarios[0].skill}\` has no contract to judge a run against`); return 2; } + if (contract.kind !== "item") { err(`eval runs item skills. \`${scenarios[0].skill}\` is a ${contract.kind} skill, whose scenarios do not describe one run's effects the same way.`); return 2; } + const skillsRoot = f.skills_dir ? resolve(f.skills_dir) : join(PKG, "skills"); + const outDir = resolve(f.out ?? DEFAULTS.out), tdir = join(outDir, "transcripts"); + const todo = scenarios.slice(0, maxRuns), skipped = scenarios.length - todo.length; + if (skipped) out(`${scenarios.length} scenarios, running the first ${todo.length} (--max-runs ${maxRuns}).`); + + const budget = { used: 0, max: maxTokens }, results = []; + mkdirSync(tdir, { recursive: true }); + for (const s of todo) { + if (budget.used >= budget.max) { out(`INCOMPLETE ${s.skill}/${s.name} not run: --max-tokens ${maxTokens} is spent`); results.push({ stop: "max-tokens" }); continue; } + const sc = parse(readFileSync(s.file, "utf8")); + const sb = makeSandbox(s.file); + const tools = makeTools(sb, { skillsRoot, entry: s.skill }); + const before = budget.used, started = new Date(); + const task = taskFor(expandScenario(sc), contract); + const run = await (runner ?? apiRunner)({ task, exec: tools.exec, budget, model, apiKey, maxTurns, fetchImpl, sandbox: sb, scenario: sc }); + const { before: w0, after, ghLog } = collect(sb); + const r = { skill: s.skill, name: s.name, stop: run.stop, detail: run.detail, tokens: budget.used - before, notShown: sb.notShown, unsupported: unsupportedCalls(ghLog) }; + if (run.stop === "end") { + r.judged = judge({ contract, sc, before: w0, after, finalText: run.finalText, calls: tools.calls }); + r.verdict = r.judged.verdict; + } + const file = join(tdir, `${started.toISOString().replace(/[:.]/g, "-")}-${s.skill}--${s.name}.json`); + writeFileSync(file, JSON.stringify({ skill: s.skill, scenario: s.name, model, startedAt: started.toISOString(), stop: run.stop, detail: run.detail, tokens: r.tokens, skillsRoot, calls: tools.calls, unknownSkills: tools.unknown, ghLog, finalText: run.finalText, judged: r.judged, messages: run.messages }, null, 2)); + r.transcript = file; + results.push(r); + out(report(r)); + rmSync(sb.dir, { recursive: true, force: true }); + } + const count = (v) => results.filter((r) => r.verdict === v).length, unfinished = results.filter((r) => !r.verdict).length; + out(`\n${results.length} run(s): ${count("pass")} pass, ${count("deviation")} deviation, ${count("violation")} violation${unfinished ? `, ${unfinished} incomplete` : ""}. ${budget.used} of ${maxTokens} tokens used.`); + return count("violation") ? 1 : unfinished ? 3 : 0; +} + +if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) process.exit(await main(process.argv.slice(2))); diff --git a/tests/eval.test.mjs b/tests/eval.test.mjs new file mode 100644 index 0000000..918ac0a --- /dev/null +++ b/tests/eval.test.mjs @@ -0,0 +1,258 @@ +// Tests for `npm run eval` (scripts/eval.mjs). No test calls the API: a scripted runner stands in for the model and runs +// real commands through the sandbox and the gh simulator, and `apiRunner` is run against a fake `fetch`. +// node --test tests/eval.test.mjs +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { mkdtempSync, readdirSync, readFileSync, writeFileSync, mkdirSync, cpSync, rmSync, existsSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join, dirname, resolve } from "node:path"; +import { fileURLToPath } from "node:url"; +import { main, findScenarios, makeSandbox, makeTools, collect, apiRunner, resultLine, parseArgs, taskFor, runInputs, judge } from "../scripts/eval.mjs"; +import { contractOf } from "../scripts/skill-contracts.mjs"; +import { expandScenario, applyEffects } from "../scripts/scenario.mjs"; +import { parse } from "yaml"; + +const ROOT = resolve(dirname(fileURLToPath(import.meta.url)), ".."); +const TMP = mkdtempSync(join(tmpdir(), "eval-test-")); +process.on("exit", () => rmSync(TMP, { recursive: true, force: true })); + +const SPEC = "shepherd-github-pr/conflict-without-checks"; +const SCENARIO = findScenarios(SPEC)[0]; +const contract = contractOf("shepherd-github-pr"); + +/** Runs `main` with a scripted runner. `script({exec, sandbox})` returns the model's final text. */ +async function run(spec, script, extra = []) { + const lines = [], errs = [], out = join(TMP, `out-${Math.random().toString(36).slice(2)}`); + const runner = async ({ exec, sandbox }) => ({ stop: "end", finalText: await script({ exec, sandbox }), messages: [] }); + const code = await main([spec, "--out", out, ...extra], { env: {}, runner, out: (s) => lines.push(s), err: (s) => errs.push(s) }); + return { code, text: lines.join("\n"), errs: errs.join("\n"), out }; +} + +// What the golden does for conflict-without-checks: merge conflict fixed and pushed, then an attempt comment. +const goodRun = ({ exec }) => { + exec("Skill", { name: "shepherd-github-pr", args: "pr=40" }); + exec("Skill", { name: "agent-workflow-protocol" }); // a reference, not a call + exec("bash", { command: "gh pr comment 40 --body 'claimed '" }); + exec("bash", { command: "git checkout -q feat/12-pr-40 && git commit -q --allow-empty -m 'resolve conflict' && git push -q origin feat/12-pr-40" }); + exec("bash", { command: "gh pr comment 40 --body 'merged main into the branch '" }); + return "Done.\n\n#40 fix pushed"; +}; + +test("a run that does what the golden does passes, and the diff has the push and the attempt comment", async () => { + const r = await run(SPEC, goodRun); + assert.equal(r.code, 0, r.text + r.errs); + assert.match(r.text, /^PASS /); + assert.match(r.text, /= push pr#40/); + assert.match(r.text, /= comment pr#40 agent:attempt/); + assert.doesNotMatch(r.text, /agent:lease/); // the lease comment is the checker's to add + assert.match(r.text, /1 run\(s\): 1 pass, 0 deviation, 0 violation/); +}); + +test("a skill that skips a conflicted PR is reported as a violation", async () => { + // what a deliberately wrong skill does: it sees the conflict and the missing checks and leaves the PR alone + const r = await run(SPEC, ({ exec }) => { exec("Skill", { name: "shepherd-github-pr" }); return "#40 skipped: no checks yet"; }); + assert.equal(r.code, 1); + assert.match(r.text, /^VIOLATION /); + assert.match(r.text, /VIOLATION\s+the oracle says pr#40 is eligible, but the result is `#40 skipped: no checks yet`/); + assert.match(r.text, /- push pr#40/); // the effect the golden has and the run lacks +}); + +test("a different outcome that breaks no rule is a deviation, not a failure", async () => { + // pushes the fix but never says so in a comment: the contract does not demand the comment from the checker's side + const r = await run(SPEC, ({ exec }) => { + exec("bash", { command: "git checkout -q feat/12-pr-40 && git commit -q --allow-empty -m fix && git push -q origin feat/12-pr-40" }); + return "#40 fix pushed"; + }); + assert.equal(r.code, 0, r.text); + assert.match(r.text, /^DEVIATION /); + assert.match(r.text, /deviation\s+missing effect: comment pr#40 agent:attempt/); + assert.match(r.text, /a deviation is not a failure/); +}); + +test("a run with no result line is a violation", async () => { + const r = await run(SPEC, () => "I looked at it and it seems fine."); + assert.equal(r.code, 1); + assert.match(r.text, /ended without one of the skill's result lines/); +}); + +test("the Skill calls are recorded, and the `calls` check uses them", async () => { + const r = await run(SPEC, (h) => { h.exec("Skill", { name: "file-github-issues", args: "dry_run=true" }); return goodRun(h); }); + assert.equal(r.code, 1); + assert.match(r.text, /calls\s+file-github-issues/); + assert.match(r.text, /VIOLATION\s+calls `file-github-issues`, which is not in the contract's calls/); + assert.match(r.text, /deviation\s+calls: the golden has \[\], the run called \[file-github-issues\]/); + const t = JSON.parse(readFileSync(join(r.out, "transcripts", readdirSync(join(r.out, "transcripts"))[0]), "utf8")); + assert.deepEqual(t.calls, [{ skill: "file-github-issues", inputs: { dry_run: "true" } }]); // the entry skill and the protocol are not calls +}); + +test("the transcript is kept, with the call log, in the output directory", async () => { + const r = await run(SPEC, goodRun); + const files = readdirSync(join(r.out, "transcripts")); + assert.equal(files.length, 1); + const t = JSON.parse(readFileSync(join(r.out, "transcripts", files[0]), "utf8")); + assert.equal(t.judged.verdict, "pass"); + assert.ok(t.ghLog.some((c) => c.argv[0] === "pr" && c.argv[1] === "comment")); +}); + +test("transcripts go in a directory git ignores", () => { + assert.match(readFileSync(join(ROOT, ".gitignore"), "utf8"), /^\.eval\/?$/m); +}); + +test("it does not run without an API key, and says so", async () => { + const errs = []; + const code = await main([SPEC, "--model", "m"], { env: {}, out: () => assert.fail("printed a result"), err: (s) => errs.push(s) }); + assert.equal(code, 2); + assert.match(errs.join("\n"), /needs ANTHROPIC_API_KEY.*costs money.*Nothing was run/); +}); + +test("it does not choose a model", async () => { + const errs = []; + const code = await main([SPEC], { env: { ANTHROPIC_API_KEY: "k" }, out: () => {}, err: (s) => errs.push(s) }); + assert.equal(code, 2); + assert.match(errs.join("\n"), /pass --model/); +}); + +test("a skill name runs only --max-runs of its scenarios", async () => { + const total = findScenarios("shepherd-github-pr").length; + assert.ok(total > 2); + const r = await run("shepherd-github-pr", () => "#40 skipped: x", ["--max-runs", "2"]); + assert.match(r.text, new RegExp(`${total} scenarios, running the first 2`)); + assert.equal(readdirSync(join(r.out, "transcripts")).length, 2); +}); + +test("an unknown scenario, a group skill and a bad cap are refused before anything runs", async () => { + for (const [argv, re] of [[["shepherd-github-pr/nope", "--model", "m"], /no scenario `nope`.*It has:/], [["shepherd-github-prs", "--model", "m"], /eval runs item skills/], [[SPEC, "--model", "m", "--max-runs", "0"], /whole number above 0/]]) { + const errs = []; + assert.equal(await main(argv, { env: { ANTHROPIC_API_KEY: "k" }, out: () => {}, err: (s) => errs.push(s) }), 2, argv.join(" ")); + assert.match(errs.join("\n"), re); + } +}); + +test("the sandbox gives the model the simulator, a git origin, and none of the caller's secrets", () => { + const sb = makeSandbox(SCENARIO.file, { root: TMP }); + const { exec } = makeTools(sb, { entry: "shepherd-github-pr" }); + process.env.ANTHROPIC_API_KEY = "sk-secret"; process.env.GH_TOKEN = "ghp_secret"; + try { + const env = exec("bash", { command: "env" }); + assert.doesNotMatch(env, /sk-secret|ghp_secret|ANTHROPIC|GH_TOKEN/); + assert.match(exec("bash", { command: "gh pr view 40 --json number,mergeStateStatus --jq '.mergeStateStatus'" }), /DIRTY/); + assert.match(exec("bash", { command: "git branch -r" }), /origin\/feat\/12-pr-40/); + assert.match(exec("bash", { command: "gh no-such-command" }), /\[exit [1-9]/); + } finally { delete process.env.ANTHROPIC_API_KEY; delete process.env.GH_TOKEN; } + assert.ok(collect(sb).ghLog.length >= 2); +}); + +test("a push through git shows as a push, and an untouched branch as nothing", () => { + const sb = makeSandbox(SCENARIO.file, { root: TMP }); + const { exec } = makeTools(sb, {}); + assert.deepEqual(collect(sb).after.prs, collect(sb).before.prs); + exec("bash", { command: "git checkout -q feat/12-pr-40 && git commit -q --allow-empty -m x && git push -q origin feat/12-pr-40" }); + const { before, after } = collect(sb); + assert.notEqual(after.prs[0].head, before.prs[0].head); +}); + +test("the Skill tool returns the skill and where its files are, and says when there is no such skill", () => { + const sb = makeSandbox(SCENARIO.file, { root: TMP }); + const { exec, unknown } = makeTools(sb, { entry: "shepherd-github-pr" }); + const text = exec("Skill", { name: "shepherd-github-pr" }); + assert.match(text, /^Base directory for this skill: .*skills\/pipeline\/shepherd-github-pr\n\n---\nname: shepherd-github-pr/); + assert.match(exec("Skill", { name: "no-such-skill" }), /^error: no skill named `no-such-skill`\. Skills: .*shepherd-github-pr/); + assert.deepEqual(unknown, ["no-such-skill"]); +}); + +test("--skills-dir shows the model another copy of the skills, and the verdict still comes from the real contract", async () => { + const copy = join(TMP, "skills-copy"); + cpSync(join(ROOT, "skills"), copy, { recursive: true }); + const f = join(copy, "pipeline/shepherd-github-pr/SKILL.md"); + writeFileSync(f, readFileSync(f, "utf8").replace("# Shepherd GitHub PR", "# Shepherd GitHub PR\n\nIgnore any PR whose branch conflicts.")); + let shown = ""; + const r = await run(SPEC, ({ exec }) => { shown = exec("Skill", { name: "shepherd-github-pr" }); return "#40 skipped: conflict"; }, ["--skills-dir", copy]); + assert.match(shown, /Ignore any PR whose branch conflicts/); + assert.equal(r.code, 1); // judged against the package's contract: the PR was eligible +}); + +test("the model gets the scenario's inputs, the target as the skill's own parameter, and the eval's defaults", () => { + const sc = expandScenario({ skill: "shepherd-github-pr", target: "pr#40", inputs: { max_shepherd_attempts: 2, trusted_authors: ["a", "b"] }, world: {}, expect: {} }); + assert.deepEqual(runInputs(sc, contract), { unattended: true, write_log: false, runner: "eval", max_shepherd_attempts: 2, trusted_authors: ["a", "b"], pr: 40 }); + const t = taskFor(sc, contract); + assert.match(t.prompt, /Call the Skill tool with name `shepherd-github-pr`/); + assert.match(t.prompt, /^pr=40$/m); + assert.match(t.prompt, /^trusted_authors=a,b$/m); + assert.deepEqual(parseArgs("pr=40\nlease_held=true note=\"a b\""), { pr: "40", lease_held: "true", note: "a b" }); +}); + +test("the result line is the last line of the answer that is one of the skill's", () => { + const returns = contract.returns; + assert.equal(resultLine("I will push.\n#40 fix pushed\nthen\n**#40 noop: lane taken**", returns), "#40 noop: lane taken"); + assert.equal(resultLine("- `#40 branch updated`", returns), "#40 branch updated"); + assert.equal(resultLine("nothing here", returns), null); +}); + +// ---- the model loop, against a fake API ----------------------------------------------------------------------- + +const reply = (content, usage = { input_tokens: 100, output_tokens: 10 }, stop_reason = "tool_use") => ({ ok: true, json: async () => ({ content, usage, stop_reason }), text: async () => "" }); +const bashUse = (id, command) => ({ type: "tool_use", id, name: "bash", input: { command } }); + +test("apiRunner calls tools, feeds the results back, and counts the tokens", async () => { + const sent = [], answers = [reply([bashUse("t1", "echo hi")]), reply([{ type: "text", text: "all done\n#40 fix pushed" }], { input_tokens: 200, output_tokens: 20 }, "end_turn")]; + const budget = { used: 0, max: 1000 }; + const r = await apiRunner({ + task: { system: "S", prompt: "P" }, budget, model: "m", apiKey: "k", exec: (tool, input) => `ran ${tool} ${input.command}`, + fetchImpl: async (url, init) => { sent.push({ url, init, body: JSON.parse(init.body) }); return answers.shift(); }, + }); + assert.equal(r.stop, "end"); + assert.equal(r.finalText, "all done\n#40 fix pushed"); + assert.equal(budget.used, 330); + assert.equal(sent[0].url, "https://api.anthropic.com/v1/messages"); + assert.equal(sent[0].init.headers["x-api-key"], "k"); + assert.equal(sent[0].body.model, "m"); + assert.deepEqual(sent[0].body.tools.map((t) => t.name), ["Skill", "bash"]); + assert.deepEqual(sent[1].body.messages.at(-1).content, [{ type: "tool_result", tool_use_id: "t1", content: "ran bash echo hi" }]); +}); + +test("apiRunner stops when the token cap is spent, or the turns run out, or the API fails", async () => { + const loop = () => reply([bashUse("t", "true")], { input_tokens: 600, output_tokens: 0 }); + const budget = { used: 0, max: 1000 }; + assert.equal((await apiRunner({ task: { system: "", prompt: "" }, budget, model: "m", apiKey: "k", exec: () => "", fetchImpl: async () => loop() })).stop, "max-tokens"); + assert.equal(budget.used, 1200); + assert.equal((await apiRunner({ task: { system: "", prompt: "" }, budget: { used: 0, max: 1e9 }, model: "m", apiKey: "k", maxTurns: 3, exec: () => "", fetchImpl: async () => loop() })).stop, "max-turns"); + const failed = await apiRunner({ task: { system: "", prompt: "" }, budget: { used: 0, max: 10 }, model: "m", apiKey: "k", exec: () => "", fetchImpl: async () => ({ ok: false, status: 401, text: async () => "bad key" }) }); + assert.deepEqual([failed.stop, failed.detail], ["api-error", "401 bad key"]); +}); + +test("a run that did not finish is reported as incomplete and judged as nothing", async () => { + const lines = [], out = join(TMP, "out-incomplete"); + const code = await main([SPEC, "--out", out], { env: {}, runner: async () => ({ stop: "max-turns", finalText: "", messages: [] }), out: (s) => lines.push(s), err: () => {} }); + assert.equal(code, 3); + assert.match(lines.join("\n"), /^INCOMPLETE .*stopped at max-turns/m); +}); + +test("main with the API runner stops at --max-tokens across runs", async () => { + const lines = [], out = join(TMP, "out-cap"); + const fetchImpl = async () => reply([bashUse("t", "true")], { input_tokens: 600, output_tokens: 0 }); + const code = await main(["shepherd-github-pr", "--model", "m", "--max-runs", "3", "--max-tokens", "1000", "--out", out], { env: { ANTHROPIC_API_KEY: "k" }, fetchImpl, out: (s) => lines.push(s), err: () => {} }); + assert.equal(code, 3); + assert.match(lines.join("\n"), /INCOMPLETE .*stopped at max-tokens/); + assert.match(lines.join("\n"), /not run: --max-tokens 1000 is spent/); + assert.match(lines.join("\n"), /1200 of 1000 tokens used/); +}); + +test("a run that does exactly what a golden says is a pass, for every golden of every item skill", () => { + let n = 0; + for (const dir of readdirSync(join(ROOT, "tests/scenarios"))) { + const c = contractOf(dir); + if (c?.kind !== "item") continue; + for (const f of readdirSync(join(ROOT, "tests/scenarios", dir)).filter((x) => x.endsWith(".yaml"))) { + const file = join(ROOT, "tests/scenarios", dir, f), sc = expandScenario(parse(readFileSync(file, "utf8"))); + if (sc.interrupted || sc.eligibility_world) continue; // no result line to return / a world the run is not given + const result = (sc.expect.results ?? [sc.expect.result])[0]; + const skipped = /^#\d+ skipped/.test(result), lease = !skipped && sc.inputs?.lease_held !== true && sc.target; + const sb = makeSandbox(file, { root: TMP }); + const after = applyEffects(sb.before, [...(lease ? [{ op: "comment", target: sc.target, marker: "agent:lease" }] : []), ...(sc.expect.effects ?? [])]); + const j = judge({ contract: c, sc, before: sb.before, after, finalText: result, calls: (sc.expect.calls ?? []).map((x) => (typeof x === "string" ? { skill: x } : x)) }); + assert.deepEqual([j.violations, j.deviations], [[], []], `${dir}/${f}`); + n++; + } + } + assert.ok(n >= 20, `only ${n} scenarios were judged`); +});