Skill eval harness + eval-validated Common Tasks section (issue #207)

[schpetbot]

Related to #207.

Builds an eval that empirically measures the issue's claim — that an agent reading the skill skips dedicated CLI subcommands and reaches for raw GraphQL via linear api — then uses it to baseline the current skill and validate a "Common Tasks" recipes section.

Headline result

Metric (36 trials/condition, codex gpt-5.6-sol, low effort)	Baseline	Post-change
Supported tasks: full success (route + all required flags)	30/30	29/30
Holdout prompts only	15/15	15/15
Controls (GraphQL genuinely required): chose linear api	6/6	6/6

The premise of #207 did not reproduce in this setup. With the skill actually read (verified per-trial via the codex event stream), routing was already perfect before any change. Per the outcome rules declared before the baseline ran (evals/linear-cli-skill/README.md), that is reported as the finding rather than manufacturing harder prompts after the fact. The Common Tasks section is therefore shipped as validated non-regressing — in particular, the new recipes did not overcorrect the controls away from legitimate linear api use — not as a measured improvement.

The one post-change failure was a subject first trying the skill's documented npx alternative (npx --yes @schpet/linear-cli issue create …); the frozen grader counts any npx use as a bypass. It was not a GraphQL fallback, and the trial then completed correctly via linear issue create. There were zero linear api fallbacks on supported tasks in either final condition.

What's in the PR

Commit 1 — eval harness + baseline (evals/linear-cli-skill/)

• run.ts spawns codex exec per trial in full isolation: fresh CODEX_HOME (copied auth + skill under test only) and a fake HOME — codex also discovers skills in ~/.agents/skills, a real leak we verified — plus recording shims for linear/curl/npx/npm on an explicit PATH, under codex's workspace-write sandbox (network denied to subject commands).
• The linear shim logs every invocation (+ GraphQL heredoc stdin), passes --help/--version/schema through to the real CLI in-repo, and returns consistency-aware canned successes.
• cases.ts: 5 recipe families × (development + holdout prompt) + 2 controls where linear api is the right answer (issue history, subscribers). Frozen before baseline.
• grade.ts: deterministic classifier (no LLM judging) + one-sided Fisher exact; 15 unit tests in test/evals/.
• Committed results are sanitized (no secrets, no absolute paths); deno task skill-eval to run (costs real tokens; never in CI).

Commit 2 — the skill change + post-change results

• ## Common Tasks near the top of SKILL.template.md (regenerated into SKILL.md): filtered queries via issue query with the issue list≡issue mine alias gotcha spelled out, create with --description-file/--no-interactive, update state/assignee/labels (label replacement semantics noted), comment from file, view/URL.
• results/comparison.md with the full per-case tables.

Methodology notes (honesty section)

• Two earlier baseline runs were voided during harness development: stateless canned outputs contradicted the subject's own actions (issue view not reflecting a just-made update; ENG-* identifiers returned for OPS-team requests), which baited subjects into GraphQL "investigation" and contaminated the route signal. The shim now reflects same-trial updates and honors query filters. Both voided runs failed in the conservative direction (they understated the baseline).
• Prompts include "Use the installed linear-cli skill", so the eval measures the effect of the skill's content, not skill auto-triggering. Skill: add a minimal "Common tasks" section to SKILL.md #207's experience may live in configurations this doesn't cover: agents that never read the skill, other models/effort levels, or tasks further from the command vocabulary. The harness makes those follow-up experiments cheap.
• Cases, grader, and thresholds were frozen before the baseline; nothing was tuned after seeing results.

Reproduce:

deno task skill-eval --condition baseline --skill-dir skills/linear-cli --model gpt-5.6-sol
deno run --allow-read --allow-write evals/linear-cli-skill/grade.ts --compare \
  evals/linear-cli-skill/results/baseline.jsonl evals/linear-cli-skill/results/post-change.jsonl