Repository navigation
Run a skill against a simulated GitHub instead of the real one #60
Description
Category: feature
Area: Evaluation (tests/)
Impact: Without it nothing can run a skill safely and repeatably, so the scenarios stay a spec that no real run is checked against.
Confidence: traced
Problem (the use case)
A maintainer who wants to see whether a model follows shepherd-github-pr has to point it at a real repository and clean up afterwards. They cannot repeat a run, inject a failure (a merge GitHub rejects, a check that never finishes) or compare two runs.
Proposed solution
Add a gh simulator: an executable named gh, placed first on PATH, that reads and writes a world file (the same format the scenarios use) and appends every call to a log.
Support the subcommands the skills and lib.sh actually use: issue (list, view, edit, close, create, comment, reopen), pr (view, list, edit, create, comment, checks, merge), label (create, list), api (the update-branch PUT, user, and the few --jq reads), run view, repo view, auth status, with --json, --jq, --label, --state and --limit where they are used. Anything else exits non-zero with unsupported: <command> so a new gh call in a skill is noticed.
- Seed it from a scenario's
world. After a run,diffWorlds(the blocking ticket) gives the effects. - Fault injection: a scenario may say
faults: [{match: "pr merge", fail: "not mergeable"}]. - Unify with the
ghmock thattests/lib.test.shalready has. There should be one mock, not two.
Acceptance criteria
- Every
ghcommand the skills use is supported, and an unsupported one fails loudly. A test greps the skills forghcalls and checks each is supported or explicitly listed as out of scope. -
tests/lib.test.shruns against the shared simulator, and still passes (154 tests). - A fault can be injected and is visible in the call log.
- The world file is the scenarios' format, so any scenario can seed a run.
Non-goals
Matching GitHub's behaviour beyond what the skills depend on (rate limits, pagination, webhooks).
Blocked by: #59
Size: large. Suggested split: (1) the issue and PR read paths, (2) the write paths, (3) api, faults and unification with lib.test.sh.
Activity
glennfaison commentedon Nov 9, 2026
Split as the issue suggests: #86 (world file, call log, read paths), #87 (write paths), #88 (
api, faults, one mock forlib.test.sh). Each blocks the next. Working through them in order, one PR each.test: a gh simulator with the read paths (#86) (#90)
test: the gh simulator writes (#87)
test: the gh simulator writes (#87) (#92)
test: the gh simulator's api, faults, and one mock for lib.test.sh (#88…
glennfaison commentedon Nov 11, 2026
Done. The simulator is in
tests/simandtests/lib.test.shruns against it, so there is one mock ofgh.api, fault injection and the unificationprs_for_issueandissue_for_praskghforauthorAssociation, which is not a PR field #89 / fix(protocol): prs_for_issue and issue_for_pr work on a real gh (#89) #93 (authorAssociationis not a PR field, soprs_for_issueandissue_for_prfailed on a realgh)Not done, and not part of this issue: a runner that drives a model against the simulator (#61), and the shepherd skill's check-state one-liner, which expects
[]fromgh pr checksfor a PR with no checks whereghexits 1 (noted in #90, no issue filed). Seedocs/adr/0014-gh-simulator.md.