Skip to content

test: the gh simulator's api, faults, and one mock for lib.test.sh (#88) - #95

Merged
glennfaison merged 3 commits into
mainfrom
feat/gh-simulator-api-faults
Nov 11, 2026
Merged

glennfaison merged 3 commits into
mainfrom
feat/gh-simulator-api-faults

Conversation

@glennfaison

Copy link
Copy Markdown
Owner

Closes #88. Completes #60.

What

The last part of the gh simulator.

  • gh api answers the nine REST calls the package makes: user, an item's comments, PATCH of a comment, a PR's files, contents (raw or base64), the tree, the PR list (paged, newest first) and one PR, and update-branch. The last three are new since Run a skill against a simulated GitHub instead of the real one #60 was written: fix(protocol): one lease comment per step, edited in place (#69) #91 added the PATCH, fix(protocol): prs_for_issue and issue_for_pr work on a real gh (#89) #93 the REST pulls. Anything else (graphql, another method on a path, -F @file) is unsupported.
  • Faults. A world, and so a scenario's world:, may carry faults: [{match, fail, exit?, before?, times?}]. A matching call fails with fail, writes nothing, and is logged as kind: "fault". before is a list of effects applied first: with fail it is a failure after something changed, without it the call goes on, which is how another run gets in ahead of this one.
  • One mock. tests/lib.test.sh now runs against the simulator and the old stub is gone. A closing test fails if any call in the run was unsupported.
  • A change to the checker. A new comment_edit effect (the comment's index and its new text), scoped like comment. Since fix(protocol): one lease comment per step, edited in place (#69) #91 a step's lease comment is edited in place, and diffWorlds reported that as unmodelled, so a run using the new claim could never be read back.
  • planned is no longer a status in the manifest: every command is supported or deliberately out of scope, and a test asserts it. ADR 0012 and 0014, AGENTS.md and a changeset are updated.

#60's acceptance criteria

  • Every gh command the skills use is supported, an unsupported one fails loudly, and a test greps the skills (test: a gh simulator with the read paths (#86) #90).
  • tests/lib.test.sh runs against the shared simulator and passes: 179 tests, the 154 the issue named plus those added since.
  • A fault can be injected and is visible in the call log.
  • The world file is the scenarios' format. Every golden scenario loads, reads, and has its effects replayed as gh calls (test: the gh simulator writes (#87) #92). A golden scenario whose world carries faults is judged as it was (tested on all of them).

What the conversion found

Making the shell tests go through something as strict as gh showed three things the stub let through:

  1. Faults counted by position. My first version counted times by the fault's index in the list, over a log that lives for the whole run, so an earlier, different fault at index 0 used up the budget of a later one. A fault is now identified by its definition (a hash, recorded as faultId), and a test covers it.
  2. A repository with a label catalog refuses labels it does not have. log_issue runs gh label create agent:log, which gave the test world a catalog of one label, and the next claim failed with 'agent:triage' not found. Real gh does the same. The tests now start from the catalog setup-github-agent-workflow leaves.
  3. Issues and PRs share one number sequence. A log issue made after PRs 70 to 74 is number 75, not 51. The tests reset both.

(#89 was the fourth, and is already fixed.)

Choices worth a look

  • A comment the simulator derives from a world field cannot be edited. A seeded lease: live stands for a comment, but there is no stored comment to change, so PATCH of it is unsupported (visible in the log) and claim falls back to posting a comment, as it does when an edit fails.
  • update-branch makes a BEHIND PR CLEAN and gives it a new head (a push effect). Nothing else about merge state changes, so the diff stays free of unmodelled.
  • A PR's association (default NONE) is what REST shows as author_association. A comment can carry an association too. A rename in a PR's files is {path, previous}.

The cost

Each simulated call starts Node: about 75 ms, half of it loading yaml through the checker. tests/lib.test.sh makes about 390 calls and takes about 45 seconds where the stub took 10, so npm test is slower. I tried Node's compile cache and it saved only 12 ms a call, so I left it out. If this matters, the next step is a long-running simulator process rather than a faster start. The ADR says so.

Verified

glennfaison and others added 3 commits November 10, 2026 19:44
Part of #60. gh api answers the nine REST calls lib.sh and the skills make
(user, an item's comments, PATCH of a comment, a PR's files, contents, the
tree, the PR list, one PR, update-branch). A world's faults make a matching
call fail without changing anything, or apply effects first, and are marked
in the call log. The checker gets a comment_edit effect, so a lease edited in
place is read back by diffWorlds.

Co-authored-by: Claude <noreply@anthropic.com>
Part of #60. The stub gh in tests/lib.test.sh is gone: the tests set a world file
up with a small w helper and read the result back through gh. The simulator
counts a fault by its definition, a scenario whose world carries faults is
judged as it was, and the last test fails if any call in the run was one the
simulator does not cover.

Co-authored-by: Claude <noreply@anthropic.com>
@glennfaison
glennfaison merged commit 6da1eff into main Nov 11, 2026
2 checks passed
@glennfaison
glennfaison deleted the feat/gh-simulator-api-faults branch November 11, 2026 01:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

gh simulator (3/3): api, fault injection, and one mock for lib.test.sh

1 participant