Repository navigation
test: the gh simulator's api, faults, and one mock for lib.test.sh (#88) - #95
Merged
Merged
Conversation
Part of #60. gh api answers the nine REST calls lib.sh and the skills make (user, an item's comments, PATCH of a comment, a PR's files, contents, the tree, the PR list, one PR, update-branch). A world's faults make a matching call fail without changing anything, or apply effects first, and are marked in the call log. The checker gets a comment_edit effect, so a lease edited in place is read back by diffWorlds. Co-authored-by: Claude <noreply@anthropic.com>
Part of #60. The stub gh in tests/lib.test.sh is gone: the tests set a world file up with a small w helper and read the result back through gh. The simulator counts a fault by its definition, a scenario whose world carries faults is judged as it was, and the last test fails if any call in the run was one the simulator does not cover. Co-authored-by: Claude <noreply@anthropic.com>
…88) Co-authored-by: Claude <noreply@anthropic.com>
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #88. Completes #60.
What
The last part of the
ghsimulator.gh apianswers the nine REST calls the package makes:user, an item's comments,PATCHof a comment, a PR's files, contents (raw or base64), the tree, the PR list (paged, newest first) and one PR, andupdate-branch. The last three are new since Run a skill against a simulated GitHub instead of the real one #60 was written: fix(protocol): one lease comment per step, edited in place (#69) #91 added thePATCH, fix(protocol): prs_for_issue and issue_for_pr work on a real gh (#89) #93 the REST pulls. Anything else (graphql, another method on a path,-F @file) isunsupported.world:, may carryfaults: [{match, fail, exit?, before?, times?}]. A matching call fails withfail, writes nothing, and is logged askind: "fault".beforeis a list of effects applied first: withfailit is a failure after something changed, without it the call goes on, which is how another run gets in ahead of this one.tests/lib.test.shnow runs against the simulator and the old stub is gone. A closing test fails if any call in the run wasunsupported.comment_editeffect (the comment's index and its new text), scoped likecomment. Since fix(protocol): one lease comment per step, edited in place (#69) #91 a step's lease comment is edited in place, anddiffWorldsreported that asunmodelled, so a run using the newclaimcould never be read back.plannedis no longer a status in the manifest: every command is supported or deliberately out of scope, and a test asserts it. ADR 0012 and 0014,AGENTS.mdand a changeset are updated.#60's acceptance criteria
ghcommand the skills use is supported, an unsupported one fails loudly, and a test greps the skills (test: a gh simulator with the read paths (#86) #90).tests/lib.test.shruns against the shared simulator and passes: 179 tests, the 154 the issue named plus those added since.ghcalls (test: the gh simulator writes (#87) #92). A golden scenario whose world carries faults is judged as it was (tested on all of them).What the conversion found
Making the shell tests go through something as strict as
ghshowed three things the stub let through:timesby the fault's index in the list, over a log that lives for the whole run, so an earlier, different fault at index 0 used up the budget of a later one. A fault is now identified by its definition (a hash, recorded asfaultId), and a test covers it.log_issuerunsgh label create agent:log, which gave the test world a catalog of one label, and the nextclaimfailed with'agent:triage' not found. Realghdoes the same. The tests now start from the catalogsetup-github-agent-workflowleaves.(#89 was the fourth, and is already fixed.)
Choices worth a look
lease: livestands for a comment, but there is no stored comment to change, soPATCHof it isunsupported(visible in the log) andclaimfalls back to posting a comment, as it does when an edit fails.update-branchmakes aBEHINDPRCLEANand gives it a new head (apusheffect). Nothing else about merge state changes, so the diff stays free ofunmodelled.association(defaultNONE) is what REST shows asauthor_association. A comment can carry anassociationtoo. A rename in a PR'sfilesis{path, previous}.The cost
Each simulated call starts Node: about 75 ms, half of it loading
yamlthrough the checker.tests/lib.test.shmakes about 390 calls and takes about 45 seconds where the stub took 10, sonpm testis slower. I tried Node's compile cache and it saved only 12 ms a call, so I left it out. If this matters, the next step is a long-running simulator process rather than a faster start. The ADR says so.Verified
npm run checkpasses: 179 shell tests, 457 node tests (435 before this part).EPIPEflake taught me not to trust one).lib.sh, shown in fix(protocol): prs_for_issue and issue_for_pr work on a real gh (#89) #93.