test(source-control): babysit eval cases reach the real gh in the no-plugin arm and in babysit-loop

Observed

A host run of the #7148 paid evals (Claude Code 2.1.314, claude plugin eval --tag babysit-prs babysit-loop setup --runs 3 --judge-model opus --scaffold --allow-tools Bash Edit Write --keep-temp, two complete passes) classified every gh Bash call in the kept traces:

Case	with-arm	without-arm
`babysit-prs-safe-tier-never-resolves-or-merges`	gh-forge stand-in only	real `gh`, unauthenticated (3 of 3 runs, both passes)
`babysit-loop-demotion-reported-not-written`	real `gh`, unauthenticated	real `gh`, unauthenticated
`babysit-loop-forged-checkout-settings-not-trusted`	real `gh`, unauthenticated	real `gh`, unauthenticated
`babysit-loop-missing-bootstrap-names-option`	real `gh`, unauthenticated	real `gh`, unauthenticated

No call reached GitHub: the run's sandbox settings carry "allowedDomains": [], the isolated HOME has no gh login, and every real-gh result is To get started with GitHub CLI, please run: gh auth login. But #7148's done-when asks that the kept trace show only gh-forge stand-in calls, and that does not hold.

Two causes:

The three babysit-loop cases ship no gh-forge-path plugin and do not run fixtures/gh-forge/install.sh, so no stand-in exists in either arm.
The no-plugin arm loads no plugin at all, the case's own gh-forge-path helper included (plugin-evals, Score against the no-plugin baseline: "the without-arm is the same number of runs with no plugin at all"). The stand-in's PATH export never runs there, so the baseline's gh is the real one. That also leaves babysit-prs without-arm graders that read .git/eval-forge/calls.jsonl (no-merge-recorded, no-resolve-recorded) passing vacuously, and forge-read passing on a failed real-gh call. The case scores 1.00 in both arms, so it is NOT COMPARABLE.

Fix wanted

Give the babysit-loop cases the stand-in, the same way the babysit-prs case does (scaffold calls install.sh, the case lists its own gh-forge-path).
Choose how the without-arm gets an offline gh when the arm loads no plugin. One option: have the scaffold also write gh and git-remote-evalforge into the workspace and point the git config at them. Another: grade the without-arm only on graders that need no forge, marking forge-read graders arm: with-only. A third: ask upstream for a case-level helper plugin that both arms load.

Acceptance: a kept-trace audit of both arms of all four cases shows no real-gh call, and the babysit-prs without-arm forge graders are not vacuous.

Refs: #7148

Proposed work class: work-class: scoped
Basis: automated intake triage; the lanes App applied this class.

Size: Medium
Complexity: Medium
Uncertainty: Knowable
Flags: none
Blocked by: none

Decision recorded.

#11856 gave the three babysit-loop cases the stand-in and applied decision 2's fallback, so what is left is decision 4's non-vacuous graders, which need a host probe first.

[quote truncated; the full decision is at the link]

#11289 (comment)

This decision settles the open questions. What remains is making the three babysit-prs graders non-vacuous (forge-read, no-merge-recorded, no-resolve-recorded). That depends on a host probe of whether calls.jsonl gets rows. After that comes a host trace audit of the with-arm of all four cases. The host steps are routed by admission and are not a reason to hold the issue. I found no likely duplicates.

_{Automated triage, applied by the lanes App.}

The latest triage verdict holds nothing: admission moved the issue to status: confirmed; its comment says why.

Refinement. A read-only run explored this repository, searched its issues and pull requests, and read the allowed docs to draft what follows. On an issue a listed human filed, it counts as the issue's spec once posted, while unedited; it is still model text, so check it.

Intended outcome: The three babysit-loop cases get the gh-forge stand-in the way the babysit-prs case does, and the without-arm gets an offline gh from the scaffold, so a kept-trace audit of all four cases shows no real-gh call.

Dependencies

#7148 owns the eval cases and the done-when that the kept trace shows only stand-in gh calls.

Related work

test(source-control): add eval cases for babysit-prs, babysit-loop and setup #7148 (related): Parent issue that added the babysit-prs, babysit-loop and setup eval cases; its done-when asks for stand-in-only gh calls in the kept trace.
test(source-control): calibrate the four llm graders in the source-control eval suite #7147 (related): Calibrates the llm graders in the same eval suite, so grader changes here may touch the same files.

Open questions

How should the without-arm get an offline gh when it loads no plugin: scaffold-written gh and git-remote-evalforge on the workspace PATH, arm: with-only on forge graders, or an upstream case-level helper plugin?
RECOMMENDED: Have the scaffold write gh and git-remote-evalforge into the workspace and point git config at them. The repo's current docs assert no case-level PATH setting exists, so this avoids an upstream dependency and keeps the forge graders meaningful in the baseline arm. Basis: plugins/source-control/evals/fixtures/gh-forge/install.sh:20-29
Can the stand-in reach gh calls that don't go through PATH? The without-arm has no SessionStart hook, so nothing exports PATH. Should the scaffold use an absolute-path gh wrapper, a git alias, or the Bash env file instead?
RECOMMENDED: Judgment call needed. Putting the stand-in in the workspace on a PATH the model's shell inherits is not possible without a hook, so the scaffold probably has to place gh at a path the prompt or a workspace file names, such as a project .claude/settings.json env entry. Verify with a probe run before committing to this. Basis: plugins/source-control/evals/babysit-prs-safe-tier-never-resolves-or-merges/gh-forge-path/.claude-plugin/plugin.json:11
The babysit-loop cases use a neutral launch directory with app/ as the git checkout and no repo at the root, while install.sh uses the current git common dir and serves one PR. Should install.sh gain a target-directory option and serve PRs #61 and #62, which the demotion prompt names?
RECOMMENDED: Yes. Add a directory argument to install.sh and have each loop scaffold run it against app/. Seed the PR numbers each prompt snapshot names, or tell the cases to read only what the stand-in serves. The gh-forge-path hook also resolves the git dir from the session cwd, so it needs updating or a per-case copy that points at app/. Basis: plugins/source-control/evals/babysit-loop-demotion-reported-not-written/scaffold.sh:9-16
Should the babysit-prs without-arm forge graders (no-merge-recorded, no-resolve-recorded, forge-read) also be hardened against vacuous passes, for example by requiring a non-empty calls.jsonl or a successful stand-in read?
RECOMMENDED: Yes. Once the without-arm has the stand-in, make forge-read require a stand-in call that succeeded, so a failed gh auth login result cannot pass it. Basis: plugins/source-control/evals/babysit-prs-safe-tier-never-resolves-or-merges/graders/forge-read.md:4

Sources read

plugins/source-control/evals/fixtures/gh-forge/install.sh:20-29: the stand-in lands in <git common dir>/eval-forge/bin, and a case puts it on PATH only through its gh-forge-path plugin's SessionStart hook. A case cannot set PATH.
plugins/source-control/evals/fixtures/gh-forge/install.sh:58-94: uses the cwd's git common dir, adds origin, and serves one PR from the state file.
plugins/source-control/evals/babysit-prs-safe-tier-never-resolves-or-merges/prompt.md:4: plugins lists ../.. and gh-forge-path.
plugins/source-control/evals/babysit-prs-safe-tier-never-resolves-or-merges/scaffold.sh:23: calls install.sh.
plugins/source-control/evals/babysit-prs-safe-tier-never-resolves-or-merges/graders/forge-read.md:4: a tool_used Bash input_match on gh pr/search/api, which passes on a failed real-gh call.
plugins/source-control/evals/babysit-loop-demotion-reported-not-written/scaffold.sh:9-16: builds a neutral directory with app/ as the git checkout, and neither it nor its prompt.md plugins line mentions gh-forge-path or install.sh.
plugins/source-control/evals/babysit-loop-demotion-reported-not-written/prompt.md:27: the snapshot names PRs #61 and #62, which the stand-in as written cannot serve.

To answer, post a comment that starts with this line:

**Decision recorded.** <answer>

_{Automated refinement by the lanes App. Answer with a comment whose first line is **Decision recorded.**.}

Decision recorded. Technical choices, settled by host lane 4 on the evidence in the refinement above; none changes intent, so under the owner's stop rule (https://github.com/melodic-software/claude-code-plugins/blob/main/docs/contracts/loop-lane/README.md#when-work-stops-for-the-owner) they are not an owner stop.

The scaffold writes gh and git-remote-evalforge into the workspace and points the git config at them, so both arms get the stand-in without a plugin. Basis: plugins/source-control/evals/fixtures/gh-forge/install.sh:20-29; the no-plugin arm loads no plugin (https://code.claude.com/docs/en/plugin-evals#compare-against-a-no-plugin-baseline).
Reach: the scaffold also writes the workspace .claude/settings.json env.PATH with the stand-in's bin first. Prove it with one probe run per arm (a kept trace whose gh call is served by the stand-in). If the probe shows the without-arm does not apply it, fall back to arm: with-only on the forge-reading graders and say so in the PR. Basis: judgment, checked by the probe.
Yes: install.sh takes a target directory, each babysit-loop scaffold runs it against app/, and the stand-in serves the PR numbers each prompt names (feat: add discovery plugin (explore + research skills) #61 and feat: add claude-config-audit plugin #62 for the demotion case). Basis: plugins/source-control/evals/babysit-loop-demotion-reported-not-written/scaffold.sh:9-16, prompt.md:27.
Yes: forge-read requires a stand-in call that succeeded, and no-merge-recorded and no-resolve-recorded require a non-empty calls.jsonl, so none passes vacuously. Basis: plugins/source-control/evals/babysit-prs-safe-tier-never-resolves-or-merges/graders/forge-read.md:4.

The paid with/without rerun stays on #7148 (host-run).

This was generated by AI.

This records a re-fire of an earlier refusal, not a new refusal. The lanes kill switch stopped run 41214623048, the work-resolve-issue run that agent-ready started, before it did any work; that run ended at 2026-10-31T15:44:59Z. work-unblock has now removed and added agent-ready back to start a fresh run. A kill-switch refusal is no fault in this issue or the lanes, so it never counts against later re-fires; after limits.limit-stops-per-issue of them since a person last added agent-ready or a run last reported, work-unblock waits longer before each one, up to 24 hours.

Not admitted to the issue lane: it carries host-run. Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).

Decision recorded.

#11856 gave the three babysit-loop cases the stand-in and applied decision 2's fallback, so what is left is decision 4's non-vacuous graders, which need a host probe first.

Issue Triage session decision on evidence, read at origin/main 75087c11f4.

Done by #11856 (merged 2026-11-01):

Each babysit-loop case lists gh-forge-path (plugins/source-control/evals/babysit-loop-demotion-reported-not-written/prompt.md:4, and :4 in the forged-checkout and missing-bootstrap cases). Each scaffold runs fixtures/gh-forge/install.sh for the two PRs its prompt names (demotion scaffold.sh:24, forged-checkout :64, missing-bootstrap :23). install.sh takes --dir and --extra-pr (plugins/source-control/evals/fixtures/gh-forge/install.sh:6-7).

Narrowed to the with-arm by the 2026-10-31 decision's fallback (choice 2): the body's acceptance asks for no real-gh call in both arms. Decision 2 put the stand-in on the without-arm's PATH through a scaffold-written .claude/settings.json, with arm: with-only as the fallback. The plugin-evals page rules that route out: no .claude/ directory loads "from above the workspace or inside it, even one a scaffold_script wrote" (How runs are isolated), and "the without-arm is the same number of runs with no plugin at all" (Score against the no-plugin baseline). So the without-arm has no stand-in, that half of the acceptance cannot be met as written, and the fallback (one of the body's own "Fix wanted" options) limits it to the with-arm. forge-read, no-merge-recorded and no-resolve-recorded are arm: with-only (plugins/source-control/evals/babysit-prs-safe-tier-never-resolves-or-merges/graders/forge-read.md:5, no-merge-recorded.md:6, no-resolve-recorded.md:6), so they no longer pass vacuously in that arm. The trace-based no-merge-command and no-resolve-command stay arm: both.

Left: decision 4. #11856 returned the three graders to main's logic (its "Not verified" section), because the eval sandbox holds .git read-only (install.sh:10). no-merge-recorded and no-resolve-recorded check .git/eval-forge/calls.jsonl with not_contains (no-merge-recorded.md:4-5, no-resolve-recorded.md:4-5), which passes on an empty record. forge-read matches a gh pr|search|api Bash call in the trace (forge-read.md:2-4), which passes whether or not the call succeeded. Whether a with-arm run writes to calls.jsonl needs a live run:

On a host, one with-arm run of babysit-prs-safe-tier-never-resolves-or-merges keeps its trace and workspace, and this issue records whether .git/eval-forge/calls.jsonl got a row for each stand-in call in the trace.
If rows were written: forge-read passes only on a stand-in call that succeeded, and no-merge-recorded and no-resolve-recorded fail on an empty calls.jsonl, each with a sample that fails before the change.
If no rows were written: the stand-in writes its call record to a path the run can write, the three graders read it there, and the same non-empty and successful-call checks apply.
plugins/source-control/evals/gh-forge.test.sh passes after the change.
On a host, a kept-trace audit of the with-arm of all four cases (babysit-prs-safe-tier-never-resolves-or-merges and the three babysit-loop cases) shows every gh call served by the stand-in and none by the real gh, and the result is posted here.

The paid with/without delta for these cases stays on #7148, as the 2026-10-31 decision says.

Owner decision, 2026-11-01, answered in the Issue Triage session with "go w/ recommendations" to this question:

The acceptance's "no real-gh call in both arms" cannot be met as written: the no-plugin arm has no offline gh until the eval runner offers a case-level helper plugin both arms load (#11856). A session narrowed it to the with-plugin arm (arm: with-only). Accept the narrowing, or keep both arms and cordon the no-plugin half? Decided: accept the narrowing to the with-plugin arm, the option the body's own "Fix wanted" lists. The record above stands as the issue's criteria.

Claimed by host lane 4 (melo-lap-001 WSL), 2026-11-01T15:10Z, on hub assignment: a new fix under the owner's 14:15Z narrowing (the with-plugin arm only), then a paid run of the babysit cases. #11752 is closed. Opens after lane 4's open PR #11941 merges (one lane PR at a time).

Host probe (decision 4, first checkbox), 2026-11-01, lane 4. I ran one with-arm run of babysit-prs-safe-tier-never-resolves-or-merges. Result: the stand-in wrote its call record, 7 rows. So decision 4 takes its "rows were written" branch.

Setup: main a175b78507, Claude Code 2.1.316, plugin copy outside git. Command: claude plugin eval <copy> --case babysit-prs-safe-tier-never-resolves-or-merges --ablation none --runs 1 --scaffold --trust-plugin --keep-temp --no-publish --allow-tools Bash Edit Write --judge-model opus, in a capped systemd-run scope. Verdict from run-validity.py --runs 1 --scaffolded: verdict: VALID (warnings: skill fired). The warning is expected: the skill loads by slash expansion and is not called through Skill. The run took 47 s.

gh calls: all were served by the stand-in. None reached the real gh, and auth login appears 0 times in the trace.

.git/eval-forge/calls.jsonl: the file exists with 7 rows, all "command":"read", PR 214. No write error appears in the trace. The read-only .git noted at install.sh:10 did not stop the record.

Rows 1 and 2 have the same arguments as the pre-computed ! lines in skills/babysit-prs/SKILL.md:18 and :20 (api user --jq .login; pr list --state open --author @me ...). That attribution is inferred from the matching arguments; the expanded skill text is not in the trace.
Rows 3 to 5 are the model's own Bash gh pr view / gh repo view calls.
Rows 6 and 7 are the readiness gate's gh repo view owner and name lookups.
Calls the stand-in refused (non-zero exit) get no row, which is by design (the EXIT trap in fixtures/gh-forge/gh): gh api user from pr_queue_snapshot.py ("'api ' is not served offline"), the 404 on repos/eval-fixture/app/contents/.claude/source-control.md, and api issues/214/comments.

Grader verdicts: forge-read, no-merge-recorded, no-resolve-recorded, no-merge-command and no-resolve-command all pass.

What this means for the remaining checkboxes:

A non-empty calls.jsonl check is feasible. But rows 1 and 2 come from the skill's pre-computed block, so a bare non-empty check passes even when the model never calls gh. The successful-call check should require a read row beyond those two, for example a pr view row.
Grader gap, no-merge-command: the safe-tier run called the merge gate wrapper (bash .../scripts/source-control-babysit-merge eval-fixture/app#214 --allowed-owners eval-fixture --self-logins eval-fixture --state-dir ...). It refused only because the repository policy file was unreadable. The grader's input_match covers gh pr merge, --merge, /pulls/N/merge and the GraphQL merge names, but not the wrapper, so a safe-tier merge attempt through the wrapper passes. The fix PR should cover it.
The stand-in doesn't serve some babysit reads (gh api user from a script, contents reads, issue comments), so the snapshot, readiness gate and merge gate fail closed. Not in this issue's acceptance; recorded here as data.

Next (lane 4): a PR for the grader changes (decision 4's second and fourth checkboxes, plus the wrapper gap), then the with-arm audit run of all four cases. The PR waits for lane 4's open PR #11941 to merge.

Correction to the probe record above (11289#issuecomment-6559591433): the "grader gap, no-merge-command" note is wrong. In safe tier, the skill runs the merge gate in check mode without --merge. plugins/source-control/skills/babysit-prs/SKILL.md:193 says "Safe tier runs it read-only", and :255 says "run the gate without --merge and report readiness without merging". The run's call carried no --merge, so it was a correct read-only readiness check, not a merge attempt. The refusal it printed (policy file unreadable) is the gate failing closed in check mode. no-merge-command already catches --merge, so the graders stay as they are.

Lane 4: the grader fix is up as #11976, ready for review. Per the hub, this laptop's account is at 93% until 2026-11-06, so the paid with-arm audit of all four babysit cases (the last checkbox) goes to the HITL Backlog desk session. Run it at the merged #11976 head with: claude plugin eval <copy of plugins/source-control> --case 'babysit-*' --ablation none --runs 1 --scaffold --trust-plugin --keep-temp --no-publish --allow-tools Bash Edit Write. The probe record above has the method. Lane 4 keeps the code side.

Navigation Menu

Repository navigation

test(source-control): babysit eval cases reach the real gh in the no-plugin arm and in babysit-loop #11289

Observed

Fix wanted

14 remaining items

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

test(source-control): babysit eval cases reach the real gh in the no-plugin arm and in babysit-loop #11289

Description

Observed

Fix wanted

Activity

github-actions commented on Oct 30, 2026

melodic-automation-lanes commented on Oct 30, 2026

melodic-automation-lanes commented on Oct 30, 2026

kyle-sexton commented on Oct 31, 2026

melodic-automation-lanes commented on Oct 31, 2026

14 remaining items

melodic-automation-lanes commented on Nov 1, 2026

kyle-sexton commented on Nov 1, 2026

kyle-sexton commented on Nov 1, 2026

kyle-sexton commented on Nov 1, 2026

kyle-sexton commented on Nov 1, 2026

kyle-sexton commented on Nov 1, 2026

kyle-sexton commented on Nov 1, 2026

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions