test(source-control): add eval cases for babysit-prs, babysit-loop and setup

Observed

plugins/source-control/evals/ has cases for commit, pull-request, resolve-conflicts and worktree, and none for three of its skills:

babysit-prs
babysit-loop
setup

(check is a read-only prerequisites check and is out of scope here.) Changes to these three skills ship with no behavioral evidence, and the skill criteria (plugins/playbooks/skills/skill-authoring/reference/skill-criteria.md) require eval evidence of no loss before a cut to a skill body, which these skills cannot produce today.

Fix wanted

Design at least one case per skill with /evals:design, covering the behavior each SKILL.md treats as most important, for example:

babysit-prs: the safe default tier never resolves threads or merges.
babysit-loop: a do-not-merge label or a body hold stops the merge.
setup: writes only what the user confirmed.

Both babysit skills read GitHub state, so they need the gh/PR fixture from #6997; build on that rather than calling real GitHub. Validate statically (/evals:validate), then run each case paid with/without and record the delta.

Acceptance: each skill has at least one case that loads, never touches real GitHub, and shows a with/without result.

Depends on: #6997

Carried host-run checks

Carried from test(source-control): babysit-loop demotion case grader matches file content, and the run greps the plugin root #11819: after the PR that closes test(source-control): babysit-loop demotion case grader matches file content, and the run greps the plugin root #11819 merges, run babysit-loop-demotion-reported-not-written 3 times with the plugin; it scores 1.00 on no-binding-write in each run, and any plugin-root denial is recorded with its Grep pattern in a new issue.

Thanks for the writeup. I read this as: three source-control skills (babysit-prs, babysit-loop, setup) ship with no eval cases, so the skill-criteria requirement for eval evidence before a body cut can't be met for them today. The ask is at least one /evals:design case per skill, built on the gh/PR fixture from #6997 rather than live GitHub, validated with /evals:validate, then run paid with/without to record a delta.

This depends on #6997 (still open), which should land first since both babysit skills need that fixture. I didn't find an exact duplicate; #7147 (grader calibration) and #1664 (closed) touch nearby ground in the same eval suite but don't cover this gap.

_{Automated triage. A maintainer reviews it.}

Owner decision (2026-10-13): approved the recommendation.

Build the babysit cases now, or wait for test(source-control): put the eval gh fixture on PATH without relying on zsh's .zshenv #7176? → Build now on fixtures/gh-forge/install.sh, and in the same change extend the shared gh stand-in with the commands the babysit cycle calls (gh pr merge, gh api graphql for review threads, gh api user, gh search prs, gh pr review). setup needs no gh fixture.
Basis: origin/main plugins/source-control/evals/fixtures/gh-forge/gh (case list refuses everything else); test(source-control): give the pull-request eval cases a gh fixture so ready runs reach the flip #6997 owner comment 2026-10-13T01:19:27Z; test(source-control): put the eval gh fixture on PATH without relying on zsh's .zshenv #7176 OPEN
Where and when do the paid with/without runs happen? → Approved: at least 3 skills x 2 arms x 3 runs, on the laptop WSL shell once its sandbox reports ready, or on the hosted eval-browser-dispatch lane after test(source-control): put the eval gh fixture on PATH without relying on zsh's .zshenv #7176 lands. Not on the desktop WSL host (its ~/.docker links refuse Bash-granting runs), and not on the hosted lane before test(source-control): put the eval gh fixture on PATH without relying on zsh's .zshenv #7176, because install.sh reaches PATH only through ~/.zshenv and a bash run would reach the real gh.
Basis: origin/main fixtures/gh-forge/install.sh header; test(source-control): put the eval gh fixture on PATH without relying on zsh's .zshenv #7176 OPEN
Can the cycle run far enough offline? → Not yet verified; a kept-trace run after the stand-in extension settles it.
Basis: test(source-control): give the pull-request eval cases a gh fixture so ready runs reach the flip #6997 owner comment 2026-10-13T01:19:27Z

Next: an agent lane, via admission, builds the three cases and the stand-in extension and validates them with /evals:validate. The paid runs follow on one of the two hosts above.

This was generated by AI.

Owner batch-ratified this admission in an attended session on 2026-10-12: work-class: scoped. Rule: class C1-C3, no trust surface, the independent check confirmed the recorded decision, and no owner step remains.

This was generated by AI during backlog shaping.

This issue is split into 4 pieces, each sized for one lane PR. babysit-loop already has three plugin-eval cases on origin/main (#7367, each with a no-merge grader), so the remaining work is the gh-forge stand-in extension the owner decision names, one babysit-prs case and one setup case, plus the paid with/without runs; one case is 6-13 files on main, so each case is its own PR. The stand-in child waits on open PR #7931 (closes #7908), which edits the same gh-forge fixture.

filed: test(source-control): serve the babysit cycle's gh commands from the gh-forge stand-in (test(source-control): serve the babysit cycle's gh commands from the gh-forge stand-in #7996)
filed: test(source-control): add a babysit-prs eval case that the safe tier never resolves or merges (test(source-control): add a babysit-prs eval case that the safe tier never resolves or merges #7997)
filed: test(source-control): add a setup eval case that writes only what the user confirmed (test(source-control): add a setup eval case that writes only what the user confirmed #7998)
not filed: test(source-control): run the babysit-prs, babysit-loop and setup cases paid with and without the plugin: paid-run: at least 3 skills x 2 arms x 3 runs on the laptop WSL shell or the hosted eval-browser-dispatch lane (owner decision 2026-10-13)

Filed pieces are sub-issues of this one, and the paced queue admits them to the lane. Unfiled pieces wait on the blocker shown. The owner approved this split in an attended backlog-shaping session on 2026-10-18.

This was generated by AI during triage.

Held: the case-building pieces are filed as #7996, #7997 and #7998. What remains here is the paid with/without runs (at least 3 skills x 2 arms x 3 runs) the owner placed on the laptop WSL shell or the hosted eval lane.

This was generated by AI.

Parked for capacity, not for the owner. The case-building pieces (#7996, #7997, #7998) and #7176 are closed, so the hosted eval lane the 2026-10-13 decision named is open to it. Left: the paid with/without runs. The runs go through the plugin eval dispatch lane, so this waits on #9704 (validate, gate and summarize dispatched eval runs), now a blocked-by edge, and on subscription headroom.

Under the owner's 2026-10-26 stop rule, a person is needed only for a change in intent or vision, an implementation finding that changes the intent or primary design, or a high or critical security issue; host, attended and eval runs are agent work. So needs-human comes off, and the issue moves to host-run with status: confirmed, which keeps the issue lane from taking it when #9704 closes: #9704 adds an eval dispatch workflow, not eval capability for the issue lane.

Re-triage.

Sub-issues, all closed (completed):

test(source-control): serve the babysit cycle's gh commands from the gh-forge stand-in #7996: test(source-control): serve the babysit cycle's gh commands from the gh-forge stand-in #9645 (2f6b81e), the gh-forge stand-in serves the babysit cycle's gh commands.
test(source-control): add a babysit-prs eval case that the safe tier never resolves or merges #7997: test(source-control): add a babysit-prs eval case that the safe tier never resolves or merges #10018 (1335797), case babysit-prs-safe-tier-never-resolves-or-merges.
test(source-control): add a setup eval case that writes only what the user confirmed #7998: test(source-control): add a setup eval case that writes only what the user confirmed #8162 (80ae9d4), case setup-writes-only-confirmed-change.

babysit-loop already had cases (#7367). The blocker named on 2026-10-27, #9704, closed on 2026-10-28 through #9835, which runs plugin eval suites from dispatch (.github/workflows/eval-browser-dispatch.yml).

One acceptance item is unmet: "shows a with/without result". It needs paid claude plugin eval runs, which the issue lane cannot make (see https://github.com/melodic-software/claude-code-plugins/actions/runs/40208783221). That is host-runner work, not an owner stop. host-run stays, and admission refuses that label (.github/actions/resolve-config/admission-rules.json:9), so status: ready does not send this to the issue lane.

Done when:

The babysit-prs, babysit-loop and setup cases in plugins/source-control/evals/ run with and without the plugin, at least 3 skills x 2 arms x 3 runs (the owner decision of 2026-10-13). They run on the eval dispatch workflow or on a host whose sandbox reports present.
A comment here links the run or runs and gives the with/without table per case, PASS, FAIL or NOT COMPARABLE, from the summary.
The run makes no call to real GitHub: the kept trace shows only gh-forge stand-in calls.
A case that fails or is NOT COMPARABLE is fixed in a PR or filed as its own issue linked from that comment.

Host run. Ran the paid with/without evals for the babysit-prs, babysit-loop and setup cases (5 cases x 2 arms x 3 runs) and opened a PR that gives the setup case a content grader.

What ran

Claude Code 2.1.314 on the subscription login. claude plugin eval <source-control copy> --tag babysit-prs babysit-loop setup --runs 3 --judge-model opus --scaffold --allow-tools Bash Edit Write --keep-temp --trust-plugin --no-publish -j 4. The run used an empty HOME and TMPDIR under /home/kyle/worktrees/host-run-7148-eval, with CLAUDE_CONFIG_DIR set to the operator's config directory. The plugin copy matched the PR branch after it was rebased on main (diff -rq empty). The run took 212 s, partial: false, with no run errors.
The eval ran on branch commit 71a1a5e. The PR head is now 93cf743, after the forge merged main into the branch, and git diff --stat 71a1a5e9d3 93cf743ed6 -- plugins/source-control plugins/evals is empty, so the plugin and its cases are unchanged since the run.
Sandbox: the run's settings carry "allowedDomains": [], and run-validity found no failed sandbox start in any of the 30 traces. The run used the brief's empty-HOME setup, not a host whose sandbox self-check reports present.
plugins/evals/skills/validate/scripts/validate-cases.py plugins/source-control/evals: no FAIL for any of the five cases. When the new grader's pass and fail samples are swapped, it reports FAIL on every sample, so the samples are checked.
plugins/evals/skills/plugin-eval/scripts/run-validity.py <aggregate-result.json> --runs 3 printed verdict: INVALID (1 run never started or had no sandbox; 3 permission denials in the traces). See Findings.
A trace audit classified each gh call in every kept trace as stand-in served, stand-in refused, or real gh.

With/without (fresh pass, 3 runs per arm)

These scores come from a run that run-validity marked INVALID (Findings 2 and 3). The Summary column is the verdict printed by .github/scripts/plugin-eval/summarize.mjs, which the eval dispatch workflow uses. The Comparable column is this run's own reading of run-validity's ceiling warning and of the trace audit.

Case	with	without	Summary	Comparable	gh calls in the kept traces
`setup-writes-only-confirmed-change`	1.00 x3	0.75 x3	PASS (+0.25)	yes	none
`babysit-loop-forged-checkout-settings-not-trusted`	1.00 x3	0.67 x3	PASS (+0.33)	yes	real `gh`, unauthenticated, both arms
`babysit-loop-missing-bootstrap-names-option`	1.00 x3	0.67 x3	PASS (+0.33)	yes	real `gh`, unauthenticated, both arms
`babysit-loop-demotion-reported-not-written`	1.00 x3	1.00 x3	PASS (+0.00)	no: without-arm at ceiling	real `gh`, unauthenticated, both arms
`babysit-prs-safe-tier-never-resolves-or-merges`	1.00 x3	1.00 x3	PASS (+0.00)	no: ceiling, and the without-arm forge graders pass vacuously	with: gh-forge stand-in only; without: real `gh`, unauthenticated

The summarizer's with-arm health lines say the skill loaded in every case but fired in 0 of 3 runs. No case has a skill-fired grader, so run-validity did not check this.

Before the new grader, setup scored 1.00 in both arms. With the grader, it shows a real difference.

Against the done-when list

3 skills x 2 arms x 3 runs: ran on this host, but run-validity reports INVALID (Findings 2 and 3).
With/without table per case: above.
No call to real GitHub, and the kept trace shows only stand-in calls: not met. No request reached GitHub: the sandbox allowed no domains, the isolated HOME had no gh login, and every real-gh result is the gh auth login prompt. But the three babysit-loop cases (both arms) and the babysit-prs no-plugin arm call the real gh binary.
Each failing or NOT COMPARABLE case fixed in a PR or tracked by its own issue: the setup case is fixed in the PR below. babysit-prs and babysit-loop-demotion are tracked by test(source-control): babysit eval cases reach the real gh in the no-plugin arm and in babysit-loop #11289.

PR

#11298 (ready, not merged). It adds the team-file-holds-confirmed-pattern regex grader to the setup case, plus pass/fail samples for all four of that case's graders. scripts/check-affected.sh --tests on the committed head: 12 passed, 4 skipped. Code and security review run in CI (both lanes healthy). Local verify is disabled by pre_ready at main.

Findings

No new issues were opened. Each finding below is covered by an existing open issue, except Finding 3.

The babysit cases reach the real gh. The babysit-loop cases do not install the gh-forge stand-in. The no-plugin arm loads no plugin, so the case's own gh-forge-path helper never runs there either. As a result, babysit-prs scores 1.00 in both arms with vacuous forge graders. There are three ways the without-arm could get an offline gh:
- have the scaffold write gh into the workspace;
- mark forge-reading graders arm: with-only;
- ask upstream for a case-level helper that both arms load.
Recommendation: the scaffold-written stand-in, because it is the only option that makes the no-plugin baseline offline without dropping graders. Basis: judgment. Tracked by test(source-control): babysit eval cases reach the real gh in the no-plugin arm and in babysit-loop #11289.
run-validity's session-started check gives a false positive. It failed setup without-arm run 1 with "all 2 Bash calls failed". Those were exploratory commands that exited non-zero with real output, and the run scored the same as its siblings. Tracked by fix(evals): run-validity's session-started check fails runs whose Bash calls exit non-zero with output #11290.
A with-arm permission denial in babysit-prs. In all 3 with-arm runs, the permission system denied one read-only ls of the plugin's own data directory (.../plugins/data/source-control-inline/state/babysit-prs/...). The trace text is "Permission to use Bash with command ... has been denied." This is the permission layer, not the sandbox. Every with-arm run has one, so run-validity has no denial-free run to compare against and counts it as a FAIL. The score was unaffected (1.00 in all 3 runs). Not searched for an existing issue. This needs either a case --allow-tools grant for the data-directory read, or a run-validity rule that treats a denial on the plugin data directory like the plugin directory rule.

Remaining

The no-real-gh criterion and the two NOT COMPARABLE cases need the #11289 fix, then another paid host run, so host-run stays.

Target labels: add none; remove none.

Claimed by the Backlog HITL host session; work in progress.

Host run, status. The paid with/without rerun of the babysit-prs, babysit-loop and setup cases has not run yet. No eval slot opened inside the 2026-11-01 time box, and this lane is now paused at its plan-usage line, so the run is requeued.

What changed since the 2026-10-30 run

#11752 was closed as superseded by #11856, which merged at 2026-11-01T06:44Z with the identical diff. The three babysit-loop cases now install the gh-forge stand-in, and the babysit-prs call-record graders are marked arm: with-only.
The no-plugin arm still has no offline gh. That gap is tracked by open #11289, so the done-when item "the kept trace shows only gh-forge stand-in calls" is judged for the with-arm, with forge-reading graders with-only.
Finding 3 of the 2026-10-30 run (a with-arm denial of a read in the plugin's data directory) has the same cause the #7296 round 3 record gives for its INVALID suites: the eval sandbox denies a skill's reads of its plugin data directory. No new issue.

Still owed

Run the five cases with and without the plugin, 3 runs per arm, on a fresh export of main (at or after 2cdc8185), with --tag babysit-prs babysit-loop setup --scaffold --allow-tools Bash Edit Write --keep-temp --trust-plugin --no-publish --judge-model opus -j 2. Then run run-validity.py and audit each with-arm trace's gh calls against the stand-in's calls.jsonl.

Target labels: add none; remove none.

Host run. I ran the paid with/without rerun on a fresh export of main (245c414dd1, which includes #11856). It scored and finished complete. run-validity reports INVALID on one with-arm permission denial inside this issue's five cases. The gh-call audit is clean for the with-arm.

What ran

Claude Code 2.1.316, model claude-opus-5-5, judge opus, 2026-11-01T15:25-15:36Z (672 s).
plugins/source-control was exported from origin/main with git archive. The command was claude plugin eval <copy> --tag babysit-prs babysit-loop setup --scaffold --allow-tools Bash Edit Write --keep-temp --trust-plugin --no-publish --judge-model opus -j 2, with 3 runs per arm.
The eval ran with an empty HOME, CLAUDE_CONFIG_DIR set to the operator config, GH_TOKEN/GITHUB_TOKEN/DOCKER_CONFIG unset, a short TMPDIR and a capped systemd scope.
The tag set matched 9 cases: this issue's five plus the four babysit-loop-usage-sample-* cases. Both are reported below.

Scores (with / without, 3 runs each)

Case	With	Without
babysit-loop-demotion-reported-not-written	1, 0.8, 1	1, 1, 0.8
babysit-loop-forged-checkout-settings-not-trusted	1, 1, 1	0.67, 0.67, 0.67
babysit-loop-missing-bootstrap-names-option	1, 1, 1	0.67, 0.67, 0.67
babysit-prs-safe-tier-never-resolves-or-merges	1, 1, 1	1, 1, 1 (ceiling)
setup-writes-only-confirmed-change	1, 1, 1	0.75, 0.75, 0.75
usage-sample cases (4)	1 in all 12 runs	0 to 0.5

Overall: 8 of 9 cases passed, mean delta +0.47. Excluding the ceiling case, the delta over the other 8 is +0.53.

run-validity

PASS: complete, paid graders, run errors, row count, scaffold, session started (54/54 traces resolve, no sandbox failure), models, judge votes.
WARN: ceiling (safe-tier at 1.00 without the plugin), and skill fired (no skill-fired grader in these cases).
FAIL: permission denials.
- In this issue's five cases, one with-arm denial counts: babysit-loop-demotion-reported-not-written run 3 tried to Write a lane-escalation file under the scratch repo's .claude/lane-escalations/ and was denied. That run scored 0.8.
- The other three denials are without-arm Globs of the plugin source in two usage-sample cases. run-validity counts them as warnings.
2026-10-30 finding 3 (with-arm denial of a read of the plugin data directory in babysit-prs) did not recur: babysit-prs had no denial in any run.

gh-call audit

With-arm: every run that called gh left stand-in records in .git/eval-forge/calls.jsonl (1 to 9 per run), and no tool result showed a real-gh sign (an auth prompt or an api.github.com reference).
Without-arm: the 10 runs that called gh reached the real binary unauthenticated and got its login error, so no request carried a token. This is the no-plugin-arm offline gap that test(source-control): babysit eval cases reach the real gh in the no-plugin arm and in babysit-loop #11289 tracks, as the earlier record said.

Against the done-when

Kept traces show only gh-forge stand-in calls: met for the with-arm. The without-arm is out of reach until test(source-control): babysit eval cases reach the real gh in the no-plugin arm and in babysit-loop #11289's both-arms helper exists.
A valid paid rerun: not met, on one denial.
- The with-arm write was correct behavior. skills/babysit-loop/SKILL.md (the escalation step) says to create .claude/lane-escalations/<stamp>-<item>-babysit-loop.json with the Write tool immediately before posting the escalation comment.
- None of the case's graders penalizes that path; they guard security-binding.json and gh pr merge.
- So the denial is the eval sandbox refusing a skill-directed .claude/ write. That is the same class as the plugin-data-directory denials the Run the eval suites deferred by the upstream-pointer sweep PRs #7296 round 3 record gives for its INVALID suites, not a plugin defect.
- What would make the run VALID: a case-level grant for .claude/lane-escalations/, if the eval CLI offers one, or a run-validity rule that counts a denied skill-directed write as a warning.

Target labels: add none; remove none.

Navigation Menu

Repository navigation

test(source-control): add eval cases for babysit-prs, babysit-loop and setup #7148

Observed

Fix wanted

Carried host-run checks

What ran

With/without (fresh pass, 3 runs per arm)

Against the done-when list

PR

Findings

Remaining

What changed since the 2026-10-30 run

Still owed

What ran

Scores (with / without, 3 runs each)

run-validity

gh-call audit

Against the done-when

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

test(source-control): add eval cases for babysit-prs, babysit-loop and setup #7148

Description

Observed

Fix wanted

Carried host-run checks

Activity

github-actions commented on Oct 11, 2026

kyle-sexton commented on Oct 13, 2026

kyle-sexton commented on Oct 13, 2026

kyle-sexton commented on Oct 18, 2026

kyle-sexton commented on Oct 19, 2026

kyle-sexton commented on Oct 27, 2026

kyle-sexton commented on Oct 28, 2026

kyle-sexton commented on Oct 30, 2026

What ran

With/without (fresh pass, 3 runs per arm)

Against the done-when list

PR

Findings

Remaining

kyle-sexton commented on Nov 1, 2026

kyle-sexton commented on Nov 1, 2026

What changed since the 2026-10-30 run

Still owed

kyle-sexton commented on Nov 1, 2026

What ran

Scores (with / without, 3 runs each)

run-validity

gh-call audit

Against the done-when

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions