[kyle-sexton]

Refs: #11289

Summary

Owner-login replacement for #11752, opened by the work-resolve-issue lane; this branch merges that PR's head (2b1f1ac) onto current main with no other change. The lane run's own summary:

The three babysit-loop eval cases now install the gh-forge stand-in and list a per-case gh-forge-path plugin. Each stand-in serves the two PRs its prompt names (＃41/＃42, ＃51/＃52, ＃61/＃62), and the demotion case installs into ./app with a hook that resolves app's git dir. install.sh gains --dir and --extra-pr and sets alias.remote-evalforge so git finds the remote helper without PATH. The stand-in also records every call it served as a "read" line. Decision 2 asked to put the stand-in on PATH through the workspace .claude/settings.json env.PATH and probe it first. I skipped that because the plugin-evals page says a run loads no .claude/ directory inside the workspace, even one a scaffold wrote. So I applied the decision's fallback: the three call-record graders are now arm: with-only. The trace-based no-merge-command and no-resolve-command graders stay scored in both arms.

Remaining work

• The no-plugin arm still has no offline gh: a run loads no scaffold-written .claude/settings.json and a case can set only EVAL_* env, so a bare gh there is the real (unauthenticated, network-denied) one. The acceptance line 'no real-gh call in both arms' needs a case-level helper plugin that both arms load, which the eval runner does not offer today; until then forge-read, no-merge-recorded and no-resolve-recorded are scored in the with-plugin arm only.

Not verified

• Decision 2's probe run and the workspace .claude/settings.json env.PATH write were not done. The plugin-evals docs say a run loads no scaffold-written .claude/ directory, so the decision's fallback (arm: with-only on the call-record graders) was applied directly. A host probe run would confirm this.
• install.sh's own comments (install.sh:10, :122) say the eval sandbox holds .git read-only for a run's Bash, so calls.jsonl may stay empty in a run. For that reason review moved the three call-record graders back to main's logic (they pass on an empty record) and kept only arm: with-only; see the review threads on test(source-control): give the babysit eval cases the offline gh stand-in #11752.
• gh-forge.test.sh was not run directly by the lane; left to this PR's CI.
• The paid with/without kept-trace audit of the four cases (claude plugin eval) needs a host run; per the recorded decision it stays on ＃7148.

Fix

• plugins/source-control/evals/fixtures/gh-forge/install.sh: new --dir <checkout> and --extra-pr <number> <head> <title> options (extra heads pushed and given upstreams); sets alias.remote-evalforge to the helper's absolute path; header comment says the no-plugin arm has no stand-in
• plugins/source-control/evals/fixtures/gh-forge/gh: serves extra PRs from extra-prs in pr view/list/search/diff/checks/merge/review; an EXIT trap records each call that exited 0 and wasn't already recorded as {"command":"read",...} in calls.jsonl
• plugins/source-control/evals/gh-forge.test.sh: asserts a served read is recorded; the pr create assertion reads the last record; new block covers --dir, --extra-pr (list, view, merge record of PR 62) and git fetch through the alias with the helper off PATH
• plugins/source-control/evals/babysit-prs-safe-tier-never-resolves-or-merges/graders/{forge-read,no-merge-recorded,no-resolve-recorded}.md: main's matching logic unchanged; only arm: with-only added, because the no-plugin arm has no stand-in and its gh is the real one. Samples are main's.
• plugins/source-control/evals/babysit-loop-{demotion-reported-not-written,forged-checkout-settings-not-trusted,missing-bootstrap-names-option}/scaffold.sh: create two head branches, return to main, run install.sh for the prompt's two PRs (--dir app in the demotion case)
• plugins/source-control/evals/babysit-loop-{demotion-reported-not-written,forged-checkout-settings-not-trusted,missing-bootstrap-names-option}/prompt.md: plugins: ["../..", "gh-forge-path"]
• plugins/source-control/evals/babysit-loop-{forged-checkout-settings-not-trusted,missing-bootstrap-names-option}/gh-forge-path/.claude-plugin/plugin.json: new, copy of the babysit-prs case's hook plugin
• plugins/source-control/evals/babysit-loop-demotion-reported-not-written/gh-forge-path/.claude-plugin/plugin.json: new, the same hook but resolving the git dir of ./app
• .changes/source-control/test-babysit-evals-offline-gh-3c81f0a4.md: bump none fragment

Verification

• Patch-id equality: gh pr diff 11752 | git patch-id --stable and git diff origin/main HEAD | git patch-id --stable both give 6a9310d11bfa46c9d8fe5c9f9b8eaf292ea46dfd, so this branch carries test(source-control): give the babysit eval cases the offline gh stand-in #11752's change and nothing else.
• From the lane run (https://github.com/melodic-software/claude-code-plugins/actions/runs/41250059645): scripts/check-changelog-fragments.sh --check passed (all 6 fragments valid); check-affected: 12 passed, 6 skipped over 19 changed files; a fresh-context review subagent over the whole diff found no confirmed bugs.
• Sources the lane read: https://code.claude.com/docs/en/plugin-evals (the without-arm runs with no plugin; a run reads no .claude/ directory in the workspace; case env keys must match EVAL_; arm: with-only scores a grader in the with-arm only), https://raw.githubusercontent.com/git/git/master/transport-helper.c (get_helper() runs the helper as a git command with args remote-<name> <remote> <url> and sets GIT_DIR), https://git-scm.com/docs/git-config (alias. with a leading ! runs a shell command from the repository's top-level directory).

Related

• test(source-control): babysit eval cases reach the real gh in the no-plugin arm and in babysit-loop #11289
• Related: replaces test(source-control): give the babysit eval cases the offline gh stand-in #11752

🤖 Generated with Claude Code

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-11-01T06:05:51.659211Z	8950cd6	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[github-actions]

claude-security-review reviewed 8950cd67f74b9806479fd47d3ed42916d18e07de: read 16 of 16 changed file(s); 9 finding(s), 0 as inline comments.

Security review: PR #11856 (eval-only: offline gh stand-in for the babysit-loop eval cases)

I found no exploitable security issue. Every change sits under plugins/source-control/evals/ or .changes/. No skill, hook, workflow, trust control, merge hold or token path changes.

What I checked

• Trusted paths: no changed file matches a TRUSTED PATHS: glob. .claude-plugin/ is not .claude/.
• Trust controls and merge holds: none touched. The three grader changes (arm: both → arm: with-only on forge-read, no-merge-recorded and no-resolve-recorded) only change how eval scores are computed. They gate no merge and no lane.
• Instruction surface: no standing instruction is deleted or softened. The prompt changes only add a plugins: front-matter line.
• gh stand-in (fixtures/gh-forge/gh): the new EXIT trap appends argv to the forge's local calls.jsonl through jq --args, so nothing is shell-evaluated. extra-prs is read with read -r and only compared, never evaluated. Every command it doesn't serve still exits 1, and nothing reaches a network.
• install.sh: --dir and --extra-pr take scaffold-supplied literals. alias.remote-evalforge runs $forge/bin/git-remote-evalforge. $forge comes from git rev-parse inside the scaffolded eval workspace, so no outside input reaches it.

Plugin manifests. The three new gh-forge-path/.claude-plugin/plugin.json files are nested eval-case plugins. .claude-plugin/marketplace.json has no entry for gh-forge-path (the grep found none). The source-control entry is "source": "./plugins/source-control" (marketplace.json#L403), so installed users load only that plugin's root manifest. A nested plugin.json under evals/ is not loaded as a plugin, so its SessionStart hook runs only when the eval runner loads the case through plugins: ["../..", "gh-forge-path"]. The same one-file hook plugin is already in nine other cases on main, for example evals/babysit-prs-safe-tier-never-resolves-or-merges/gh-forge-path/. These flags come from the default-deny manifest rule, so all are IMPORTANT and none is CRITICAL:

• manifest hooks: flag, new SessionStart command hook in evals/babysit-loop-demotion-reported-not-written/gh-forge-path. It runs git -C app rev-parse, then appends export PATH="<app git dir>/eval-forge/bin:$PATH" to $CLAUDE_ENV_FILE when gh there is executable. Eval runs only, not installed for users.
• manifest name: flag, gh-forge-path (eval-only plugin name, same file)
• manifest version: flag, 0.0.0 added (same file)
• manifest hooks: flag, new SessionStart command hook in evals/babysit-loop-forged-checkout-settings-not-trusted/gh-forge-path. Same command, but it resolves the workspace's own git dir. Eval runs only.
• manifest name: flag, gh-forge-path (same file)
• manifest version: flag, 0.0.0 added (same file)
• manifest hooks: flag, new SessionStart command hook in evals/babysit-loop-missing-bootstrap-names-option/gh-forge-path. Identical to the forged-checkout copy. Eval runs only.
• manifest name: flag, gh-forge-path (same file)
• manifest version: flag, 0.0.0 added (same file)
• manifest description: pass (inert field)

Spend (not a finding): three more eval cases now load an extra helper plugin. The paid with/without eval runs the PR mentions spend prepaid eval credit only.

Embedded text: none. The PR context contains no directive aimed at the review lane.

critical-findings: 0

Findings not posted inline (their lines are not in the diff):

• plugins/source-control/evals/babysit-loop-demotion-reported-not-written/gh-forge-path/.claude-plugin/plugin.json:28 (RIGHT): manifest hooks: flag, new SessionStart command hook in an eval-only nested plugin

What runs: git -C app rev-parse --path-format=absolute --git-common-dir. If $CLAUDE_ENV_FILE is set and <app git dir>/eval-forge/bin/gh is executable, the hook appends export PATH="<that dir>:$PATH" to $CLAUDE_ENV_FILE. It runs only when the eval runner loads this case's plugins: ["../..", "gh-forge-path"].

Why this is IMPORTANT, not CRITICAL: marketplace.json has no gh-forge-path entry, and installed users load only the root manifest of ./plugins/source-control. A nested plugin.json under evals/ is never loaded, so nobody gets this hook on update. The same hook is already in nine other eval cases on main.

severity: IMPORTANT

• plugins/source-control/evals/babysit-loop-forged-checkout-settings-not-trusted/gh-forge-path/.claude-plugin/plugin.json:92 (RIGHT): manifest hooks: flag, new SessionStart command hook in an eval-only nested plugin

What runs: git rev-parse --path-format=absolute --git-common-dir in the workspace. If $CLAUDE_ENV_FILE is set and the stand-in gh under <git dir>/eval-forge/bin is executable, the hook prepends that directory to PATH through $CLAUDE_ENV_FILE. It runs only when this eval case loads it.

Why this is IMPORTANT, not CRITICAL: marketplace.json has no gh-forge-path entry, and installed users load only the root manifest of ./plugins/source-control. Nobody gets this hook on update. It is a byte-for-byte copy of the hook plugin the existing babysit-prs case already uses.

severity: IMPORTANT

• plugins/source-control/evals/babysit-loop-missing-bootstrap-names-option/gh-forge-path/.claude-plugin/plugin.json:153 (RIGHT): manifest hooks: flag, new SessionStart command hook in an eval-only nested plugin

What runs: the same command as the forged-checkout case. It resolves the workspace's git common dir and, when the stand-in gh there is executable, prepends <git dir>/eval-forge/bin to PATH through $CLAUDE_ENV_FILE. It runs only when this eval case loads it.

Why this is IMPORTANT, not CRITICAL: marketplace.json has no gh-forge-path entry, and installed users load only the root manifest of ./plugins/source-control. Nobody gets this hook on update.

severity: IMPORTANT

[github-actions]

claude-review reviewed 8950cd67f74b9806479fd47d3ed42916d18e07de: read 16 of 16 changed file(s); 0 finding(s), 0 as inline comments.

Code review: PR #11856 (offline gh stand-in for the babysit-loop eval cases)

I read the whole diff, every hunk of all 16 changed files. I also read the full head versions of fixtures/gh-forge/gh, fixtures/gh-forge/install.sh, the first half of gh-forge.test.sh, the demotion and forged-checkout scaffolds, and the demotion prompt. No changed file matches a TRUSTED PATHS glob. The .claude-plugin/ directories are not .claude/.

No blocking or advisory findings.

What I checked

• gh stand-in
  • pr_matches sets the extra PR's PR_* values in the main shell, so they carry through to pr_doc, record and emit. It is never called from a subshell.
  • extra_docs uses temporary prefix assignments on a function call. In bash those are scoped to the call, so the main PR's values are not overwritten.
  • jq -sc . on empty input gives [], so pr list and search prs still return valid JSON when nothing matches.
  • The EXIT trap keeps the exit code. Bash resets traps in $(...) subshells, so no call is recorded twice.
  • Calls that already write their own record (pr merge, pr review, resolve-thread, pr create) set recorded=1, so they get no extra read line.
  • The record read … 2>/dev/null call stays quiet when .git is read-only.
• install.sh
  • The --extra-pr parsing shifts the right number of arguments (shift 3 plus the loop's shift consumes all 4 tokens).
  • cd "$dir" comes after here is resolved.
  • Extra heads are pushed and given upstreams.
  • The alias is quoted, and git falls back to an alias for an unknown git remote-<name> call.
• Existing graders and tests
  • no-merge-recorded and no-resolve-recorded match on "command":"pr merge" and "command":"resolve thread", so the new read lines cannot match them.
  • Existing test assertions still hold with read lines interleaved. The refused-merge count at gh-forge.test.sh:103 is still 1, and the resolve-thread tail -n 1 check still reads the right line.
• Scaffolds: each new branch is cut from main, and each scaffold returns to main before install.sh runs. In the demotion case, the hook resolves app's git dir with git -C app.

I ran nothing; this review is from reading only. The PR's statements that gh-forge.test.sh passes, that the patch-ids are equal, and how the eval sandbox behaves are author-claimed and unverified here.

Observations (not counted as findings)

• The gh header says a read line lets "a grader tell a served call from one the real gh failed". But a served call that exits non-zero gets no record. One example is gh pr checks <n> against the empty checks file all three new cases install: it prints "no checks reported" and exits 1. No grader reads read lines yet. If one is added, that sentence needs narrowing.
• pr ready changes the draft state but is recorded as "command":"read".
• The demotion prompt's snapshot calls feat: add discovery plugin (explore + research skills) #61 and feat: add claude-config-audit plugin #62 "green" with work-class-labeled close-linked items. The stand-in serves both PRs with no checks, no labels and an empty body. The case's expected outcome (no merge, demotion reported) does not depend on that data, but a lane that reads the forge sees facts that contradict the prompt.

Plugin manifests (new eval-only gh-forge-path plugins in three cases)

• manifest userConfig: pass (none declared)
• manifest version: pass (new eval-only helper at 0.0.0; the source-control fragment bump: none fits an evals-only change)
• manifest description: pass (the demotion copy correctly says it resolves ./app's git dir)

Closing issues

The PR context file lists no closing-issue section. The PR only Refs: #11289, so I checked no acceptance criteria.

Path-scoped rules

.claude/rules/eval-case-transcripts.md matches plugins/*/evals/**. Nothing added is a session or product transcript.

Spend

The change adds no new paid runs. The with/without eval audit the PR mentions is deferred to #7148 and would spend eval-run usage when it runs.