[melodic-automation-lanes]

Refs: #11289

Summary

Opened by the work-resolve-issue lane. The run's own summary:

The three babysit-loop eval cases now install the GH -forge stand-in and list a per-case GH -forge-path plugin. Each stand-in serves the two PRs its prompt names (＃41/＃42, ＃51/＃52, ＃61/＃62), and the demotion case installs into ./app with a hook that resolves app's git dir. install.sh gains --dir and --extra-pr and sets alias.remote-evalforge so git finds the remote helper without PATH. The stand-in also records every call it served as a "read" line. Decision 2 asked to put the stand-in on PATH through the workspace .claude/settings.json env.PATH and probe it first. I skipped that because the plugin-evals page says a run loads no .claude/ directory inside the workspace, even one a scaffold wrote. So I applied the decision's fallback: the three call-record graders are now arm: with-only. The trace-based no-merge-command and no-resolve-command graders stay scored in both arms.

Remaining work

• The no-plugin arm still has no offline gh: a run loads no scaffold-written .claude/settings.json and a case can set only EVAL_* env, so a bare gh there is the real (unauthenticated, network-denied) one. The acceptance line 'no real-gh call in both arms' needs a case-level helper plugin that both arms load, which the eval runner does not offer today; until then forge-read, no-merge-recorded and no-resolve-recorded are scored in the with-plugin arm only.

Not verified

• Decision 2's probe run and the workspace .claude/settings.json env.PATH write were not done. The plugin-evals docs say a run loads no scaffold-written .claude/ directory, so the decision's fallback (arm: with-only on the call-record graders) was applied directly. A host probe run would confirm this.
• install.sh's own comments (install.sh:10, :122) say the eval sandbox holds .git read-only for a run's Bash, so calls.jsonl may stay empty in a run. For that reason review moved the three call-record graders back to main's logic (they pass on an empty record) and kept only arm: with-only; see the review threads.
• GH -forge.test.sh was not run directly here: the lane offers no wrapper for it, and check-affected's output doesn't name which tests it ran. Left to the PR's CI.
• The paid with/without kept-trace audit of the four cases (claude plugin eval) needs a host run; per the recorded decision it stays on ＃7148.
• tidy skipped: the change touches 19 files, over the 10-file tidy cutoff.

Fix

• plugins/source-control/evals/fixtures/GH -forge/install.sh: new --dir <checkout> and --extra-pr <number> <head> <title> options (extra heads pushed and given upstreams); sets alias.remote-evalforge to the helper's absolute path; header comment says the no-plugin arm has no stand-in
• plugins/source-control/evals/fixtures/GH -forge/gh: serves extra PRs from extra-prs in pr view/list/search/diff/checks/merge/review; an EXIT trap records each call that exited 0 and wasn't already recorded as {"command":"read",...} in calls.jsonl
• plugins/source-control/evals/GH -forge.test.sh: asserts a served read is recorded; the pr create assertion reads the last record; new block covers --dir, --extra-pr (list, view, merge record of PR 62) and git fetch through the alias with the helper off PATH
• plugins/source-control/evals/babysit-prs-safe-tier-never-resolves-or-merges/graders/{forge-read,no-merge-recorded,no-resolve-recorded}.md: main's matching logic unchanged; only arm: with-only added, because the no-plugin arm has no stand-in and its gh is the real one. Samples are main's.
• plugins/source-control/evals/babysit-loop-{demotion-reported-not-written,forged-checkout-settings-not-trusted,missing-bootstrap-names-option}/scaffold.sh: create two head branches, return to main, run install.sh for the prompt's two PRs (--dir app in the demotion case)
• plugins/source-control/evals/babysit-loop-{demotion-reported-not-written,forged-checkout-settings-not-trusted,missing-bootstrap-names-option}/prompt.md: plugins: ["../..", "GH -forge-path"]
• plugins/source-control/evals/babysit-loop-{forged-checkout-settings-not-trusted,missing-bootstrap-names-option}/GH -forge-path/.claude-plugin/plugin.json: new, copy of the babysit-prs case's hook plugin
• plugins/source-control/evals/babysit-loop-demotion-reported-not-written/GH -forge-path/.claude-plugin/plugin.json: new, the same hook but resolving the git dir of ./app
• .changes/source-control/test-babysit-evals-offline-GH -3c81f0a4.md: bump none fragment

Verification

The runner ran each check from the base copy against 374cb3f:

• scripts/check-changelog-fragments.sh --check: passed

The run reports:

• read https: //code.claude.com/docs/en/plugin-evals: the without-arm runs with no plugin at all; a run reads no .claude/ directory in the workspace, even one a scaffold_script wrote; case env keys must match EVAL_*; arm: with-only leaves a grader out of both arms' scores and reports it as a with-arm
• read https: //raw.githubusercontent.com/git/git/master/transport-helper.c: get_helper() runs the helper as a git command (git_cmd = 1) with args remote-<name> <remote> <url> and sets GIT_DIR
• read https: //git-scm.com/docs/git-config: alias.* with a leading ! runs a shell command from the repository's top-level directory and appends the extra arguments
• /home/runner/work/claude-code-plugins/claude-code-plugins/.lane/checks/bin/check-changelog-fragments: all 6 fragments valid (ran again after the review fix)
• /home/runner/work/claude-code-plugins/claude-code-plugins/.lane/checks/bin/check-affected: 12 passed, 6 skipped over 19 changed files (ran again after the review fix)
• fresh-context review subagent over the whole diff: no confirmed bugs; it traced every sample through the regexes and the existing GH -forge.test.sh assertions against the read recording; its note that extra heads had no upstream is fixed
• The lane run: https://github.com/melodic-software/claude-code-plugins/actions/runs/41250059645

Left out: the items under Not verified.

Related

#11289

[github-actions]

claude-security-review: 3 inline finding(s) on 374cb3f282db54210048285e0a9708502f500629; the summary is in this lane's pull request comment.

claude-lane: claude-security-review

[github-actions]

claude-security-review reviewed 2b1f1acf3bb6192e005f926adb13b6dc3a88aa5b: read 2 of 2 changed file(s); 0 finding(s), 0 as inline comments.

Security review (incremental, e851cf6..2b1f1ac)

No security issues found.

I read every hunk of the incremental diff. Two files changed since the last review:

• plugins/source-control/evals/babysit-prs-safe-tier-never-resolves-or-merges/graders/forge-read.md: the eval grader's arm: changed from both to with-only. Its input_match regex is unchanged. This only decides which eval arm scores the grader. It doesn't touch a trust control, token scope, merge hold, workflow, manifest or standing instruction.
• .changes/source-control/test-babysit-evals-offline-gh-3c81f0a4.md: the changelog fragment's prose now says forge-read is with-only as well. bump: none is unchanged.

Neither file matches a trusted-path glob. The change has no path to a leak, a compromise or a charge. The PR context file has no embedded instructions aimed at this lane.

Spend: none beyond normal eval runs.

critical-findings: 0

[github-actions]

claude-review: 1 inline finding(s) on 374cb3f282db54210048285e0a9708502f500629; the summary is in this lane's pull request comment.

claude-lane: claude-review

[github-actions]

claude-review reviewed 2b1f1acf3bb6192e005f926adb13b6dc3a88aa5b: read 2 of 2 changed file(s); 0 finding(s), 0 as inline comments.

Incremental review (e851cf6 → 2b1f1ac)

I read both changed files in .claude-lane/incremental.diff in full, and used the full PR diff for context.

plugins/source-control/evals/babysit-prs-safe-tier-never-resolves-or-merges/graders/forge-read.md: The grader's arm changes from both to with-only. The grader is a tool_used regex over the Bash trace. In the no-plugin arm, gh is the real, unauthenticated binary with no network, so a typed gh pr view would match the trace without reading any forge. Scoring that arm on this grader would report a read that never reached a forge. with-only fixes this: the grader becomes a with-arm indicator, the same as no-merge-recorded and no-resolve-recorded. The trace-based no-merge-command and no-resolve-command stay arm: both. That is still right, because a typed merge or resolve is a violation in either arm whether it succeeds or not.

.changes/source-control/test-babysit-evals-offline-gh-3c81f0a4.md: The fragment now says all three graders are with-only and explains why. It matches the three grader files at head. bump: none fits a change that only touches evals.

Out of date in the PR description (not a code finding): The PR body's Summary and Fix sections still say forge-read is "now a regex over calls.jsonl" and that the three record graders "can no longer pass on an empty record". At head, forge-read is a trace-based tool_used grader, and the changelog fragment says the record graders still pass on an empty record. The fragment and the code agree with each other. Only the description is out of date, so a reader should trust the fragment over the description.

The PR context has no closing issue (it uses Refs: #11289, not a closing keyword), so this review has no acceptance criteria to check. This PR changes no plugin manifests, and no settled-threads file was given.

No findings.

[melodic-automation-lanes]

Unattended merge: native auto-merge leaves this pull request unarmed at head 2b1f1acf3bb6192e005f926adb13b6dc3a88aa5b. The lanes clear this reason on their own; the next scheduled run judges it again.

pr-merge's gate refuses its verifier verdict (verify:fail)

[melodic-automation-lanes]

pr-unblock / address-feedback finished round 1 of max 4 on 374cb3f.

[melodic-automation-lanes]

The pr-unblock lane stopped address-feedback on this pull request: the lane left a pr-verify note open (uncertain, deferred or answered incorrect, or still open after the round cap and one fresh fixer round); pr-verify judges the next head afresh. The lane sets no label: the run ends in a failing pr-unblock/address-feedback status on the head the lane gated.

• Reason: feedback-verifier-open
• Head: 2b1f1acf3bb6192e005f926adb13b6dc3a88aa5b
• Run: https://github.com/melodic-software/claude-code-plugins/actions/runs/41266129156

When a retry on this head can change the outcome, the lane runs one on its own, up to its per-head cap, and edits this comment if it stops again; a run that does not stop clears the status. When no retry can (a review thread the lane does not take or settle, or a failing check after the retries are spent), a person resolves the threads or gets the checks green, then dispatches the workflow below: that run finds the cause gone and clears the status. On a pull request the lanes App opened, the dispatchers start that run once per stop; on any other, a person does. A new commit to the branch starts clean.

To run it again on this head now, dispatch the pr-unblock workflow:

gh workflow run pr-unblock.yml -R melodic-software/claude-code-plugins -f pr-number=11752 -f activity=address-feedback

[melodic-automation-lanes]

pr-unblock / address-feedback finished round 2 of max 4 on ab43e18.

[github-actions]

claude-review: 1 inline finding(s) on ab43e1875d301715a6e74f01f2574d77623903cf; the summary is in this lane's pull request comment.

claude-lane: claude-review

[melodic-automation-lanes]

pr-unblock / address-feedback finished round 3 of max 4 on e851cf6.

[melodic-automation-lanes]

pr-unblock / address-feedback finished round 4 of max 4 on 2b1f1ac.

[melodic-automation-lanes]

The pr-unblock lane stopped on this pull request and needs a person: the lane has addressed review threads on this pull request as many rounds as its cap allows, and an open thread is a security finding rated CRITICAL or unrated, an intent finding, or a person holds the pull request, so a person decides.

• Reason: feedback-round-cap
• Resume key: 2b1f1acf3bb6192e005f926adb13b6dc3a88aa5b (the head the lane stopped on)

To resume, deal with the reason above, remove the needs-human label (and do-not-merge, if set), then dispatch the pr-unblock workflow with pr-number=11752 and activity=address-feedback:

gh workflow run pr-unblock.yml -R melodic-software/claude-code-plugins -f pr-number=11752 -f activity=address-feedback

[kyle-sexton]

Hold: replaced by an owner-login PR because pr-verify left no verdict on this head after the needs-human stop cleared. Releases when: the replacement merges and this PR is closed.

[kyle-sexton]

Superseded by #11856, which merged with the identical diff (patch-id 6a9310d11b).