perf(agents): pilot the effort replay on one merged PR

Parent: #5682 (slice 1 of 3). Runs first; slice 2 waits on it.
Runs in: HOST. Replays agents at a base commit with model runs on the operator subscription (or the eval lane's MAINTENANCE role); the issue lane cannot run agents at a base commit; the host runner works it.

Problem

#5682 confirms each pinned agent's effort by eval instead of judgment. The 2026-10-02 scope says: start with a one-PR pilot, size the sweep only after it, show each paid run's cost estimate before it starts. The lane bounced the whole issue on 2026-10-25 because it cannot replay agents (run 40189315978). The Haiku 5.5 arm moved to #6901 (2026-10-13 decision) and is out of scope here.

Fix

Pick one recently merged PR that went through /implementation:implement-dispatch. At its base commit, replay:

implementation:implementer at medium and at high effort;
implementation:phase-verifier at medium and at high;
implementation:implementer on Sonnet 5.5 and on Opus 5.5 (same effort).

Grade each arm by the repo's own tests and CI, the phase-verifier verdict, and the blind spot check the scope names (who performs it follows the owner's answer to the pending question on agent adjudication; until then it stays as written). Record tokens, wall time and rework per arm. Show the cost estimate before each run (subscription or prepaid spend; Q37, Q46, Q91, Q110: not a stop).

Done when

A comment on review: effort eval sweep for pinned agents, including Haiku 5.5 when it ships #5682 names the replayed PR and base commit, and gives per arm: pass or fail, verifier verdict, tokens, wall time, rework
The comment shows the cost estimate made before each run
The comment proposes the sweep size for slice 2 (PR count and arms), with the pilot data behind it

Split by the backlog HITL session (2026-10-25, owner rule Q93: split large work before a run picks it up). This was generated by AI.

Proposed work class: none

Size: Medium
Complexity: Medium
Uncertainty: Unknown
Flags: none
Blocked by: none

Understood: slice 1 of #5682. Pick one recently merged PR that went through implement-dispatch, replay the implementer and phase-verifier at medium and high effort, plus the implementer on Sonnet 5.5 and Opus 5.5, then post per-arm results, cost estimates and a proposed slice 2 sweep size on #5682.

The replay needs a host runner because the issue lane cannot run agents at a base commit. That does not block the issue. Nothing else is missing. The issue states its scope and spend rules, and the Haiku arm is out of scope (#6901). I found no likely duplicates.

_{Automated triage. A maintainer reviews it.}

Not admitted to the issue lane: Uncertainty unknown is not one of known, knowable. Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).

The latest triage verdict holds nothing: admission moved the issue to status: needs-decision; its comment says why.

Re-triage. No owner stop applies (docs/conventions/loop-lane/README.md:316-345). The replay spends model usage under the Claude subscriptions or API credits, which the owner's rule names as not cost (:337-339). The only admission refusal was "Uncertainty unknown". The rules below make the pilot's inputs known, so nothing is left to discover before it starts.

Which PR gets replayed?
Decided: the most recently merged PR on main whose body or commits show /implementation:implement-dispatch ran, with at least one implementation:phase-verifier verdict in its record. Replay at its merge base. If none was merged in the last 30 days, take the newest such PR of any age and say so. Basis: the body's own scope, "one recently merged PR that went through /implementation:implement-dispatch".
Who performs the blind spot check?
Decided: as the body says, it stays as the 2026-10-02 scope wrote it until the pending adjudication question on review: effort eval sweep for pinned agents, including Haiku 5.5 when it ships #5682 is answered. That question does not block the pilot.

The work needs a host, since the issue lane cannot replay agents at a base commit, so host-run is added.

Acceptance criteria:

A comment on review: effort eval sweep for pinned agents, including Haiku 5.5 when it ships #5682 names the replayed PR and its base commit, and gives for each of the six arms (implementer at medium and at high effort, phase-verifier at medium and at high effort, implementer on Sonnet 5.5 and on Opus 5.5): pass or fail against the repo's tests and CI, the phase-verifier verdict, tokens, wall time and rework.
The comment shows the cost estimate made before each run.
The comment proposes the sweep size for slice 2 (PR count and arms) and cites the pilot data behind it.
No Haiku 5.5 arm was run (multi-agent: paired model-routing eval for retrieval, review and mechanical-edit roles #6901 owns it).

Host run. I ran the one-PR effort-replay pilot from #5682 against #8786 at its base commit: five replay arms, then three phase-verifier passes over the implementer outputs.

Posted on #10214 and as the slice 1 result on #5682.

Replayed PR

PR: #8786, fix(ci): reopen a monitor's closed issue for its key instead of filing a recurrence. Merged 2026-10-21. 13 files, +608/-141.
Base: f2aa201decd3dd248ffe653890c3ca36a5b43323, the first parent of the squash commit.
Reference: squash commit 933f0b279792a624f6e06adb4a8b509310b2222b.
Why this PR: it is the newest merged PR whose record shows /implementation:implement-dispatch actually ran. Its body has "run main-side, outside the worker" and a phase-verifier verdict at a named SHA. Newer PRs with a verifier line show no dispatch run:
- feat(app-control): find plugin-shipped control skills in /app-control:drive #9615: neither its body nor its 29 commits show one.
- The machine-health series refactor(machine-health): derive the elevation banner coverage from the catalog #8910 to feat(machine-health): add the short report and monthly schedule #9243, and feat(harness-ops): add a cost section to the morning brief #8790: their commit bodies do not show one either.
- Several PRs mention implement-dispatch only because they edit it (docs(implementation): align implement-dispatch with the verifier's test-edit FAIL rule #8781, feat(implementation): pick each spawn's model and effort from stated complexity in implement-dispatch #8787).

Setup

Version and login: Claude Code 2.1.314. Every run used claude -p on the subscription login. apiKeySource was none in every run's init event.
Agents: the implementation:implementer and implementation:phase-verifier bodies came from current origin/main, so the pins under test are today's. They ran through --agents with --model and --effort set on the command line.
- Each run's init event confirms the model and the tool set.
- The init event does not report effort. Thinking tokens show it did take effect: high used more than medium in both pairs. Implementer: 10,587 vs 6,308. Verifier on the reference: 3,697 vs 2,252. No effortLevel is set in user settings.
Arm command: claude -p <brief> --agents agents.json --agent replay-implementer|replay-verifier --model opus|sonnet --effort medium|high --permission-mode auto --disallowedTools "Bash(git push:*)" "Bash(git commit:*)" "Bash(git add:*)" "Bash(gh:*)" Agent --output-format stream-json --verbose, under timeout 2700.
Arms: five runs cover the six arms. Opus 5.5 at the pinned medium serves as both the implementer's medium-effort arm and its Opus arm.
Isolation:
- Each implementer arm got its own detached worktree at the base.
- Each brief declared commit authority: orchestrator.
- The command line denied git push, git commit, git add, gh and the Agent tool.
- No arm ran a git write or a gh command. The gh lines in the logs are script text the arms edited, not commands they ran.
- I made the local commits afterwards. Nothing was pushed.
Inputs, the same for every arm:
- One implementer brief, one verifier brief and nine acceptance criteria (C1 to C9).
- All three are reconstructed from fix(ci): reopen a monitor's closed issue for its key instead of filing a recurrence #8786's Fix and Verification sections. The original dispatch brief is not public.
How each arm was graded:
- Own tests: node --test over both .github/scripts/maintenance-* directories, plus bash scripts/upstream-drift-issues.test.sh.
- Holdout: copy the arm's changes into a fresh worktree at the base, then git checkout <reference> -- the five test files and run the same commands. On the reference itself they pass 113/113 and 123/0.
- Phase verifier: pinned Opus at high.
- scripts/check-affected.sh --tests: run on the arm's committed head.
- CI: not run, because nothing was pushed.
Rework: the number of items a fix round would have to address at first return, meaning verifier FAIL criteria plus failing holdout checks.
Parallel runs: all five arms ran at the same time, so the wall times include contention.

Cost estimates, written before each run

These were written to a timestamped file before any arm launched, 2026-10-30T03:45:45Z. Basis: judgment, scaled from a probe run.

Arm	Estimate before the run	Actual
implementer Opus medium	2-5M tokens, 0.15-0.3M output	6.30M, 66k output
implementer Opus high	3-7M tokens, 0.2-0.5M output	7.32M, 48k output
implementer Sonnet medium	2-6M tokens, 0.15-0.4M output	2.33M, 23k output
phase-verifier Opus medium	0.5-1.5M tokens	0.65M
phase-verifier Opus high	0.7-2M tokens	0.86M
3 verifier passes over the arms (written 03:58Z)	0.8-1M tokens each	0.43M / 0.72M / 0.69M

Token totals are input, output, cache read and cache creation together, nearly all of it cache reads. All usage came from the subscription.

Results per arm

Arm	Own tests	Holdout failures (node tests / drift assertions)	Phase verifier (Opus high)	Tokens	Wall time	Turns	Rework
Implementer, Opus 5.5, medium (pinned)	pass: 100/100, drift 108/0	14 / 17	PASS	6.30M	10.8 min	45	14 tests and 17 assertions; 0 verifier
Implementer, Opus 5.5, high	pass: 110/110, drift 107/0	13 / 15	PASS	7.32M	10.1 min	47	13 tests and 15 assertions; 0 verifier
Implementer, Sonnet 5.5, medium	pass: 100/100, drift 94/0	13 / 17	FAIL (C4)	2.33M	2.9 min	26	13 tests and 17 assertions; 1 verifier
Phase-verifier, Opus, medium, on the reference diff	ran the tests: 113/113, 123/0	n/a	FAIL (C4), 8 of 9 pass	0.65M	2.9 min	13	n/a
Phase-verifier, Opus, high (pinned), on the reference diff	ran the tests: 113/113, 123/0	n/a	FAIL (C4), 8 of 9 pass	0.86M	2.7 min	16	n/a

What the columns add up to:

Scope: every arm stayed inside the file fence. C6 (no "Recurrence of" or "Regression of" left) passed for all of them.
check-affected.sh --tests: it reported 1 failed suite for every arm and for the reference. The failure is the same environment gap each time: htmlhint is missing because these worktrees have no npm ci. It is not caused by any arm's code.
Holdout failures: they mostly measure what the reconstructed brief left out, not arm quality. The reference also re-triages labels on reopen, mutes not_planned and duplicate closes in the drift filer, and skips other lanes' issues. My nine criteria do not state those behaviours. All three arms miss them by about the same amount: 13 or 14 node tests and 15 to 17 drift assertions.

Findings

Implementer, medium vs high effort: no quality difference on this PR. Both passed their own tests and the verifier. High used 16% more tokens and finished in about the same wall time.
Implementer, Sonnet vs Opus:
- Sonnet used 37% of the Opus-medium tokens and 27% of the wall time.
- It failed the verifier on C4: the lane sweep has no closed-unchanged skip.
- That gap needs care in reading:
  - The reference PR has the same gap.
  - So does the Opus-high arm, which the verifier passed: neither of its lane files has a closed_at comparison.
  - The Opus-medium arm added a guard, !(now > closed_at), but its caller passes now: new Date(). The guard compares the run time, not the observation time, so it never skips.
Verifier verdicts are not stable on C4:
- Medium and high agreed on the reference diff: both FAILed C4.
- The pinned high verifier then gave PASS on the Opus-high arm, which lacks the same guard, and FAIL on the Sonnet arm.
- On this one PR, verdict variance across diffs was larger than the effort difference.
Candidate defect in merged code (UNVERIFIED): both verifiers report the same gap in fix(ci): reopen a monitor's closed issue for its key instead of filing a recurrence #8786's lane sweep.
- The gap: planFiling reopens any closed, non-muted match with no comparison to closed_at.
- The effect: a lane failure that is fixed and closed inside the 24-hour audit window would be reopened by the next sweep on evidence from before the close.
- Unconfirmed: whether the original brief intended this is unknown. I did not file it, because this row's scope is the pilot record only.
Brief reconstruction is the main source of noise. The real dispatch brief and criteria are not on GitHub.
Few PRs qualify: only 1 of the newest ~13 merged PRs with a verifier line also shows implement-dispatch ran and has Linux-runnable deterministic tests.

Proposed sweep size for slice 2

PRs: 5 merged PRs that show a dispatch run and have Linux-runnable tests. Where the session records still hold the original dispatch brief and criteria, take them from there. Finding 5 is why.
Implementer arms:
- Opus medium (pinned) and Sonnet medium on all 5.
- Opus high on 2 of the 5 only. Finding 1: no gain at +16% tokens.
Verifier arms: medium and high on the reference diff and on every arm diff, with each verdict run twice to measure repeat variance. Finding 3.
Size: about 80 runs: per PR, 2.4 implementer runs plus (1 + 2.4) diffs × 2 efforts × 2 repeats = 13.6 verifier runs. Roughly 5 × (6.3 + 2.3 + 0.4 × 7.3 + 13.6 × 0.7) M ≈ 105M tokens on the subscription. Basis: the per-arm actuals above.
Blind human spot check:
- Still pending; the scope as written assigns it to a human.
- I prepared a blinded packet: four diffs labelled A to D (three arms plus the reference) and the criteria.
- The key is kept separately.
- Both are under /home/kyle/worktrees/host-run-10214/.

Criteria status

Per-arm results: done except CI. CI was not run because nothing was pushed. check-affected is red on an environment gap, and the reference shows the same failure.
Cost estimates made before each run: done.
Sweep proposal: done.
No Haiku 5.5 arm was run (multi-agent: paired model-routing eval for retrieval, review and mechanical-edit roles #6901 owns it).
Next: slice 2 can be picked up.

Target labels: add agent-ready; remove host-run.

The work-resolve-issue lane did not start this issue (probe-first). Its Uncertainty says only an experiment answers it: run a probe or spike first, then set Uncertainty to Known.

Reason: design

Run: https://github.com/melodic-software/claude-code-plugins/actions/runs/41007198312

To retry, remove needs-human and add agent-ready again.

Closing as completed. The host run met each acceptance item: the slice-1 result on #5682 names #8786 and base f2aa201. It gives per-arm pass or fail against the repo's tests (CI was not run, since nothing was pushed; the run used holdout tests and check-affected instead), verifier verdicts, tokens, wall time and rework. It shows the cost estimates written before each run and proposes the slice-2 size (5 PRs, about 80 runs). No Haiku arm was run. The later lane probe-first stop came from the issue's stale Uncertainty value, not from open work.

Navigation Menu

Repository navigation

perf(agents): pilot the effort replay on one merged PR #10214

Problem

Fix

Done when

Replayed PR

Setup

Cost estimates, written before each run

Results per arm

Findings

Proposed sweep size for slice 2

Criteria status

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

perf(agents): pilot the effort replay on one merged PR #10214

Description

Problem

Fix

Done when

Activity

github-actions commented on Oct 26, 2026

melodic-automation-lanes commented on Oct 26, 2026

melodic-automation-lanes commented on Oct 26, 2026

kyle-sexton commented on Oct 28, 2026

kyle-sexton commented on Oct 30, 2026

Replayed PR

Setup

Cost estimates, written before each run

Results per arm

Findings

Proposed sweep size for slice 2

Criteria status

melodic-automation-lanes commented on Oct 30, 2026

kyle-sexton commented on Oct 30, 2026

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions