review: effort eval sweep for pinned agents, including Haiku 5.5 when it ships

Confirm each agent's effort pin by eval rather than judgment, and test Haiku 5.5 for the three mechanical agents and the loop-lane fast tier once it ships. Nothing moves to Haiku until this sweep says so.

Follow-on from the Sonnet 5.5 prompting-guide work (plan: docs/topics/sonnet-5-5-prompting-digest/PLAN.md on branch feat/sonnet-5-5-prompting-digest; interview answers Q10, Q48).

Grading: use the repo's own tests and CI and the phase-verifier verdict. A spot check of arm output tries a deterministic grader first; otherwise three blind agents label it, given the rubric and not the arm names or verdicts. A label is accepted only on unanimous agreement, and any disagreement goes to the owner as one intent question.

Done when

A replay sweep of merged PRs at medium and at high effort for each pinned agent is posted here with per-agent results.
Each agent's effort pin matches the sweep, or a PR that changes it is linked here.

feat(playbooks): apply the Sonnet 5.5 prompting guide across the mark…

Scope addition from the Spending Your Effort session, approved by its user on 2026-10-02.

Replay sweep. Re-run recently merged PRs at their base commits:

the implementer and phase-verifier at medium vs high effort
for the implementer, a Sonnet 5.5 vs Opus 5.5 arm

Grading. Use the repo's own tests and CI, the phase-verifier verdict, and a blind human spot check. Record tokens, wall time and rework per arm.

Order and cost. Start with a one-PR pilot, and size the sweep only after the pilot. Show each paid run's cost estimate before it starts.

This sits beside the existing scope: the effort eval sweep, and Haiku 5.5 for the three mechanical agents and the loop-lane fast tier once it ships. The effort pins it would revisit are recorded in #5767.

feat(playbooks): apply the Sonnet 5.5 prompting guide across the mark…

feat(effort): make effort an explicit, recorded choice across lanes, …

This was generated by AI during triage.

Agent Brief

Type: Task
Summary: Confirm each pinned agent's effort level by eval rather than judgment, run the replay sweep from the scope comment, and test Haiku 5.5 for the three mechanical agents and the loop-lane fast tier once it is available.

Current behavior:
Every agent pins its effort (11 high, 4 medium) by judgment and user decision (#5767; #5685 ledger rows Q10 and Q48). Nothing moves to Haiku until this sweep says so.

Desired behavior:

A one-PR pilot first: replay a recently merged PR at its base commit, with the implementer and phase-verifier at medium vs high, and for the implementer a Sonnet 5.5 vs Opus 5.5 arm.
Grading per arm from the repo's own tests and CI, the phase-verifier verdict, and a blind human spot check. Record tokens, wall time and rework per arm.
Size the full sweep only after the pilot. Show each paid run's cost estimate before it starts.
When Haiku 5.5 is available, run the same comparison for the three mechanical agents and the loop-lane fast tier.
A recorded recommendation per pin (keep or change), with the arm data behind it.

Acceptance criteria:

The pilot's per-arm results (pass/fail, verifier verdict, tokens, wall time, rework) are recorded.
Each paid run had its cost estimate shown and approved before it started.
Each effort pin listed in feat(playbooks): apply the Sonnet 5.5 prompting guide across the marketplace #5767 has a keep-or-change recommendation tied to sweep data.

Out of scope:

Changing any pin or moving any agent to Haiku inside this item. That follows as its own change once the data is in.
Gate-based verification design (verification: gate-based verification across Sonnet 5.5, Opus 5.5 and Fable 5.1 #5678). This sweep supplies evidence for its review-on-Sonnet question, but the two are separate.

Routing: human-gated, needs decision

Each paid run needs an approved cost estimate (from the scope comment), and the blind spot check needs a human. An unattended lane cannot do either. The Haiku arm also waits on that model being available (judgment; not checked against the models page this session).

Proposed work class: structural (C4) for the replay arms. They run implementer agents that commit and push at base commits, so this is not read-only. A pilot that only reads recorded results would be read-only (C1).
Basis: judgment, from the scope comment's "Show each paid run's cost estimate before it starts" and the C1 definition (no repository mutation) in the autonomy work-class reference.

Priority: medium. No deadline, and nothing is blocked on it.

Related, not a cluster: #5678 (review-on-Sonnet question). No edge added.

Stale reference: the body's PLAN.md pointer on feat/sonnet-5-5-prompting-digest. That branch merged as #5767, and the plan is on #5685.

Decision recorded. The owner approved applying independently verified recommendations (high-risk set, reviewed) on 2026-10-13.

Where does the Haiku 5.5 arm live? → Move it to multi-agent: paired model-routing eval for retrieval, review and mechanical-edit roles #6901, which owns every Haiku 5.5 role test; review: effort eval sweep for pinned agents, including Haiku 5.5 when it ships #5682 keeps the effort pins and the replay sweep
Basis: multi-agent: paired model-routing eval for retrieval, review and mechanical-edit roles #6901 body ('Haiku 5.5 may take no role until that role passes this eval'); https://platform.claude.com/docs/en/about-claude/models/overview.md fetched 2026-10-12 (lists Haiku 5.5)

Confirm each agent's effort pin by eval (a replay sweep of merged PRs at medium vs high, plus a Sonnet vs Opus implementer arm) and test Haiku 5.5 for mechanical roles. Haiku 5.5 is now listed on Anthropic's models overview, and #6901 already plans a paired Haiku 5.5 routing eval for the same roles.

Verification notes:

multi-agent: paired model-routing eval for retrieval, review and mechanical-edit roles #6901's roles are retrieval, review and mechanical-edit. review: effort eval sweep for pinned agents, including Haiku 5.5 when it ships #5682's Haiku arm also names 'the loop-lane fast tier'. When the arm moves, add that tier to multi-agent: paired model-routing eval for retrieval, review and mechanical-edit roles #6901's scope explicitly so it is not dropped.

Verified by an independent fresh-context check (confirmed, risk high): gh issue view 5682 (scope comment 2026-10-02: pilot first, show each paid run's cost estimate; triage); gh issue view 6901 (OPEN, created 2026-10-11; 'Haiku 5.5 may take no role until that role passes this eval'; arms Haiku 5.5 high/xhigh, advisor, Sonnet, Opus); https://platform.claude.com/docs/en/about-claude/models/overview.md fetched 2026-10-12: Claude Haiku 5.5 listed (claude-haiku-5-5)

Admission to an agent lane stays with the work-class rules and owner ratification.

Parked (owner decision 2026-10-21): no eval spend until #7719 resumes hardening. This stays open and resumes then.

Decision recorded.

This was generated by AI.

The owner decided this in the Backlog HITL session on 2026-10-26 (ledger row Q120 in .work/backlog-to-zero/interview-checklist.md, a local, untracked file), answering: "go with whatever the facts/data/metrics + /discovery:research says and ensure we're aligned with SOLID, DRY/SSOT principles".

Rule (extends Q114): adjudication first tries a deterministic grader. Otherwise three blind agents label, given the rubric and not the old labels or verdicts. A label is accepted only on unanimous agreement, and any disagreement goes to the owner as one intent question.

This replaces the "blind human spot check" in the scope comment's Grading paragraph and the brief's grading step. The issue body now states the grading rule; the rest is unchanged.

needs-human is removed: the spot check was the human step, and the replay sweep itself is eval-host work, which the host runner takes as agent work.

Host run. I ran the one-PR effort-replay pilot from #5682 against #8786 at its base commit: five replay arms, then three phase-verifier passes over the implementer outputs.

Posted on #10214 and as the slice 1 result on #5682.

Replayed PR

PR: #8786, fix(ci): reopen a monitor's closed issue for its key instead of filing a recurrence. Merged 2026-10-21. 13 files, +608/-141.
Base: f2aa201decd3dd248ffe653890c3ca36a5b43323, the first parent of the squash commit.
Reference: squash commit 933f0b279792a624f6e06adb4a8b509310b2222b.
Why this PR: it is the newest merged PR whose record shows /implementation:implement-dispatch actually ran. Its body has "run main-side, outside the worker" and a phase-verifier verdict at a named SHA. Newer PRs with a verifier line show no dispatch run:
- feat(app-control): find plugin-shipped control skills in /app-control:drive #9615: neither its body nor its 29 commits show one.
- The machine-health series refactor(machine-health): derive the elevation banner coverage from the catalog #8910 to feat(machine-health): add the short report and monthly schedule #9243, and feat(harness-ops): add a cost section to the morning brief #8790: their commit bodies do not show one either.
- Several PRs mention implement-dispatch only because they edit it (docs(implementation): align implement-dispatch with the verifier's test-edit FAIL rule #8781, feat(implementation): pick each spawn's model and effort from stated complexity in implement-dispatch #8787).

Setup

Version and login: Claude Code 2.1.314. Every run used claude -p on the subscription login. apiKeySource was none in every run's init event.
Agents: the implementation:implementer and implementation:phase-verifier bodies came from current origin/main, so the pins under test are today's. They ran through --agents with --model and --effort set on the command line.
- Each run's init event confirms the model and the tool set.
- The init event does not report effort. Thinking tokens show it did take effect: high used more than medium in both pairs. Implementer: 10,587 vs 6,308. Verifier on the reference: 3,697 vs 2,252. No effortLevel is set in user settings.
Arm command: claude -p <brief> --agents agents.json --agent replay-implementer|replay-verifier --model opus|sonnet --effort medium|high --permission-mode auto --disallowedTools "Bash(git push:*)" "Bash(git commit:*)" "Bash(git add:*)" "Bash(gh:*)" Agent --output-format stream-json --verbose, under timeout 2700.
Arms: five runs cover the six arms. Opus 5.5 at the pinned medium serves as both the implementer's medium-effort arm and its Opus arm.
Isolation:
- Each implementer arm got its own detached worktree at the base.
- Each brief declared commit authority: orchestrator.
- The command line denied git push, git commit, git add, gh and the Agent tool.
- No arm ran a git write or a gh command. The gh lines in the logs are script text the arms edited, not commands they ran.
- I made the local commits afterwards. Nothing was pushed.
Inputs, the same for every arm:
- One implementer brief, one verifier brief and nine acceptance criteria (C1 to C9).
- All three are reconstructed from fix(ci): reopen a monitor's closed issue for its key instead of filing a recurrence #8786's Fix and Verification sections. The original dispatch brief is not public.
How each arm was graded:
- Own tests: node --test over both .github/scripts/maintenance-* directories, plus bash scripts/upstream-drift-issues.test.sh.
- Holdout: copy the arm's changes into a fresh worktree at the base, then git checkout <reference> -- the five test files and run the same commands. On the reference itself they pass 113/113 and 123/0.
- Phase verifier: pinned Opus at high.
- scripts/check-affected.sh --tests: run on the arm's committed head.
- CI: not run, because nothing was pushed.
Rework: the number of items a fix round would have to address at first return, meaning verifier FAIL criteria plus failing holdout checks.
Parallel runs: all five arms ran at the same time, so the wall times include contention.

Cost estimates, written before each run

These were written to a timestamped file before any arm launched, 2026-10-30T03:45:45Z. Basis: judgment, scaled from a probe run.

Arm	Estimate before the run	Actual
implementer Opus medium	2-5M tokens, 0.15-0.3M output	6.30M, 66k output
implementer Opus high	3-7M tokens, 0.2-0.5M output	7.32M, 48k output
implementer Sonnet medium	2-6M tokens, 0.15-0.4M output	2.33M, 23k output
phase-verifier Opus medium	0.5-1.5M tokens	0.65M
phase-verifier Opus high	0.7-2M tokens	0.86M
3 verifier passes over the arms (written 03:58Z)	0.8-1M tokens each	0.43M / 0.72M / 0.69M

Token totals are input, output, cache read and cache creation together, nearly all of it cache reads. All usage came from the subscription.

Results per arm

Arm	Own tests	Holdout failures (node tests / drift assertions)	Phase verifier (Opus high)	Tokens	Wall time	Turns	Rework
Implementer, Opus 5.5, medium (pinned)	pass: 100/100, drift 108/0	14 / 17	PASS	6.30M	10.8 min	45	14 tests and 17 assertions; 0 verifier
Implementer, Opus 5.5, high	pass: 110/110, drift 107/0	13 / 15	PASS	7.32M	10.1 min	47	13 tests and 15 assertions; 0 verifier
Implementer, Sonnet 5.5, medium	pass: 100/100, drift 94/0	13 / 17	FAIL (C4)	2.33M	2.9 min	26	13 tests and 17 assertions; 1 verifier
Phase-verifier, Opus, medium, on the reference diff	ran the tests: 113/113, 123/0	n/a	FAIL (C4), 8 of 9 pass	0.65M	2.9 min	13	n/a
Phase-verifier, Opus, high (pinned), on the reference diff	ran the tests: 113/113, 123/0	n/a	FAIL (C4), 8 of 9 pass	0.86M	2.7 min	16	n/a

What the columns add up to:

Scope: every arm stayed inside the file fence. C6 (no "Recurrence of" or "Regression of" left) passed for all of them.
check-affected.sh --tests: it reported 1 failed suite for every arm and for the reference. The failure is the same environment gap each time: htmlhint is missing because these worktrees have no npm ci. It is not caused by any arm's code.
Holdout failures: they mostly measure what the reconstructed brief left out, not arm quality. The reference also re-triages labels on reopen, mutes not_planned and duplicate closes in the drift filer, and skips other lanes' issues. My nine criteria do not state those behaviours. All three arms miss them by about the same amount: 13 or 14 node tests and 15 to 17 drift assertions.

Findings

Implementer, medium vs high effort: no quality difference on this PR. Both passed their own tests and the verifier. High used 16% more tokens and finished in about the same wall time.
Implementer, Sonnet vs Opus:
- Sonnet used 37% of the Opus-medium tokens and 27% of the wall time.
- It failed the verifier on C4: the lane sweep has no closed-unchanged skip.
- That gap needs care in reading:
  - The reference PR has the same gap.
  - So does the Opus-high arm, which the verifier passed: neither of its lane files has a closed_at comparison.
  - The Opus-medium arm added a guard, !(now > closed_at), but its caller passes now: new Date(). The guard compares the run time, not the observation time, so it never skips.
Verifier verdicts are not stable on C4:
- Medium and high agreed on the reference diff: both FAILed C4.
- The pinned high verifier then gave PASS on the Opus-high arm, which lacks the same guard, and FAIL on the Sonnet arm.
- On this one PR, verdict variance across diffs was larger than the effort difference.
Candidate defect in merged code (UNVERIFIED): both verifiers report the same gap in fix(ci): reopen a monitor's closed issue for its key instead of filing a recurrence #8786's lane sweep.
- The gap: planFiling reopens any closed, non-muted match with no comparison to closed_at.
- The effect: a lane failure that is fixed and closed inside the 24-hour audit window would be reopened by the next sweep on evidence from before the close.
- Unconfirmed: whether the original brief intended this is unknown. I did not file it, because this row's scope is the pilot record only.
Brief reconstruction is the main source of noise. The real dispatch brief and criteria are not on GitHub.
Few PRs qualify: only 1 of the newest ~13 merged PRs with a verifier line also shows implement-dispatch ran and has Linux-runnable deterministic tests.

Proposed sweep size for slice 2

PRs: 5 merged PRs that show a dispatch run and have Linux-runnable tests. Where the session records still hold the original dispatch brief and criteria, take them from there. Finding 5 is why.
Implementer arms:
- Opus medium (pinned) and Sonnet medium on all 5.
- Opus high on 2 of the 5 only. Finding 1: no gain at +16% tokens.
Verifier arms: medium and high on the reference diff and on every arm diff, with each verdict run twice to measure repeat variance. Finding 3.
Size: about 80 runs: per PR, 2.4 implementer runs plus (1 + 2.4) diffs × 2 efforts × 2 repeats = 13.6 verifier runs. Roughly 5 × (6.3 + 2.3 + 0.4 × 7.3 + 13.6 × 0.7) M ≈ 105M tokens on the subscription. Basis: the per-arm actuals above.
Blind human spot check:
- Still pending; the scope as written assigns it to a human.
- I prepared a blinded packet: four diffs labelled A to D (three arms plus the reference) and the criteria.
- The key is kept separately.
- Both are under /home/kyle/worktrees/host-run-10214/.

Criteria status

Per-arm results: done except CI. CI was not run because nothing was pushed. check-affected is red on an environment gap, and the reference shows the same failure.
Cost estimates made before each run: done.
Sweep proposal: done.
No Haiku 5.5 arm was run (multi-agent: paired model-routing eval for retrieval, review and mechanical-edit roles #6901 owns it).
Next: slice 2 can be picked up.

Target labels: add agent-ready; remove host-run.

Not admitted to the issue lane: Size large: every sub-issue is closed, but it is still open, so an item no sub-issue covers or a sub-issue's close keeps it from closing as a finished split (owner Q93). Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).

Effort replay sweep (#10253, slice 2), host run on melo-lap-001. Full per-PR, per-arm results are in #10253 (comment). That comment covers pass rate, verifier verdicts, tokens, wall time and rework, with the cost estimate written before each batch.

What ran:

5 merged PRs.
12 implementer arms: Opus medium and Sonnet medium on all 5, Opus high on 2.
68 phase-verifier runs: medium and high, twice each, on every reference and arm diff.
About 116M subscription tokens.

Keep or change, per effort pin:

Agent	Pin	Recommendation	Basis
`implementation:implementer`	medium	keep	Opus high beat medium on 1 of 2 PRs, at 22-106% more tokens. Holdout was identical. Sonnet medium was mixed at 63% of the tokens.
`implementation:phase-verifier`	high	keep, with a note	On 16 of 17 diffs, the medium and high pairs gave the same verdicts, at 1.8x the tokens for high. Repeat variance within one effort matched the gap between efforts. The one defect needing a cross-file trace was caught by one run at each effort. That is not enough to show medium loses nothing.
the other 18 pins	as pinned	no data	This sweep replays dispatch implementer and verifier runs only. That is the scope the slice-1 pilot proposed (#10253, pilot comment 6515138086). The other pins are discovery, review, planning, user-experience, performance and songwriting agents; no replay here exercised them.

The other 18 pins:

high: discovery:intent-tracer, discovery:research-verifier, discovery:researcher, plugin-quality:auditor, review:architecture-guardian, review:brief-reviewer, review:ci-log-auditor, review:code-reviewer, review:doc-drift-detector, review:security-reviewer, user-experience:evaluator.
medium: discovery:explorer, implementation:scoped-implementer, performance:go-faster-sweeper, planning:plan-reviewer, planning:upload-reader, review:ecosystem-specialist, songwriting:object-writer.

Open: #10253's second done-when asks for a data-tied recommendation for every pin from #5767. This sweep meets that for 2 of the 20 pins now present. Measuring the rest needs replay designs for non-dispatch agents. That is beyond the size the pilot set, so it goes to the owner as a scope question rather than a silent extension.

Slice 2 (#10253) is closed by owner decision (a), 2026-11-01: the 18 pins the sweep did not exercise keep their current values ("keep, no data"), and no slice 3 is filed. Per-pin outcome: implementation:implementer medium keep, implementation:phase-verifier high keep (with a note), the other 18 as pinned. The table is in 5682#issuecomment-6556136118.

Decision recorded. (agent; not an owner decision)

The 18 pinned agents that the slice-2 sweep did not exercise keep their current pins. No slice 3 is filed. This is option (a) of the parked-review question.

Why it is not an owner stop: the standing direction (s2) sends the owner only intent or vision changes, design-altering findings, and HIGH/CRITICAL security or cost risks. Keeping existing effort pins is none of those. It changes nothing, and reverting it is a one-line pin edit per agent.
Basis: slice 2 found no quality gain from high effort on the implementer: the same verifier verdicts on 16 of 17 diffs, at 1.8x the tokens (issuecomment-6556136118). A session also recorded the owner's 11-01 choice as (a). That record is not in the owner's own words, so this decision does not rest on it.
Done-when 2 holds. Done-when 1 holds for the 2 measured pins. For the other 18, this decision replaces measurement with "keep the current pin". If any pinned agent later shows an effort-related quality or cost regression, the case for measuring it reopens as a new issue.

Closing as completed on that basis.

Navigation Menu

Repository navigation

review: effort eval sweep for pinned agents, including Haiku 5.5 when it ships #5682

Done when

Agent Brief

Routing: human-gated, needs decision

14 remaining items

Replayed PR

Setup

Cost estimates, written before each run

Results per arm

Findings

Proposed sweep size for slice 2

Criteria status

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

review: effort eval sweep for pinned agents, including Haiku 5.5 when it ships #5682

Description

Done when

Activity

kyle-sexton commented on Oct 2, 2026

kyle-sexton commented on Oct 4, 2026

Agent Brief

Routing: human-gated, needs decision

kyle-sexton commented on Oct 13, 2026

kyle-sexton commented on Oct 22, 2026

14 remaining items

kyle-sexton commented on Oct 26, 2026

kyle-sexton commented on Oct 30, 2026

Replayed PR

Setup

Cost estimates, written before each run

Results per arm

Findings

Proposed sweep size for slice 2

Criteria status

melodic-automation-lanes commented on Oct 30, 2026

kyle-sexton commented on Nov 1, 2026

kyle-sexton commented on Nov 1, 2026

kyle-sexton commented on Nov 2, 2026

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions