Skip to content

feat: rerun an attempts run's judge while you pick, and count each attempt's changes from its worktree - #152

Open
sean-brydon wants to merge 3 commits into
mainfrom
feat/rerun-judge
Open

sean-brydon wants to merge 3 commits into
mainfrom
feat/rerun-judge

Conversation

@sean-brydon

Copy link
Copy Markdown
Collaborator

Closes #142.

Why it showed "0 files, 0 commits"

There were two causes:

  1. The count was a snapshot. The Changes line showed the figure the run took once, at its collect step. The file list under it was read live from the worktree, and it listed only uncommitted files. When collect ran before the agent had done its work, the two disagreed. One likely trigger: wait also stops on waiting, so an agent stuck at a trust prompt is collected and judged before it does anything. That behaviour is unchanged here.
  2. The base was wrong. collect compared against the local main. In a checkout without one, git merge-base failed silently and the diff was taken against HEAD, so fully committed work counted 0 files and 0 commits.

Fix: a new attemptBase uses the run's base, else origin/<default>, else the local default branch, the same way Review counts a worktree's commits. Compare now recounts each attempt live (committed plus uncommitted), and the Changes line and the file list come from the same count.

Rerun judge

Engine (runs/rejudge.go): Engine.Rejudge works while an attempts run waits at its pick.

  • It re-reads the worktrees, then asks the same judge agent, headless and read-only, with the same retry-once rule.
  • It validates the reply and records a new evaluation: when it ran, each attempt's HEAD and whether it had uncommitted changes, the diffstat and check result, and the verdict or the error. If the judge was an agent but the checks supplied the winner, the evaluation is marked as a fallback.
  • Earlier evaluations are kept in run.evaluations.
  • The run stays at its gate: no pull request, no archiving, and the coding agents aren't touched.
  • One rerun at a time per run; a second request gets 409. A pick stops a rerun in flight and waits for it. A rerun cut short by a berthd restart is marked failed.

API and CLI:

  • POST /v1/runs/{id}/judge returns 202 with the new evaluation, 409 if the run isn't at its pick or a rerun is running, and 404 if there is no such run.
  • GET /v1/review?run= adds evaluations, can_rejudge and each attempt's live now.
  • New berth run judge RUN, and run.judging / run.judged events.
  • berth_run_get includes the latest evaluation.

App (views/review/judge-panel.tsx):

  • The panel shows the current evaluation: a verdict, the checks' pick, a fallback or a failure, each with its own wording.
  • It shows when the evaluation ran and the version of each worktree it saw.
  • Rerun judge shows "Judging…" while it runs, and becomes Retry after a failure.
  • It says what changed since the previous evaluation, with the earlier ones in a collapsible list.
  • Attempt cards get a Version row and a "checks' pick" label.

Proof (mock)

Before a rerun, then judging:

A reply that isn't valid: a failure with Retry, with the checks' pick clearly marked as a fallback:

A rerun's verdict, what changed, and the earlier evaluations:

Tests

--- PASS: TestDiffSummaryCountsCommittedAndUncommitted
--- PASS: TestAttemptBaseWithoutLocalDefaultBranch
--- PASS: TestRejudgeOverTheAPI
--- PASS: TestRefreshReadsWorktreesAgain
--- PASS: TestRejudgeUpdatesTheVerdictAndStaysAtTheGate
--- PASS: TestRejudgeInvalidReplyIsAFailureWithTheChecksFallback
--- PASS: TestRejudgeOneAtATimeAndAPickStopsIt        # 8 concurrent requests: 1 starts, 7 refused
--- PASS: TestRejudgeNeedsAPickGate
--- PASS: TestRejudgeAcrossRestart
  • go test -race -count=3 ./internal/box/runs/ passes.
  • e2e/rerun-judge.spec.ts passed 3/3 at --repeat-each=3. The new a11y scene review-compare-rejudged passes in light and dark.
  • pnpm test (528), the app checks, build, tsc -p e2e and the docs build pass.
  • Locally TestAttachPassesShiftEnter fails against tmux 3.7c, unrelated to this change.

Conflicts with #151: both change compare-view.tsx, mock-runs.ts, orchestrate-core.ts, runsapi.go and the changelog. Whichever lands second gets rebased.

🤖 Generated with Claude Code

@vercel

vercel Bot commented Oct 18, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
berth Ready Ready Preview Oct 19, 2026 11:42am UTC
berth-docs Ready Ready Preview Oct 19, 2026 11:42am UTC

Request Review

sean-brydon and others added 2 commits October 18, 2026 18:27
… and an attempt's changes count all its work

POST /v1/runs/{id}/judge (and `berth run judge`) reads each attempt's
worktree again and asks the same judge, headless and read-only, about the
attempts as they are now. The run stays at its pick gate: no PR, nothing
archived, the coding agents untouched. Every evaluation is kept on the run
with when it ran, each worktree's HEAD and dirty state and diffstat, and
its verdict or error; a reply that isn't valid is a failed evaluation with
the checks' pick flagged as a fallback. One rerun at a time per run; a pick
stops one in flight, and one berthd stopped in the middle of ends failed.

The attempt's diffstat was counted against the location's local default
branch: with no local main, merge-base failed and committed work showed as
"0 files, 0 commits". It is now counted against the run's base, else
origin's default branch, committed or not, and Review's Compare reads it
from the worktree as it is, with the file list from the same count.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…sions it saw and the earlier evaluations

Compare's judge panel shows the current evaluation (a verdict, the checks'
pick, or a fallback or failure with Retry), when it ran and which version
of each worktree it judged, marks an attempt that changed since, and lists
the earlier evaluations with what changed between them. A fallback by the
checks is worded and coloured apart from a judge's verdict. Each attempt's
Changes line and file list are its worktree now, against its base.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch was successfully deployed

2 active deployments
Preview – berth-docs — 379b18e8 Deployed Oct 19, 2026 by vercel[bot]
Preview – berth — 379b18e8 Deployed Oct 19, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add “Rerun judge” for existing attempts without restarting the coding agents

1 participant