Repository navigation
feat: rerun an attempts run's judge while you pick, and count each attempt's changes from its worktree - #152
Open
sean-brydon wants to merge 3 commits into
Open
sean-brydon wants to merge 3 commits into
sean-brydon wants to merge 3 commits into
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
… and an attempt's changes count all its work
POST /v1/runs/{id}/judge (and `berth run judge`) reads each attempt's
worktree again and asks the same judge, headless and read-only, about the
attempts as they are now. The run stays at its pick gate: no PR, nothing
archived, the coding agents untouched. Every evaluation is kept on the run
with when it ran, each worktree's HEAD and dirty state and diffstat, and
its verdict or error; a reply that isn't valid is a failed evaluation with
the checks' pick flagged as a fallback. One rerun at a time per run; a pick
stops one in flight, and one berthd stopped in the middle of ends failed.
The attempt's diffstat was counted against the location's local default
branch: with no local main, merge-base failed and committed work showed as
"0 files, 0 commits". It is now counted against the run's base, else
origin's default branch, committed or not, and Review's Compare reads it
from the worktree as it is, with the file list from the same count.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…sions it saw and the earlier evaluations Compare's judge panel shows the current evaluation (a verdict, the checks' pick, or a fallback or failure with Retry), when it ran and which version of each worktree it judged, marks an attempt that changed since, and lists the earlier evaluations with what changed between them. A fallback by the checks is worded and coloured apart from a judge's verdict. Each attempt's Changes line and file list are its worktree now, against its base. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sean-brydon
force-pushed
the
feat/rerun-judge
branch
from
October 18, 2026 17:33
a7de643 to
3b52340
Compare
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #142.
Why it showed "0 files, 0 commits"
There were two causes:
collectstep. The file list under it was read live from the worktree, and it listed only uncommitted files. Whencollectran before the agent had done its work, the two disagreed. One likely trigger:waitalso stops onwaiting, so an agent stuck at a trust prompt is collected and judged before it does anything. That behaviour is unchanged here.collectcompared against the localmain. In a checkout without one,git merge-basefailed silently and the diff was taken against HEAD, so fully committed work counted 0 files and 0 commits.Fix: a new
attemptBaseuses the run'sbase, elseorigin/<default>, else the local default branch, the same way Review counts a worktree's commits. Compare now recounts each attempt live (committed plus uncommitted), and the Changes line and the file list come from the same count.Rerun judge
Engine (
runs/rejudge.go):Engine.Rejudgeworks while an attempts run waits at its pick.run.evaluations.API and CLI:
POST /v1/runs/{id}/judgereturns 202 with the new evaluation, 409 if the run isn't at its pick or a rerun is running, and 404 if there is no such run.GET /v1/review?run=addsevaluations,can_rejudgeand each attempt's livenow.berth run judge RUN, andrun.judging/run.judgedevents.berth_run_getincludes the latest evaluation.App (
views/review/judge-panel.tsx):Proof (mock)
Before a rerun, then judging:
A reply that isn't valid: a failure with Retry, with the checks' pick clearly marked as a fallback:
A rerun's verdict, what changed, and the earlier evaluations:
Tests
go test -race -count=3 ./internal/box/runs/passes.e2e/rerun-judge.spec.tspassed 3/3 at--repeat-each=3. The new a11y scenereview-compare-rejudgedpasses in light and dark.pnpm test(528), the app checks, build,tsc -p e2eand the docs build pass.TestAttachPassesShiftEnterfails against tmux 3.7c, unrelated to this change.Conflicts with #151: both change
compare-view.tsx,mock-runs.ts,orchestrate-core.ts,runsapi.goand the changelog. Whichever lands second gets rebased.🤖 Generated with Claude Code