Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
e641af8
chore(quest): claim quest/m0/routing-evidence/review-tallies.md
kixelated Nov 11, 2026
487c625
feat(review): tally every reviewer's findings across a branch's rounds
kixelated Nov 11, 2026
6326793
fix(review): drop --new's unsettled findings from tallies, count a br…
kixelated Nov 11, 2026
0e96a16
fix(usage): count a branch's tally once per head repository, from a P…
kixelated Nov 11, 2026
fcd3fed
test(usage): fetch the no-implementer fork PR so its dedupe case is e…
kixelated Nov 11, 2026
89b95c1
Merge remote-tracking branch 'origin/main' into quest/m0/routing-evid…
kixelated Nov 12, 2026
179c2bc
Merge remote-tracking branch 'origin/main' into quest/m0/routing-evid…
kixelated Nov 12, 2026
00fec4b
feat(review): settle low findings with --settle so tallies count them
kixelated Nov 12, 2026
a6e2b90
fix(review): --settle reports no reviewer usage
kixelated Nov 12, 2026
0834454
fix(review): save a settlement once recorded, on the last round's base
kixelated Nov 12, 2026
64e548b
fix(review): settle only a session whose reviewed head HEAD contains,…
kixelated Nov 12, 2026
9b9ff6e
Name --base in the settle line, matching every round
kixelated Nov 12, 2026
2b0ff4f
Review again after a settle, keep deleted forks apart, fix settle labels
kixelated Nov 12, 2026
b510daa
Review the whole change on a head without the settled one
kixelated Nov 12, 2026
22d6288
Remove quest review --settle
kixelated Nov 12, 2026
aeb4491
Merge remote-tracking branch 'origin/main' into quest/m0/routing-evid…
kixelated Nov 12, 2026
82b0f7a
feat(usage): count a merged PR's last all-low round as settled
kixelated Nov 12, 2026
f76457e
quest: drop charts' links to the finished review-tallies quest
kixelated Nov 12, 2026
61c71df
fix(review): read a settled count back, keep the head short for a lef…
kixelated Nov 12, 2026
6817927
fix(usage): count a branch's latest tally by its rounds, keep a recor…
kixelated Nov 12, 2026
b584660
Merge remote-tracking branch 'origin/main' into quest/m0/routing-evid…
kixelated Nov 12, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 25 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -1409,7 +1409,9 @@ reviewer's prompt cache; Claude Code is asked for its 1-hour cache, and Codex
keeps its default. Each reviewer's session is kept per branch in the clone's
common Git directory, so any worktree of the clone resumes it, except opencode's,
which stays in the checkout it started in, so another worktree starts a new one;
`--new` starts over. `--branch` names the branch instead of the checked-out one, which a
`--new` starts new sessions, but continues each reviewer's tally of its findings
on the branch, less those it drops unsettled, unless the saved state is one
this quest cannot read. `--branch` names the branch instead of the checked-out one, which a
detached `HEAD` requires, such as a checkout of a branch another worktree
holds, so its rounds continue that branch's session.

Expand Down Expand Up @@ -1456,15 +1458,22 @@ finding count, then each finding as its number, severity, `path:line`, and title
indented body, then `earlier` with the last round's head and its findings, each
as its number, disposition, severity, `path:line`, and title over its reason; last
come a `record` line and a `next` line. `--json` prints one object with
`rounds`, one per reviewer, `record`, and `next`.
`rounds`, one per reviewer, `tallies`, every reviewer's findings across the
branch's rounds, `record`, and `next`.

After each round, `quest review` rewrites the `## Review` section of the open PR
from the branch into the base, found among the upstream's PRs from the
repository the base's remote pushes to, and prints `record <url>`. The record
names the reviewed head and the base's short SHA, then for each reviewer its
harness, model, and effort, the session and round, the score and rationale,
each finding with its location, and the last round's findings with their
dispositions. `quest status` takes the last full SHA the section names as the
harness, model, and effort, the session and round, the score and rationale, its tally, each finding with
its location, and the last round's findings with their dispositions. The
tally, as `- Rounds: 3; findings 4 (high 1, medium 3): fixed 3, refuted 1, open 0`,
counts the reviewer's rounds on the branch and each finding it raised once,
by its latest severity, however many rounds keep it open. A finding counts
for the reviewer that raised it when a new size pick's reviewer fixes or
refutes it, but one that reviewer keeps open becomes its own, since its round
reports it again. `--new` drops the findings it leaves unsettled from the
tallies. A reviewer the run left out keeps a block with only its tally. `quest status` takes the last full SHA the section names as the
reviewed head, so the head appears in full only once no reviewer reports a
finding, and model text
appears as inline code, which mentions or links nothing and never holds a full
Expand Down Expand Up @@ -1502,7 +1511,7 @@ moq-dev/quest, PRs updated and sessions active since 2026-11-06; costs (~$) esti

#569 S merged: quest: finish model routing (quest/m0/model-routing/README)
implementer: Claude Code, claude-opus-5-5, unknown effort
review: Codex, gpt-6.1-sol, medium effort; round 1, score 10/10, 0 open, last round 0 fixed and 0 refuted
review: Codex, gpt-6.1-sol, medium effort; round 1, score 10/10; rounds 1, findings 0: fixed 0, refuted 0, open 0
online review: review (OpenAI), not at the final head; 0 findings: 0 accepted, 0 declined, 0 unclassified, 0 unanswered
land fixes: codex, gpt-6.1-sol, medium effort
sessions:
Expand All @@ -1522,9 +1531,11 @@ remotes, or, once the checkout is deleted, a Codex log for that exact directory
names (never a parent's, which a nested clone may not share); otherwise it stays unattributed. A session is reviewing when the PR's `## Review` record names
it or Codex marks it a review, implementing when it began before the PR opened,
and iterating otherwise. An opencode session, whose log quest does not read, is
the reviewing session of the newest PR whose `## Review` record names it. Each PR shows its `## Models` size and implementers, and as `unread:` each Implementer line it cannot read; each reviewer's latest `quest review` round's
reviewer, round, score, open findings, and the last round's findings fixed or
refuted; the land fixer's fixes; the later PRs that list it under
the reviewing session of the newest PR whose `## Review` record names it. Each PR shows its `## Models` size and implementers, and as `unread:` each Implementer line it cannot read; each reviewer the
`quest review` record names, with its latest round and score and its tally of
findings across every round, where a merged PR whose last round left only low
findings open counts them settled, since `/quest-review` fixes or refutes those
before merge without another round; the land fixer's fixes; the later PRs that list it under
`## Caused by`; and each online reviewer's record, read from GitHub.

An online reviewer is any bot account whose review has a body or opens a
Expand All @@ -1544,7 +1555,11 @@ a bot resolving its own thread or a later review leaving it out, and as
unanswered otherwise.

Rollups follow: spend by size, role, harness, model, and effort; accuracy
by size and implementer, where an `Implementer:` line's own size, as
by size and implementer, which sums the tallies, so it counts every finding
caught before merge. Since a branch's tallies span every PR from it, only the
PR whose record holds its latest tally, by the most rounds and then the newest,
counts them, and a record written before tallies
counts as unreviewed. An `Implementer:` line's own size, as
`(iterate, XS)`, counts over the PR's; and, over the merged PRs, each online
reviewer's coverage, final heads reviewed, findings per PR reviewed, and
answers. Sessions on no PR here, from any
Expand Down
3 changes: 1 addition & 2 deletions quest/m0/routing-evidence/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,8 +30,7 @@ Decided 2026-11-11 (user, /quest-plan, from an audit of PRs merged
- Experiments? ✅ Shadow reviews and a writer A/B (rejected: shadow reviews
only; none).
- Records?
- ✅ [Cumulative review tallies](/quest/m0/routing-evidence/review-tallies.md)
and online reviewers in usage
- ✅ Cumulative review tallies and online reviewers in usage
- ✅ [Attribution through token-max](/quest/m0/routing-evidence/usage-token-max.md),
kept after the maintainer first left it out, because the writer A/B
compares cost and `quest usage` crashes on this host
Expand Down
9 changes: 3 additions & 6 deletions quest/m0/routing-evidence/charts.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,16 +69,13 @@ Agent's choices, for the implementer to revisit:
readout: which `quest usage --json` fields feed each axis, how cells with
too few PRs are marked rather than dropped, and that each chart names its
date range and PR counts.
- `quest usage --json` reports each reviewer's latest round, and
[review tallies](/quest/m0/routing-evidence/review-tallies.md) add totals
across rounds. If the tally line does not also record the first round's
findings, which the readout compares, add them there.
- `quest usage --json` reports each reviewer's latest round and its tally,
which sums findings across rounds. If the readout compares first-round
findings on their own, record them beside the tally.
- Make the first charts from the data on main when this lands, link them in
this quest's PR, and say which cells are still too thin.

## Required

- [Review tallies](/quest/m0/routing-evidence/review-tallies.md) - the
findings each reviewer raised across rounds
- [usage through token-max](/quest/m0/routing-evidence/usage-token-max.md) -
the cost axis, which today misses sub-agent transcripts, 72% of host spend
27 changes: 0 additions & 27 deletions quest/m0/routing-evidence/review-tallies.md

This file was deleted.

5 changes: 0 additions & 5 deletions quest/m0/routing-evidence/shadow-review.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,8 +33,3 @@ Agent's choices, for the implementer to revisit:
that matches a routed finding on the same file, within a few lines, counts
as found by both. The implementer checks a sample of matches by hand
before trusting the rule.

## Required

- [Review tallies](/quest/m0/routing-evidence/review-tallies.md) - counts the
shadow's findings across rounds
2 changes: 0 additions & 2 deletions quest/m0/routing-evidence/writer-ab.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,5 @@ Agent's choices, for the implementer to revisit:

## Required

- [Review tallies](/quest/m0/routing-evidence/review-tallies.md) - counts the
findings the arms are compared on
- [usage through token-max](/quest/m0/routing-evidence/usage-token-max.md) -
counts sub-agent writers' cost, without which an arm's cost is undercounted
4 changes: 2 additions & 2 deletions quest/m1/model-check.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,8 +30,8 @@ with `quest usage --since <the last run's date>`, whose accuracy rollup reads ea
`## Models` size and implementers and its `## Review` reviewer. It tallies each implementer by the size
on its own `Implementer:` line where it carries one, such as `(iterate, XS)`,
not the PR's `## Models` size, so a small iterate on a large PR counts as the
work it was routed for. It counts fixed and refuted findings only from the
round before each PR's latest, the one its record keeps, and leaves design
work it was routed for. It counts each reviewer's findings, fixed, refuted, settled (a merged PR's last all-low round), and open,
across every round on the branch, from the tally its record keeps, and leaves design
checkpoints and outages to count by hand:

- Tally by size and by implementing harness, model, and effort: review
Expand Down
17 changes: 9 additions & 8 deletions src/cli/main.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1108,13 +1108,14 @@ export function main(argv: string[], out: Output): number | Promise<number> {
"before the PR opened, and iterating otherwise. An opencode session, whose log quest does not read, belongs to the newest PR whose " +
"## Review record names it, as reviewing. Tokens and cost come from ccusage (session --json), the " +
"cost an estimate at each model's API list price. Each PR also shows its ## Models size and implementers, and as unread each Implementer line it cannot read, " +
"each reviewer's latest ## Review round, with its round, score, and findings, " +
"each reviewer the ## Review record names, with its latest round and score and its tally of findings across every round, " +
"where a merged PR whose last round left only low findings open counts them settled, since the author fixes or refutes those before merge, " +
"the later PRs that list it under ## Caused by, and each online reviewer's record from GitHub: whether it " +
"reviewed the last head, and its findings accepted, declined, unclassified, or unanswered. An online reviewer " +
"is a bot whose review has a body or opens a thread, or a review whose body opens with Automated review by <name>; a finding is a " +
"review thread it opens or a [P<n>] line in its review body, and its first answer's first word takes or " +
"declines it. Rollups follow: spend by size, role, harness, model, and " +
"effort, accuracy by size and implementer, where an Implementer line's own size, as (iterate, XS), wins " +
"effort, accuracy by size and implementer, summing each branch's tallies once, where an Implementer line's own size, as (iterate, XS), wins " +
"over the PR's, and each online reviewer's coverage and findings over the merged PRs. Without ccusage on PATH, it says how to install it and reports the PRs' records alone. " +
"--json prints the same report as one object.\n\n" +
"Uses GH_TOKEN, GITHUB_TOKEN, or gh auth token. Exits 0 when done, 2 for invalid arguments, and 1 for " +
Expand Down Expand Up @@ -1217,7 +1218,7 @@ export function main(argv: string[], out: Output): number | Promise<number> {
"starts a new session that reviews the whole change and still weighs --responses. A round that would " +
"resume on the head the last round reviewed, when that round found nothing, runs no reviewer and " +
"repeats that round, marked unchanged, so on the same head only a reviewer with findings to settle runs. " +
"--new starts new sessions. Each reviewer's session is kept per branch in the clone's common Git directory, so any worktree of the clone resumes it, except opencode's, which stays in the checkout it started in. " +
"--new starts new sessions, continuing each reviewer's tally of its findings on the branch, less those it drops unsettled, unless the saved state is one this quest cannot read. Each reviewer's session is kept per branch in the clone's common Git directory, so any worktree of the clone resumes it, except opencode's, which stays in the checkout it started in. " +
"A checked-out branch with an upstream other than the base continues its upstream branch's session, so a " +
"temporary branch tracking the PR's head shares the head's rounds. " +
"--branch names the branch instead of the checked-out one, as a detached HEAD requires, such as a " +
Expand All @@ -1242,11 +1243,11 @@ export function main(argv: string[], out: Output): number | Promise<number> {
"`next checkpoint` when a reviewer's every third round reports other findings, to reconsider the approach " +
"before fixing them, `next respond` to fix or refute the findings, or `next done`. A reply without a valid score " +
"fails as any malformed reply does. --json prints " +
"one object: `rounds`, one per reviewer, `record`, and `next`.\n\n" +
"one object: `rounds`, one per reviewer, `tallies`, every reviewer's findings across the branch's rounds, `record`, and `next`.\n\n" +
"After each round, it rewrites the `## Review` section of the branch's open PR into the base, as the " +
"user the GitHub token names, keeping the rest of the body, and prints `record <url>`: the reviewed " +
"head, named in full only once no reviewer reports a finding, the base, and each reviewer, its " +
"session and round, score, and each finding with its disposition. With no " +
"session and round, score, its tally of findings across the branch's rounds, and each finding with its disposition. With no " +
"such PR, or a remote off GitHub, it prints `record skipped: <why>` and changes nothing.\n\n" +
"Exits 0 once the round finishes, whatever it found, 2 for invalid arguments " +
"or a checkout or CLI that cannot be reviewed, including a login the API rejects, 3 when a reviewer's " +
Expand Down Expand Up @@ -1309,17 +1310,17 @@ export function main(argv: string[], out: Output): number | Promise<number> {
},
);
pending = run.then(
async (rounds) => {
async ({ rounds, tallies }) => {
// The rounds are saved in their sessions either way, so their report prints before a failed record.
let record: Recorded | undefined;
let failure: unknown;
try {
record = await recordRounds(root(), rounds, github);
record = await recordRounds(root(), rounds, tallies, github);
} catch (error) {
failure = error;
}
if (options.json)
print(JSON.stringify({ rounds, record: record ?? null, next: nextStep(rounds) }));
print(JSON.stringify({ rounds, tallies, record: record ?? null, next: nextStep(rounds) }));
else {
for (const line of roundLines(rounds)) print(line);
if (record !== undefined)
Expand Down
Loading
Loading