diff --git a/README.md b/README.md index 33904db8..2eabfc2c 100644 --- a/README.md +++ b/README.md @@ -1409,7 +1409,9 @@ reviewer's prompt cache; Claude Code is asked for its 1-hour cache, and Codex keeps its default. Each reviewer's session is kept per branch in the clone's common Git directory, so any worktree of the clone resumes it, except opencode's, which stays in the checkout it started in, so another worktree starts a new one; -`--new` starts over. `--branch` names the branch instead of the checked-out one, which a +`--new` starts new sessions, but continues each reviewer's tally of its findings +on the branch, less those it drops unsettled, unless the saved state is one +this quest cannot read. `--branch` names the branch instead of the checked-out one, which a detached `HEAD` requires, such as a checkout of a branch another worktree holds, so its rounds continue that branch's session. @@ -1456,15 +1458,22 @@ finding count, then each finding as its number, severity, `path:line`, and title indented body, then `earlier` with the last round's head and its findings, each as its number, disposition, severity, `path:line`, and title over its reason; last come a `record` line and a `next` line. `--json` prints one object with -`rounds`, one per reviewer, `record`, and `next`. +`rounds`, one per reviewer, `tallies`, every reviewer's findings across the +branch's rounds, `record`, and `next`. After each round, `quest review` rewrites the `## Review` section of the open PR from the branch into the base, found among the upstream's PRs from the repository the base's remote pushes to, and prints `record `. The record names the reviewed head and the base's short SHA, then for each reviewer its -harness, model, and effort, the session and round, the score and rationale, -each finding with its location, and the last round's findings with their -dispositions. `quest status` takes the last full SHA the section names as the +harness, model, and effort, the session and round, the score and rationale, its tally, each finding with +its location, and the last round's findings with their dispositions. The +tally, as `- Rounds: 3; findings 4 (high 1, medium 3): fixed 3, refuted 1, open 0`, +counts the reviewer's rounds on the branch and each finding it raised once, +by its latest severity, however many rounds keep it open. A finding counts +for the reviewer that raised it when a new size pick's reviewer fixes or +refutes it, but one that reviewer keeps open becomes its own, since its round +reports it again. `--new` drops the findings it leaves unsettled from the +tallies. A reviewer the run left out keeps a block with only its tally. `quest status` takes the last full SHA the section names as the reviewed head, so the head appears in full only once no reviewer reports a finding, and model text appears as inline code, which mentions or links nothing and never holds a full @@ -1502,7 +1511,7 @@ moq-dev/quest, PRs updated and sessions active since 2026-11-06; costs (~$) esti #569 S merged: quest: finish model routing (quest/m0/model-routing/README) implementer: Claude Code, claude-opus-5-5, unknown effort - review: Codex, gpt-6.1-sol, medium effort; round 1, score 10/10, 0 open, last round 0 fixed and 0 refuted + review: Codex, gpt-6.1-sol, medium effort; round 1, score 10/10; rounds 1, findings 0: fixed 0, refuted 0, open 0 online review: review (OpenAI), not at the final head; 0 findings: 0 accepted, 0 declined, 0 unclassified, 0 unanswered land fixes: codex, gpt-6.1-sol, medium effort sessions: @@ -1522,9 +1531,11 @@ remotes, or, once the checkout is deleted, a Codex log for that exact directory names (never a parent's, which a nested clone may not share); otherwise it stays unattributed. A session is reviewing when the PR's `## Review` record names it or Codex marks it a review, implementing when it began before the PR opened, and iterating otherwise. An opencode session, whose log quest does not read, is -the reviewing session of the newest PR whose `## Review` record names it. Each PR shows its `## Models` size and implementers, and as `unread:` each Implementer line it cannot read; each reviewer's latest `quest review` round's -reviewer, round, score, open findings, and the last round's findings fixed or -refuted; the land fixer's fixes; the later PRs that list it under +the reviewing session of the newest PR whose `## Review` record names it. Each PR shows its `## Models` size and implementers, and as `unread:` each Implementer line it cannot read; each reviewer the +`quest review` record names, with its latest round and score and its tally of +findings across every round, where a merged PR whose last round left only low +findings open counts them settled, since `/quest-review` fixes or refutes those +before merge without another round; the land fixer's fixes; the later PRs that list it under `## Caused by`; and each online reviewer's record, read from GitHub. An online reviewer is any bot account whose review has a body or opens a @@ -1544,7 +1555,11 @@ a bot resolving its own thread or a later review leaving it out, and as unanswered otherwise. Rollups follow: spend by size, role, harness, model, and effort; accuracy -by size and implementer, where an `Implementer:` line's own size, as +by size and implementer, which sums the tallies, so it counts every finding +caught before merge. Since a branch's tallies span every PR from it, only the +PR whose record holds its latest tally, by the most rounds and then the newest, +counts them, and a record written before tallies +counts as unreviewed. An `Implementer:` line's own size, as `(iterate, XS)`, counts over the PR's; and, over the merged PRs, each online reviewer's coverage, final heads reviewed, findings per PR reviewed, and answers. Sessions on no PR here, from any diff --git a/quest/m0/routing-evidence/README.md b/quest/m0/routing-evidence/README.md index 3ae24716..ec83129c 100644 --- a/quest/m0/routing-evidence/README.md +++ b/quest/m0/routing-evidence/README.md @@ -30,8 +30,7 @@ Decided 2026-11-11 (user, /quest-plan, from an audit of PRs merged - Experiments? ✅ Shadow reviews and a writer A/B (rejected: shadow reviews only; none). - Records? - - ✅ [Cumulative review tallies](/quest/m0/routing-evidence/review-tallies.md) - and online reviewers in usage + - ✅ Cumulative review tallies and online reviewers in usage - ✅ [Attribution through token-max](/quest/m0/routing-evidence/usage-token-max.md), kept after the maintainer first left it out, because the writer A/B compares cost and `quest usage` crashes on this host diff --git a/quest/m0/routing-evidence/charts.md b/quest/m0/routing-evidence/charts.md index 6ccfed08..580c174f 100644 --- a/quest/m0/routing-evidence/charts.md +++ b/quest/m0/routing-evidence/charts.md @@ -69,16 +69,13 @@ Agent's choices, for the implementer to revisit: readout: which `quest usage --json` fields feed each axis, how cells with too few PRs are marked rather than dropped, and that each chart names its date range and PR counts. -- `quest usage --json` reports each reviewer's latest round, and - [review tallies](/quest/m0/routing-evidence/review-tallies.md) add totals - across rounds. If the tally line does not also record the first round's - findings, which the readout compares, add them there. +- `quest usage --json` reports each reviewer's latest round and its tally, + which sums findings across rounds. If the readout compares first-round + findings on their own, record them beside the tally. - Make the first charts from the data on main when this lands, link them in this quest's PR, and say which cells are still too thin. ## Required -- [Review tallies](/quest/m0/routing-evidence/review-tallies.md) - the - findings each reviewer raised across rounds - [usage through token-max](/quest/m0/routing-evidence/usage-token-max.md) - the cost axis, which today misses sub-agent transcripts, 72% of host spend diff --git a/quest/m0/routing-evidence/review-tallies.md b/quest/m0/routing-evidence/review-tallies.md deleted file mode 100644 index bfa3cd17..00000000 --- a/quest/m0/routing-evidence/review-tallies.md +++ /dev/null @@ -1,27 +0,0 @@ -# [M] The review record tallies every round - -## Goal - -A PR body's `## Review` record shows, for each reviewer, the findings it -raised across every round on the branch, by severity, and how many were -fixed, refuted, or left open, beside the latest round it already shows. -`quest usage` reads those tallies, so its accuracy rollup counts every -finding caught before merge, not only the round before the latest. - -## Plan - -Decided 2026-11-11 (user, /quest-plan, -[routing evidence](/quest/m0/routing-evidence/README.md)). Today the record -keeps only the latest round. The review state under the clone's Git -directory holds every round, but only on the host that ran it. - -Agent's choices, for the implementer to revisit: - -- Keep one tally line per reviewer, such as - `- Rounds: 3; findings 4 (high 1, medium 3): fixed 3, refuted 1, open 0`, - computed from the saved session's rounds and their dispositions. A - `--new` session continues the old one's tally, so restarting a reviewer - does not reset its count. -- Drop `quest usage`'s "round before the latest" rule once the tally line is - there. Leave older records, which have no tally line, uncounted, since - nothing is in production. diff --git a/quest/m0/routing-evidence/shadow-review.md b/quest/m0/routing-evidence/shadow-review.md index 5792b7f4..3094e566 100644 --- a/quest/m0/routing-evidence/shadow-review.md +++ b/quest/m0/routing-evidence/shadow-review.md @@ -33,8 +33,3 @@ Agent's choices, for the implementer to revisit: that matches a routed finding on the same file, within a few lines, counts as found by both. The implementer checks a sample of matches by hand before trusting the rule. - -## Required - -- [Review tallies](/quest/m0/routing-evidence/review-tallies.md) - counts the - shadow's findings across rounds diff --git a/quest/m0/routing-evidence/writer-ab.md b/quest/m0/routing-evidence/writer-ab.md index 858dae63..11ab9623 100644 --- a/quest/m0/routing-evidence/writer-ab.md +++ b/quest/m0/routing-evidence/writer-ab.md @@ -58,7 +58,5 @@ Agent's choices, for the implementer to revisit: ## Required -- [Review tallies](/quest/m0/routing-evidence/review-tallies.md) - counts the - findings the arms are compared on - [usage through token-max](/quest/m0/routing-evidence/usage-token-max.md) - counts sub-agent writers' cost, without which an arm's cost is undercounted diff --git a/quest/m1/model-check.md b/quest/m1/model-check.md index a665b719..c264481c 100644 --- a/quest/m1/model-check.md +++ b/quest/m1/model-check.md @@ -30,8 +30,8 @@ with `quest usage --since `, whose accuracy rollup reads ea `## Models` size and implementers and its `## Review` reviewer. It tallies each implementer by the size on its own `Implementer:` line where it carries one, such as `(iterate, XS)`, not the PR's `## Models` size, so a small iterate on a large PR counts as the -work it was routed for. It counts fixed and refuted findings only from the -round before each PR's latest, the one its record keeps, and leaves design +work it was routed for. It counts each reviewer's findings, fixed, refuted, settled (a merged PR's last all-low round), and open, +across every round on the branch, from the tally its record keeps, and leaves design checkpoints and outages to count by hand: - Tally by size and by implementing harness, model, and effort: review diff --git a/src/cli/main.ts b/src/cli/main.ts index 90cfef1f..f4468967 100644 --- a/src/cli/main.ts +++ b/src/cli/main.ts @@ -1108,13 +1108,14 @@ export function main(argv: string[], out: Output): number | Promise { "before the PR opened, and iterating otherwise. An opencode session, whose log quest does not read, belongs to the newest PR whose " + "## Review record names it, as reviewing. Tokens and cost come from ccusage (session --json), the " + "cost an estimate at each model's API list price. Each PR also shows its ## Models size and implementers, and as unread each Implementer line it cannot read, " + - "each reviewer's latest ## Review round, with its round, score, and findings, " + + "each reviewer the ## Review record names, with its latest round and score and its tally of findings across every round, " + + "where a merged PR whose last round left only low findings open counts them settled, since the author fixes or refutes those before merge, " + "the later PRs that list it under ## Caused by, and each online reviewer's record from GitHub: whether it " + "reviewed the last head, and its findings accepted, declined, unclassified, or unanswered. An online reviewer " + "is a bot whose review has a body or opens a thread, or a review whose body opens with Automated review by ; a finding is a " + "review thread it opens or a [P] line in its review body, and its first answer's first word takes or " + "declines it. Rollups follow: spend by size, role, harness, model, and " + - "effort, accuracy by size and implementer, where an Implementer line's own size, as (iterate, XS), wins " + + "effort, accuracy by size and implementer, summing each branch's tallies once, where an Implementer line's own size, as (iterate, XS), wins " + "over the PR's, and each online reviewer's coverage and findings over the merged PRs. Without ccusage on PATH, it says how to install it and reports the PRs' records alone. " + "--json prints the same report as one object.\n\n" + "Uses GH_TOKEN, GITHUB_TOKEN, or gh auth token. Exits 0 when done, 2 for invalid arguments, and 1 for " + @@ -1217,7 +1218,7 @@ export function main(argv: string[], out: Output): number | Promise { "starts a new session that reviews the whole change and still weighs --responses. A round that would " + "resume on the head the last round reviewed, when that round found nothing, runs no reviewer and " + "repeats that round, marked unchanged, so on the same head only a reviewer with findings to settle runs. " + - "--new starts new sessions. Each reviewer's session is kept per branch in the clone's common Git directory, so any worktree of the clone resumes it, except opencode's, which stays in the checkout it started in. " + + "--new starts new sessions, continuing each reviewer's tally of its findings on the branch, less those it drops unsettled, unless the saved state is one this quest cannot read. Each reviewer's session is kept per branch in the clone's common Git directory, so any worktree of the clone resumes it, except opencode's, which stays in the checkout it started in. " + "A checked-out branch with an upstream other than the base continues its upstream branch's session, so a " + "temporary branch tracking the PR's head shares the head's rounds. " + "--branch names the branch instead of the checked-out one, as a detached HEAD requires, such as a " + @@ -1242,11 +1243,11 @@ export function main(argv: string[], out: Output): number | Promise { "`next checkpoint` when a reviewer's every third round reports other findings, to reconsider the approach " + "before fixing them, `next respond` to fix or refute the findings, or `next done`. A reply without a valid score " + "fails as any malformed reply does. --json prints " + - "one object: `rounds`, one per reviewer, `record`, and `next`.\n\n" + + "one object: `rounds`, one per reviewer, `tallies`, every reviewer's findings across the branch's rounds, `record`, and `next`.\n\n" + "After each round, it rewrites the `## Review` section of the branch's open PR into the base, as the " + "user the GitHub token names, keeping the rest of the body, and prints `record `: the reviewed " + "head, named in full only once no reviewer reports a finding, the base, and each reviewer, its " + - "session and round, score, and each finding with its disposition. With no " + + "session and round, score, its tally of findings across the branch's rounds, and each finding with its disposition. With no " + "such PR, or a remote off GitHub, it prints `record skipped: ` and changes nothing.\n\n" + "Exits 0 once the round finishes, whatever it found, 2 for invalid arguments " + "or a checkout or CLI that cannot be reviewed, including a login the API rejects, 3 when a reviewer's " + @@ -1309,17 +1310,17 @@ export function main(argv: string[], out: Output): number | Promise { }, ); pending = run.then( - async (rounds) => { + async ({ rounds, tallies }) => { // The rounds are saved in their sessions either way, so their report prints before a failed record. let record: Recorded | undefined; let failure: unknown; try { - record = await recordRounds(root(), rounds, github); + record = await recordRounds(root(), rounds, tallies, github); } catch (error) { failure = error; } if (options.json) - print(JSON.stringify({ rounds, record: record ?? null, next: nextStep(rounds) })); + print(JSON.stringify({ rounds, tallies, record: record ?? null, next: nextStep(rounds) })); else { for (const line of roundLines(rounds)) print(line); if (record !== undefined) diff --git a/src/cli/review-record.ts b/src/cli/review-record.ts index cb474fdd..d25d8253 100644 --- a/src/cli/review-record.ts +++ b/src/cli/review-record.ts @@ -5,7 +5,7 @@ import { recorded } from "../github/models"; import { type ClaimDependencies, repositoryPath, upstream } from "./claims"; import { git } from "./git"; import { requireRemote, string } from "./github"; -import { type PullModels, type Round, located, parseBase } from "./review"; +import { type PullModels, type Round, SEVERITIES, type Tally, located, parseBase } from "./review"; const HARNESSES: Record = { claude: "Claude Code", codex: "Codex", opencode: "opencode" }; @@ -25,10 +25,42 @@ function code(text: string): string { const capitalized = (word: string) => word[0]!.toUpperCase() + word.slice(1); -/** The `## Review` section's text for a run's `rounds` of the same head, without its heading. */ -export function reviewRecord(rounds: Round[]): string { +/** A tally's counts, as the record's `Rounds:` line and `quest usage` write them. */ +export interface Counts { + rounds: number; + findings: Tally["findings"]; + fixed: number; + refuted: number; + /** Open findings of a merged PR's last all-low round, which the review rule fixes or refutes before merge. */ + settled: number; + open: number; +} + +/** The counts as the record's `Rounds:` line ends, after the rounds, naming only the severities found. */ +export function findingsText(counts: Omit): string { + const total = SEVERITIES.reduce((sum, severity) => sum + counts.findings[severity], 0); + const severities = SEVERITIES.filter((severity) => counts.findings[severity] > 0) + .map((severity) => `${severity} ${counts.findings[severity]}`) + .join(", "); + return `findings ${total}${severities && ` (${severities})`}: fixed ${counts.fixed}, refuted ${counts.refuted}${counts.settled > 0 ? `, settled ${counts.settled}` : ""}, open ${counts.open}`; +} + +/** A tally's counts, where a finding neither fixed nor refuted is open. */ +const counted = (tally: Tally): Counts => ({ + ...tally, + settled: 0, + open: SEVERITIES.reduce((sum, severity) => sum + tally.findings[severity], 0) - tally.fixed - tally.refuted, +}); + +/** + * The `## Review` section's text for a run's `rounds` of the same head, without its heading, with every tally + * on the branch: each round's reviewer's beside its round, and a reviewer the run left out on its own. + */ +export function reviewRecord(rounds: Round[], tallies: Tally[]): string { const [shared] = rounds; - const clean = rounds.every((round) => round.findings.length === 0); + // A reviewer the run left out may still hold open findings, which keep the head from passing too. + const clean = + rounds.every((round) => round.findings.length === 0) && tallies.every((tally) => counted(tally).open === 0); const lines = [ MARK, "", @@ -43,6 +75,9 @@ export function reviewRecord(rounds: Round[]): string { `- Session: ${code(round.session)}, round ${round.round}`, `- Score: ${round.score}/10, ${code(round.rationale)}`, ); + const tally = tallies.find((each) => each.reviewer === round.reviewer); + if (tally === undefined) throw new Error(`no tally for ${round.reviewer}'s round`); + lines.push(`- Rounds: ${tally.rounds}; ${findingsText(counted(tally))}`); if (round.findings.length === 0) lines.push("- Findings: none"); else lines.push("- Findings, open until the next round settles them:"); for (const finding of round.findings) @@ -54,24 +89,30 @@ export function reviewRecord(rounds: Round[]): string { ` - ${capitalized(finding.disposition)}: ${finding.severity}, ${code(located(finding))}: ${code(finding.title)}; ${code(finding.reason)}`, ); } + for (const tally of tallies.filter((each) => !rounds.some((round) => round.reviewer === each.reviewer))) + lines.push( + "", + `- Reviewer: ${HARNESSES[tally.reviewer]}, ${tally.model}, ${tally.effort} effort`, + `- Rounds: ${tally.rounds}; ${findingsText(counted(tally))}`, + ); return lines.join("\n"); } -/** What a PR body's `## Review` record says of one reviewer's latest round, as `reviewRecord` writes it. */ +/** What a PR body's `## Review` record says of one reviewer, as `reviewRecord` writes it. */ export interface ReviewSummary { /** The reviewer's harness as the record names it, such as `Codex`. */ harness: string; model: string; effort: string; - session: string; - round: number; - /** Null on a record written before rounds were scored. */ + /** Its latest session and round, null when the latest run left it out. */ + session: string | null; + round: number | null; + /** Null when the latest run left it out, or on a record written before rounds were scored. */ score: number | null; - /** The latest round's findings, open until the next round settles them. */ - findings: number; - /** The round before's findings this round found fixed, and those whose refutation held. */ - fixed: number; - refuted: number; + /** Its findings across every round on the branch, null on a record written before tallies. */ + tally: Counts | null; + /** The severity of each finding its latest round left open. */ + open: Tally["findings"]; } const MARK = "Rewritten by `quest review` after each round."; @@ -99,23 +140,59 @@ export function parseReview(body: string): ReviewSummary[] { return found; }; const reviewer = match(/^- Reviewer: (.+?), (\S+), (\S+) effort\r?$/m); - const session = match(/^- Session: (`+) ?(.*?) ?\1, round (\d+)\r?$/m); + const tally = parseCounts(text); + // A reviewer the latest run left out has only its tally. + const session = + tally !== null && !/^- Session: /m.test(text) + ? null + : match(/^- Session: (`+) ?(.*?) ?\1, round (\d+)\r?$/m); const score = /^- Score: (\d+)\/10,/m.exec(text); - const items = (prefix: RegExp) => text.split(/\r?\n/).filter((line) => prefix.test(line)).length; + const open = { high: 0, medium: 0, low: 0 }; + for (const [, severity] of text.matchAll(/^ - (High|Medium|Low), /gm)) + open[severity!.toLowerCase() as keyof typeof open]++; return { harness: reviewer[1]!, model: reviewer[2]!, effort: reviewer[3]!, - session: session[2]!, - round: Number(session[3]), + session: session && session[2]!, + round: session && Number(session[3]), score: score ? Number(score[1]) : null, - findings: items(/^ {2}- (High|Medium|Low), /), - fixed: items(/^ {2}- Fixed: /), - refuted: items(/^ {2}- Refuted: /), + tally, + open, }; }); } +/** A block's `Rounds:` line, or null without one; malformed or inconsistent counts fail. */ +function parseCounts(text: string): Counts | null { + const line = /^- Rounds: (.*?)\r?$/m.exec(text); + if (!line) return null; + const found = + /^(\d+); findings (\d+)(?: \(([^)]*)\))?: fixed (\d+), refuted (\d+)(?:, settled (\d+))?, open (\d+)$/.exec( + line[1]!, + ); + if (!found) throw new Error(`the ## Review record's Rounds line is malformed: ${line[1]}`); + const findings = { high: 0, medium: 0, low: 0 }; + for (const part of found[3]?.split(", ") ?? []) { + const [severity, count] = part.split(" "); + if (!SEVERITIES.includes(severity as keyof typeof findings) || !/^\d+$/.test(count ?? "")) + throw new Error(`the ## Review record's Rounds line is malformed: ${line[1]}`); + findings[severity as keyof typeof findings] = Number(count); + } + const counts = { + rounds: Number(found[1]), + findings, + fixed: Number(found[4]), + refuted: Number(found[5]), + settled: Number(found[6] ?? 0), + open: Number(found[7]), + }; + const total = SEVERITIES.reduce((sum, severity) => sum + findings[severity], 0); + if (total !== Number(found[2]) || counts.fixed + counts.refuted + counts.settled + counts.open !== total) + throw new Error(`the ## Review record's Rounds line does not add up: ${line[1]}`); + return counts; +} + /** `body` with `record` as its `## Review` section, which it gains when it has none; the rest stays as written. */ export function withReview(body: string, record: string): string { const section = reviewSection(body); @@ -174,11 +251,16 @@ export async function pullModels( } /** Record a run's `rounds` in the `## Review` section of their branch's open PR. The edit is the token's user's, which `/quest-merge` counts only when that user wrote the PR. */ -export async function recordRounds(root: string, rounds: Round[], deps: ClaimDependencies): Promise { +export async function recordRounds( + root: string, + rounds: Round[], + tallies: Tally[], + deps: ClaimDependencies, +): Promise { const [shared] = rounds; const pull = await openPull(root, shared!.base.ref, shared!.branch, deps); if ("skipped" in pull) return pull; - const next = withReview(pull.body, reviewRecord(rounds)); + const next = withReview(pull.body, reviewRecord(rounds, tallies)); if (next !== pull.body) await pull.client.request(pull.path, { method: "PATCH", body: { body: next } }); return { url: pull.url }; } diff --git a/src/cli/review.ts b/src/cli/review.ts index 71f98f9a..14e430f3 100644 --- a/src/cli/review.ts +++ b/src/cli/review.ts @@ -303,6 +303,28 @@ export interface ReviewDependencies { export const processDependencies = (): ReviewDependencies => ({ env: process.env }); +/** One reviewer's findings across every round on a branch, which a `--new` session continues. */ +export interface Tally { + reviewer: ReviewHarness; + /** Its latest round's model and effort. */ + model: string; + effort: string; + rounds: number; + /** + * Each finding it raised, once however many rounds keep it open, by its latest severity. One a replacing + * reviewer keeps open becomes that reviewer's, since its round reports it again unlinked to the earlier one. + */ + findings: Record<(typeof SEVERITIES)[number], number>; + fixed: number; + refuted: number; +} + +/** What a run of `quest review` gives: each reviewer's round, and every tally on the branch, the run's first. */ +export interface Reviewed { + rounds: Round[]; + tallies: Tally[]; +} + /** * A branch's review sessions, one per reviewer, kept in the clone's common Git directory so every worktree * resumes them. They are in the order the last run reported them, which numbers their findings. @@ -310,6 +332,7 @@ export const processDependencies = (): ReviewDependencies => ({ env: process.env interface State { branch: string; sessions: Session[]; + tallies: Tally[]; } /** One reviewer's session on a branch. */ @@ -1405,8 +1428,18 @@ export function parseState(value: unknown): State { typeof state === "object" && state !== null && Array.isArray(state.sessions) && + Array.isArray(state.tallies) && + state.tallies.every( + (tally) => + typeof tally?.reviewer === "string" && + typeof tally.rounds === "number" && + typeof tally.fixed === "number" && + typeof tally.refuted === "number" && + SEVERITIES.every((severity) => typeof tally.findings?.[severity] === "number"), + ) && state.sessions.every( (session) => + state.tallies!.some((tally) => tally.reviewer === session?.reviewer) && typeof session?.reviewer === "string" && typeof session.session === "string" && typeof session.root === "string" && @@ -1525,7 +1558,7 @@ function sandboxFor(root: string, env: NodeJS.ProcessEnv): Sandbox { * and `SECOND_REVIEWER`, or only `--harness`'s, each resuming its session on the branch unless `fresh`. * Every reviewer runs at once, and the first to fail stops the others and fails the whole, saving no round. */ -export async function review(options: ReviewOptions, deps: ReviewDependencies): Promise { +export async function review(options: ReviewOptions, deps: ReviewDependencies): Promise { // Native Windows installs the reviewer CLIs as batch shims, which `run` cannot start without a shell. if (process.platform === "win32") throw new ReviewInputError("quest review does not support native Windows yet; use WSL"); @@ -1559,8 +1592,8 @@ export async function review(options: ReviewOptions, deps: ReviewDependencies): const statePath = sessionFile(git(root, ["rev-parse", "--path-format=absolute", "--git-common-dir"]), branch); const dir = dirname(statePath); mkdirSync(dir, { recursive: true }); - const stored = options.fresh ? undefined : load(statePath); - const sessions = stored === undefined ? [] : parseState(stored).sessions; + const saved = savedState(statePath, options.fresh); + const sessions = options.fresh || saved === undefined ? [] : saved.sessions; // The last run numbered each session's findings after those of the sessions before it. const firsts = new Map(); let next = 1; @@ -1644,8 +1677,81 @@ export async function review(options: ReviewOptions, deps: ReviewDependencies): // The first failure stopped the others, so it is the one to report. if (controller.signal.aborted) throw controller.signal.reason; const done = results.map((result) => (result as PromiseFulfilledResult).value); - save(statePath, { branch, sessions: [...done.map((each) => each.session), ...kept] } satisfies State); - return done.map((each) => each.round); + const tallies = tallied( + options.fresh ? dropped(saved) : (saved?.tallies ?? []), + done.map((each, i) => ({ round: each.round, settles: seats[i]!.session?.reviewer })), + ); + save(statePath, { branch, sessions: [...done.map((each) => each.session), ...kept], tallies } satisfies State); + return { rounds: done.map((each) => each.round), tallies }; +} + +/** + * A branch's saved state, or undefined when it has none. `--new` drops the sessions but continues their tallies, + * less the findings it drops unsettled, so it reads them too, starting them over only when the state is one this quest cannot read, which is what + * `--new` is for. + */ +function savedState(path: string, fresh: boolean | undefined): State | undefined { + try { + const stored = load(path); + return stored === undefined ? undefined : parseState(stored); + } catch (error) { + if (fresh && error instanceof ReviewInputError) return undefined; + throw error; + } +} + +/** + * The saved tallies without the findings `--new` drops unsettled, so a finding no session can settle stays + * open nowhere, and one the new session reports again counts once. + */ +function dropped(saved: State | undefined): Tally[] { + const tallies = (saved?.tallies ?? []).map((tally) => ({ ...tally, findings: { ...tally.findings } })); + for (const session of saved?.sessions ?? []) { + const tally = tallies.find((each) => each.reviewer === session.reviewer)!; + for (const finding of session.rounds.at(-1)!.findings) tally.findings[finding.severity]--; + } + return tallies; +} + +/** + * `tallies` after a run's `rounds`, the run's reviewers first. A round counts its findings for its own reviewer + * and settles the last round's for the reviewer that raised them, `settles`, which is another when the round + * replaced it; one it keeps open moves to its own count, since the round reports it again. + */ +export function tallied(tallies: Tally[], rounds: { round: Round; settles?: ReviewHarness }[]): Tally[] { + const by = new Map(tallies.map((tally) => [tally.reviewer, { ...tally, findings: { ...tally.findings } }])); + for (const { round, settles } of rounds) { + if (round.unchanged) continue; + const settled = round.earlier?.findings ?? []; + if (settled.length > 0) { + const raiser = settles === undefined ? undefined : by.get(settles); + if (raiser === undefined) throw new Error(`no tally holds the findings ${round.reviewer}'s round settles`); + for (const finding of settled) { + if (finding.disposition === "fixed") raiser.fixed++; + else if (finding.disposition === "refuted") raiser.refuted++; + else raiser.findings[finding.severity]--; + } + } + const own = by.get(round.reviewer) ?? { + reviewer: round.reviewer, + model: round.model, + effort: round.effort, + rounds: 0, + findings: { high: 0, medium: 0, low: 0 }, + fixed: 0, + refuted: 0, + }; + own.model = round.model; + own.effort = round.effort; + own.rounds++; + for (const finding of round.findings) own.findings[finding.severity]++; + by.set(round.reviewer, own); + } + const run = rounds.map(({ round }) => round.reviewer); + return [ + ...run.flatMap((reviewer) => by.get(reviewer) ?? []), + ...[...by.values()].filter((tally) => !run.includes(tally.reviewer)), + ]; } /** What every reviewer of a run shares: the branch, its base, its change, and what a session is given. */ diff --git a/src/cli/usage.ts b/src/cli/usage.ts index fa4804a3..a95c2ec5 100644 --- a/src/cli/usage.ts +++ b/src/cli/usage.ts @@ -13,7 +13,8 @@ import { fixes } from "./land-fixer"; import { MERGER } from "./land-watch-fix"; import { type GitHub, type ObjectValue, type Repository, githubRepository, link, requireRemote } from "./github"; import { type OnlineReview, readOnlineReviews } from "./online-review"; -import { type ReviewSummary, parseReview } from "./review-record"; +import { type Counts, type ReviewSummary, findingsText, parseReview } from "./review-record"; +import { SEVERITIES } from "./review"; /** A session's tokens, and their cost at the model's API list price, which subscriptions do not bill. */ export interface Tokens { @@ -61,6 +62,8 @@ export interface PullRow { url: string; title: string; branch: string; + /** The head repository, lowercased as `owner/repo`, or null once a fork's is deleted. */ + repository: string | null; state: "open" | "closed" | "merged"; draft: boolean; /** Its body has a `## Waiting` section, so it was handed back to the maintainer. */ @@ -69,7 +72,7 @@ export interface PullRow { implementers: { harness: string; model: string; effort: string; size: Size | null }[]; /** The `Implementer:` items it cannot read, which count toward no implementer. */ unread: string[]; - /** Each reviewer's latest round, as the PR's `## Review` record names them. */ + /** Each reviewer the PR's `## Review` record names, with its latest round and its tally. */ review: ReviewSummary[]; /** Who made each mechanical fix the land fixer recorded, or `quest land watch` for a base merge it made alone. */ landFixes: string[]; @@ -98,16 +101,13 @@ export interface AccuracyRow { pulls: number; merged: number; handedBack: number; + /** PRs whose record tallies their reviews; a record written before tallies counts for none of what follows. */ reviewed: number; - /** The reviewers' latest rounds on these PRs that have a score, and the scores' sum. */ + /** The reviewers' latest rounds on the reviewed PRs that have a score, and the scores' sum. */ scored: number; score: number; - /** The sum of each reviewed PR's longest session's rounds. */ - rounds: number; - /** Sums over every reviewer's latest round. */ - findings: number; - fixed: number; - refuted: number; + /** Every reviewer's findings across every round on the reviewed PRs, and as `rounds`, each PR's most rounds. */ + tally: Counts; /** The land fixer's model-run fixes, leaving out the watcher's own base merges. */ landFixes: number; /** Later PRs naming these PRs under `## Caused by`. */ @@ -409,6 +409,24 @@ function causes(body: string, repository: Repository): number[] { const sized = (value: string | null | undefined): Size | null => (SIZES as readonly string[]).includes(value ?? "") ? (value as Size) : null; +/** + * A merged PR's reviews with its last round's open findings settled when all of them are low, since the review + * rule fixes or refutes those before merge without another round; nothing records which, so they count as settled. + */ +function settledOnMerge(reviews: ReviewSummary[], merged: boolean): ReviewSummary[] { + const settles = + merged && + reviews.every( + ({ tally, open }) => open.high === 0 && open.medium === 0 && (tally === null || tally.open === open.low), + ); + if (!settles) return reviews; + return reviews.map((review) => + review.tally === null + ? review + : { ...review, tally: { ...review.tally, settled: review.tally.settled + review.tally.open, open: 0 } }, + ); +} + /** Each PR's record, without its sessions yet. */ function pullRows(pulls: Pull[], repository: Repository): PullRow[] { const fixedBy = new Map(); @@ -422,6 +440,7 @@ function pullRows(pulls: Pull[], repository: Repository): PullRow[] { url: each.url, title: each.title, branch: each.branch, + repository: each.repository, state: each.state, draft: each.draft, handedBack: each.draft && handedBack(each.body), @@ -430,7 +449,7 @@ function pullRows(pulls: Pull[], repository: Repository): PullRow[] { fields ? [{ ...fields, size: sized(fields.size) }] : [], ), unread: own.implementers.filter(({ fields }) => fields === null).map(({ item }) => item), - review: parseReview(each.body), + review: settledOnMerge(parseReview(each.body), each.state === "merged"), landFixes: fixes(each.body), fixedBy: (fixedBy.get(each.number) ?? []).sort((a, b) => a - b), online: [], @@ -588,6 +607,23 @@ function spendRows(pulls: PullRow[]): SpendRow[] { function accuracyRows(pulls: PullRow[]): AccuracyRow[] { const rows = new Map(); + // A branch's saved tallies span every PR from it, so only one PR that has both a tally and an implementer + // to count it for counts them: the one whose record holds the latest tally, as the most rounds show, since + // each round rewrites only its own PR's record; the newest breaks a tie. A deleted fork has no repository, + // so its PR stands alone. + const branch = (each: PullRow) => JSON.stringify([each.repository ?? each.number, each.branch]); + const rounds = (each: PullRow) => each.review.reduce((sum, review) => sum + (review.tally?.rounds ?? 0), 0); + const latest = new Map(); + for (const each of pulls) { + if (each.implementers.length === 0 || each.review.every((review) => review.tally === null)) continue; + const other = latest.get(branch(each)); + if ( + other === undefined || + rounds(each) > rounds(other) || + (rounds(each) === rounds(other) && each.number > other.number) + ) + latest.set(branch(each), each); + } for (const each of pulls) { // A session that pushed twice is one implementer of the PR, whether or not each line names its size. const distinct = new Map( @@ -612,10 +648,14 @@ function accuracyRows(pulls: PullRow[]): AccuracyRow[] { reviewed: 0, scored: 0, score: 0, - rounds: 0, - findings: 0, - fixed: 0, - refuted: 0, + tally: { + rounds: 0, + findings: { high: 0, medium: 0, low: 0 }, + fixed: 0, + refuted: 0, + settled: 0, + open: 0, + }, landFixes: 0, fixes: 0, }; @@ -626,18 +666,22 @@ function accuracyRows(pulls: PullRow[]): AccuracyRow[] { if (each.handedBack) row.handedBack++; row.fixes += each.fixedBy.length; row.landFixes += each.landFixes.filter((by) => by !== MERGER).length; - if (each.review.length > 0) { + const tallied = + latest.get(branch(each)) === each ? each.review.filter((review) => review.tally !== null) : []; + if (tallied.length > 0) { row.reviewed++; - row.rounds += Math.max(...each.review.map((review) => review.round)); + row.tally.rounds += Math.max(...tallied.map((review) => review.tally!.rounds)); } - for (const review of each.review) { - if (review.score !== null) { + for (const { score, tally } of tallied) { + if (score !== null) { row.scored++; - row.score += review.score; + row.score += score; } - row.findings += review.findings; - row.fixed += review.fixed; - row.refuted += review.refuted; + for (const severity of SEVERITIES) row.tally.findings[severity] += tally!.findings[severity]; + row.tally.fixed += tally!.fixed; + row.tally.refuted += tally!.refuted; + row.tally.settled += tally!.settled; + row.tally.open += tally!.open; } } } @@ -718,7 +762,8 @@ export async function usage(root: string, options: UsageOptions, deps: UsageDepe const reviewers = new Map(); for (const row of [...allRows].sort((a, b) => a.number - b.number)) for (const review of row.review) - reviewers.set(review.session, { number: row.number, effort: review.effort }); + if (review.session !== null) + reviewers.set(review.session, { number: row.number, effort: review.effort }); const left = attribute(priced, sessionLogs(priced, deps), all, byNumber, reviewers); if (!narrowed) { unattributed = { sessions: left.length, ...zero() }; @@ -782,7 +827,7 @@ export function usageLines(report: Usage): string[] { for (const item of row.unread) lines.push(` unread: ${item}`); for (const r of row.review) lines.push( - ` review: ${r.harness}, ${r.model}, ${r.effort} effort; round ${r.round}, score ${r.score ?? "-"}/10, ${r.findings} open, last round ${r.fixed} fixed and ${r.refuted} refuted`, + ` review: ${r.harness}, ${r.model}, ${r.effort} effort; ${r.round === null ? "not in the latest run" : `round ${r.round}, score ${r.score ?? "-"}/10`}; ${r.tally === null ? "no tally" : `rounds ${r.tally.rounds}, ${findingsText(r.tally)}`}`, ); for (const o of row.online) lines.push(` online review: ${o.reviewer}, ${o.finalHead ? "" : "not "}at the final head; ${answers(o)}`); @@ -821,7 +866,7 @@ export function usageLines(report: Usage): string[] { a.harness, a.model, a.effort, - `${a.pulls} PRs, ${a.merged} merged, ${a.handedBack} handed back, ${a.reviewed} reviewed: score ${mean(a.score, a.scored)}, rounds ${mean(a.rounds, a.reviewed)}, ${a.findings} open, ${a.fixed} fixed, ${a.refuted} refuted; ${a.landFixes} land fixes, ${a.fixes} later fixes`, + `${a.pulls} PRs, ${a.merged} merged, ${a.handedBack} handed back, ${a.reviewed} reviewed: score ${mean(a.score, a.scored)}, rounds ${mean(a.tally.rounds, a.reviewed)}, ${findingsText(a.tally)}; ${a.landFixes} land fixes, ${a.fixes} later fixes`, ]), ), ); diff --git a/test/review.test.ts b/test/review.test.ts index 25c9cb63..df1d4e57 100644 --- a/test/review.test.ts +++ b/test/review.test.ts @@ -6,7 +6,10 @@ import { afterEach, expect, test, vi } from "vitest"; import { type Output, main } from "../src/cli/main"; import { REVIEWERS } from "../src/cli/models"; import { + type Finding, type Round, + type Settled, + type Tally, authored, baseInstructions, nextStep, @@ -15,6 +18,7 @@ import { runningHarness, parseState, sessionFile, + tallied, verdict, } from "../src/cli/review"; import { withReview } from "../src/cli/review-record"; @@ -446,7 +450,9 @@ fakes("a quest branch is reviewed at its quest's size on the base, which its own expect(sized.stdout).toContain("\nreviewer codex gpt-6.1-sol medium\n"); // A checkout on another branch, as the runner's review checkout is, takes the size of the branch it names. git(s.repo, "checkout", "-qb", "kix-review/job1"); - const [runner] = await review( + const { + rounds: [runner], + } = await review( { root: s.repo, base: "origin/main", sizeBranch: "quest/m0/big" }, { env: { ...process.env, CLAUDECODE: "1" } }, ); @@ -1290,7 +1296,15 @@ fakes("--new starts over, and a first round takes no responses", async () => { const fresh = await quest("--root", s.repo, "review", "--new", "--responses", responses); expect(fresh.code).toBe(2); expect(fresh.stderr).toContain("has none to resume"); - expect((await quest("--root", s.repo, "review", "--new")).stdout).toContain("round 1\n"); + const restarted = await quest("--root", s.repo, "review", "--new", "--json"); + expect(JSON.parse(restarted.stdout).rounds[0].round).toBe(1); + // The new sessions continue their reviewers' tallies, less the findings the dropped sessions left unsettled. + expect( + JSON.parse(restarted.stdout).tallies.map((t: Tally) => [t.reviewer, t.rounds, t.findings.high, t.fixed]), + ).toEqual([ + ["codex", 2, 1, 0], + ["opencode", 2, 1, 0], + ]); }); /** Commit `edit` to origin's `next` branch, leaving the checkout on `feature`; no flags change. */ @@ -1457,12 +1471,14 @@ fakes("each round rewrites the Review section of the branch's open PR, keeping t "- Reviewer: Codex, gpt-6.1-sol, medium effort", "- Session: `thread-1`, round 1", "- Score: 8/10, `Lands once the off-by-one is fixed.`", + "- Rounds: 1; findings 1 (high 1): fixed 0, refuted 0, open 1", "- Findings, open until the next round settles them:", " - High, `src/lib.ts:2`: `Off by one`", "", "- Reviewer: opencode, opencode-go/deepseek-v4.1-flash, high effort", "- Session: `ses_1`, round 1", "- Score: 8/10, `Lands once the off-by-one is fixed.`", + "- Rounds: 1; findings 1 (high 1): fixed 0, refuted 0, open 1", "- Findings, open until the next round settles them:", " - High, `src/lib.ts:2`: `Off by one`", "", @@ -1494,6 +1510,8 @@ fakes("each round rewrites the Review section of the branch's open PR, keeping t expect(record).toContain(`- Head: ${git(s.repo, "rev-parse", "HEAD").slice(0, 12)},`); expect(record).toContain("- Session: `thread-1`, round 2\n"); expect(record).toContain("- Score: 9/10, `Fixes #3.`\n"); + // The refuted finding counts beside the new one, across both rounds. + expect(record).toContain("- Rounds: 2; findings 2 (high 2): fixed 0, refuted 1, open 1\n"); expect(record).toContain( ` - High, \`src/lib.ts\`: \`\`Ping @someone about \`${base.slice(0, 12)}\` in #12\`\`\n`, ); @@ -1901,7 +1919,16 @@ test("saved sessions are read only in the current format, or refused naming --ne last: {}, given: { instructions: "i", rules: null }, }; - const current = { branch: "feature", sessions: [session] }; + const tally = { + reviewer: "codex", + model: "m", + effort: "e", + rounds: 1, + findings: { high: 0, medium: 0, low: 1 }, + fixed: 0, + refuted: 0, + }; + const current = { branch: "feature", sessions: [session], tallies: [tally] }; expect(parseState(current)).toEqual(current); for (const older of [ null, @@ -1913,6 +1940,10 @@ test("saved sessions are read only in the current format, or refused naming --ne { sessions: [{ ...session, given: undefined }] }, { sessions: [{ ...session, session: undefined }] }, { sessions: [{ ...session, root: undefined }] }, + // Saved before tallies, or with a session no tally counts. + { branch: "feature", sessions: [session] }, + { branch: "feature", sessions: [session], tallies: [] }, + { branch: "feature", sessions: [session], tallies: [{ ...tally, findings: { high: 0, low: 1 } }] }, ]) expect(() => parseState(older)).toThrow(/--new/); }); @@ -2002,4 +2033,107 @@ fakes("a new size pick settles the open findings of the reviewer it replaces", a ]); const review = s.calls().find((call) => call.argv.includes("--session-id"))!; expect(review.stdin).toContain("Fixed at src/lib.ts:1."); + // Sol's finding stays Sol's, fixed, while the pick that replaced it counts its own. + expect( + JSON.parse(round.stdout).tallies.map((t: Tally) => [t.reviewer, t.rounds, t.findings.high, t.fixed]), + ).toEqual([ + ["claude", 1, 1, 0], + ["opencode", 2, 0, 0], + ["codex", 1, 1, 1], + ]); +}); + +test("a tally counts each finding once, by who raised it, through every round", () => { + const finding = (severity: Finding["severity"], title: string): Finding => ({ + file: "a.ts", + line: 1, + severity, + title, + body: "b", + }); + const settled = (f: Finding, disposition: Settled["disposition"], number: number): Settled => ({ + ...f, + number, + disposition, + reason: "r", + }); + const base = { + model: "m", + effort: "high", + branch: "feature", + size: "M", + sizeFrom: "default", + session: "s", + base: { ref: "origin/main", sha: "a".repeat(40) }, + head: "b".repeat(40), + usage: [], + score: 8, + rationale: "r", + earlier: null, + unchanged: false, + } satisfies Omit; + const [a, b, c] = [finding("high", "a"), finding("medium", "b"), finding("low", "c")]; + const first = tallied([], [{ round: { ...base, reviewer: "codex", round: 1, findings: [a, b, c] } }]); + // A kept-open finding is reported again, and its new severity replaces its old one. + const second = tallied(first, [ + { + round: { + ...base, + reviewer: "codex", + round: 2, + findings: [{ ...b, severity: "high" }], + earlier: { + head: base.head, + findings: [settled(a, "fixed", 1), settled(b, "open", 2), settled(c, "refuted", 3)], + }, + }, + settles: "codex", + }, + ]); + expect(second).toEqual([ + { + reviewer: "codex", + model: "m", + effort: "high", + rounds: 2, + findings: { high: 2, medium: 0, low: 1 }, + fixed: 1, + refuted: 1, + }, + ]); + expect(first[0]!.rounds).toBe(1); + // A round that repeats the last one counts nothing; a replacing reviewer takes over what it keeps open. + const third = tallied(second, [ + { round: { ...base, reviewer: "codex", round: 2, findings: [], unchanged: true } }, + { + round: { + ...base, + reviewer: "claude", + round: 1, + findings: [{ ...b, severity: "high" }], + earlier: { head: base.head, findings: [settled({ ...b, severity: "high" }, "open", 1)] }, + }, + settles: "codex", + }, + ]); + expect(third.map((t) => [t.reviewer, t.rounds, t.findings, t.fixed, t.refuted])).toEqual([ + ["codex", 2, { high: 1, medium: 0, low: 1 }, 1, 1], + ["claude", 1, { high: 1, medium: 0, low: 0 }, 0, 0], + ]); + expect(() => + tallied( + [], + [ + { + round: { + ...base, + reviewer: "claude", + round: 1, + findings: [], + earlier: { head: base.head, findings: [settled(a, "fixed", 1)] }, + }, + }, + ], + ), + ).toThrow(/no tally/); }); diff --git a/test/usage.test.ts b/test/usage.test.ts index 61877754..b129583a 100644 --- a/test/usage.test.ts +++ b/test/usage.test.ts @@ -6,7 +6,7 @@ import { describe, expect, test } from "vitest"; import { implementer } from "../src/github/models"; import { fixes } from "../src/cli/land-fixer"; -import type { Round } from "../src/cli/review"; +import type { Round, Tally } from "../src/cli/review"; import { parseReview, reviewRecord } from "../src/cli/review-record"; import { type Priced, parseCcusage, readClaudeLog, readCodexLog, usage, usageLines } from "../src/cli/usage"; import { tempDir } from "./fixture"; @@ -217,8 +217,30 @@ describe("PR records", () => { earlier: null, }; - test("reads back each reviewer's round quest review records, after the land fixer's lines", () => { - const body = `Summary.\n\n## Review\n\nmechanical fix by codex, gpt-6.1-sol, medium effort: \`merged\` (head ${"d".repeat(40)})\n\n${reviewRecord([round, second])}\n\n## Models\n\n- Size: L\n`; + const tally = ( + reviewer: Tally["reviewer"], + rounds: number, + findings: Tally["findings"], + fixed: number, + refuted: number, + ): Tally => ({ + reviewer, + model: "m", + effort: "high", + rounds, + findings, + fixed, + refuted, + }); + const tallies = [ + tally("codex", 3, { high: 1, medium: 1, low: 2 }, 2, 1), + tally("opencode", 1, { high: 0, medium: 0, low: 0 }, 0, 0), + tally("claude", 2, { high: 2, medium: 0, low: 0 }, 1, 0), + ]; + + test("reads back each reviewer's round and tally quest review records, after the land fixer's lines", () => { + const body = `Summary.\n\n## Review\n\nmechanical fix by codex, gpt-6.1-sol, medium effort: \`merged\` (head ${"d".repeat(40)})\n\n${reviewRecord([round, second], tallies)}\n\n## Models\n\n- Size: L\n`; + expect(body).toContain("\n- Rounds: 3; findings 4 (high 1, medium 1, low 2): fixed 2, refuted 1, open 1\n"); expect(parseReview(body)).toEqual([ { harness: "Codex", @@ -227,9 +249,15 @@ describe("PR records", () => { session: THREAD, round: 3, score: 7, - findings: 1, - fixed: 2, - refuted: 1, + tally: { + rounds: 3, + findings: { high: 1, medium: 1, low: 2 }, + fixed: 2, + refuted: 1, + settled: 0, + open: 1, + }, + open: { high: 0, medium: 1, low: 0 }, }, { harness: "opencode", @@ -238,18 +266,66 @@ describe("PR records", () => { session: "ses_1", round: 1, score: 10, - findings: 0, - fixed: 0, - refuted: 0, + tally: { + rounds: 1, + findings: { high: 0, medium: 0, low: 0 }, + fixed: 0, + refuted: 0, + settled: 0, + open: 0, + }, + open: { high: 0, medium: 0, low: 0 }, + }, + // A reviewer the run left out, such as the pick a new one replaced, keeps its tally. + { + harness: "Claude Code", + model: "m", + effort: "high", + session: null, + round: null, + score: null, + tally: { + rounds: 2, + findings: { high: 2, medium: 0, low: 0 }, + fixed: 1, + refuted: 0, + settled: 0, + open: 1, + }, + open: { high: 0, medium: 0, low: 0 }, }, ]); // An open finding keeps the head short; once no reviewer reports one, it is named in full. expect(body).not.toContain("b".repeat(40)); - const clean = reviewRecord([{ ...round, findings: [] }, second]); - expect(clean).toContain(`- Head: ${"b".repeat(40)}\n`); + // Claude's left-out tally still holds an open finding, which keeps it short too. + const clean = [{ ...round, findings: [] }, second]; + expect(reviewRecord(clean, tallies)).not.toContain("b".repeat(40)); + const settledAll = [tally("codex", 3, { high: 1, medium: 1, low: 2 }, 3, 1), tallies[1]!]; + expect(reviewRecord(clean, settledAll)).toContain(`- Head: ${"b".repeat(40)}\n`); expect(fixes(body)).toEqual(["codex, gpt-6.1-sol, medium effort"]); }); + test("a record written before tallies has none, and a Rounds line that does not add up fails", () => { + const record = reviewRecord([second], [tally("opencode", 1, { high: 0, medium: 0, low: 0 }, 0, 0)]); + const older = `## Review\n\n${record.replace(/^- Rounds: .*\n/m, "")}\n`; + expect(parseReview(older)[0]!.tally).toBeNull(); + // The settled count `quest usage` prints reads back. + const settled = "1; findings 2 (low 2): fixed 1, refuted 0, settled 1, open 0"; + expect( + parseReview(`## Review\n\n${record.replace(/^- Rounds: .*$/m, `- Rounds: ${settled}`)}\n`)[0]!.tally, + ).toEqual({ rounds: 1, findings: { high: 0, medium: 0, low: 2 }, fixed: 1, refuted: 0, settled: 1, open: 0 }); + for (const line of [ + "1; findings 1 (high 1): fixed 0, refuted 0, open 0", + "1; findings 2 (high 1): fixed 1, refuted 0, open 1", + "1; findings 2 (low 2): fixed 1, refuted 0, settled 1, open 1", + "1; findings 1 (severe 1): fixed 1, refuted 0, open 0", + "one", + ]) + expect(() => + parseReview(`## Review\n\n${record.replace(/^- Rounds: .*$/m, `- Rounds: ${line}`)}\n`), + ).toThrow(/Rounds line/); + }); + test("a skipped review or a fixer-only section has no round", () => { expect(parseReview("## Review\n\nSkipped: too trivial to need one.\n")).toEqual([]); expect(parseReview("## Review\n\nmechanical fix by quest land watch: `merged main` (head x)\n")).toEqual([]); @@ -375,15 +451,26 @@ describe("usage", () => { "- Reviewer: Codex, gpt-6.1-sol, high effort", `- Session: \`${THREAD}\`, round 2`, "- Score: 9/10, `Fine.`", + "- Rounds: 2; findings 2 (high 1, low 1): fixed 1, refuted 1, open 0", "- Findings: none", "", "- Reviewer: opencode, opencode-go/deepseek-v4.1-flash, high effort", "- Session: `ses_1`, round 1", "- Score: 8/10, `Fine.`", + "- Rounds: 1; findings 0: fixed 0, refuted 0, open 0", "- Findings: none", + "", + // A reviewer the latest run left out, such as one a new size pick replaced, has only its tally. + "- Reviewer: Claude Code, claude-opus-5-5, high effort", + "- Rounds: 1; findings 1 (medium 1): fixed 1, refuted 0, open 0", ].join("\n"); // #9's claim of the same branch named ses_1 before #10 resumed it, so #10, the newer, takes it. - const body9 = body10.slice(body10.indexOf("## Review")); + // Its record and #10's tally the same branch, so the rollup counts the tally once, from #10. + const body9 = `${body10.slice(body10.indexOf("## Review"))}\n\n## Models\n\n- Size: L\n- Implementer: Claude Code, claude-opus-5-5, high effort\n`; + const fork13 = body10 + .slice(body10.indexOf("## Review")) + .replaceAll(THREAD, "thread-fork") + .replaceAll("ses_1", "ses_fork"); const body11 = "Fix.\n\n## Caused by\n\n- [acme/proj#10](https://github.com/acme/proj/pull/10) - missed a case\n"; const pulls = [ @@ -397,13 +484,54 @@ describe("usage", () => { }), // A fork's PR whose head shares #10's branch name, opened earlier and still open: no session here is // shown to be its. - pr(7, "quest/m0/a", "", "2026-11-03T00:00:00Z", { + // Its tally is the fork's own, so it counts beside #10's. + pr(7, "quest/m0/a", body9, "2026-11-03T00:00:00Z", { state: "open", merged_at: null, closed_at: null, updated_at: "2026-11-06T00:00:00Z", head: { ref: "quest/m0/a", repo: { full_name: "fork/proj" } }, }), + // The fork's newer PR has no implementer to count its tally for, so #7 still counts the fork's. + // Its own sessions, so it takes none of #10's. + pr(13, "quest/m0/a", fork13, "2026-11-04T00:00:00Z", { + state: "open", + merged_at: null, + closed_at: null, + updated_at: "2026-11-06T00:00:00Z", + head: { ref: "quest/m0/a", repo: { full_name: "fork/proj" } }, + }), + // Two deleted forks' merged PRs from a branch of the same name: neither names its repository, so each + // counts. Each merged with one finding open: #5's last round's all low, so merging settled it, and + // #6's medium, so it stays open. + ...[ + [5, "low 1): fixed 1, refuted 0, open 1", "Low"], + [6, "medium 1): fixed 1, refuted 0, open 1", "Medium"], + ].map(([number, counts, severity]) => + pr( + number as number, + "quest/m0/a", + body9.replace( + "low 1): fixed 1, refuted 1, open 0\n- Findings: none", + `${counts}\n- Findings, open until the next round settles them:\n - ${severity}, \`a:1\`: \`x\``, + ), + "2026-11-02T00:00:00Z", + { updated_at: "2026-11-06T00:00:00Z", head: { ref: "quest/m0/a", repo: null } }, + ), + ), + // Two open PRs from another branch, whose older record holds the later tally (3 Sol rounds against + // 2), as when a round rewrote only its own PR's record: the older counts, not the stale newer one. + ...[ + [3, body9.replace("- Rounds: 2;", "- Rounds: 3;")], + [4, body9], + ].map(([number, body]) => + pr(number as number, "quest/m0/b", body as string, "2026-11-02T00:00:00Z", { + state: "open", + merged_at: null, + closed_at: null, + updated_at: "2026-11-06T00:00:00Z", + }), + ), pr(8, "old", "", "2026-10-01T00:00:00Z", { updated_at: "2026-10-01T00:00:00Z" }), ]; const requests: string[] = []; @@ -526,7 +654,7 @@ describe("usage", () => { test("ties each session to its PR and role, and rolls up spend and accuracy", async () => { const { root, deps, priced } = setup(); const report = await usage(root, { since: "2026-11-05" }, deps(priced)); - expect(report.pulls.map((p) => p.number)).toEqual([11, 10, 9, 7]); + expect(report.pulls.map((p) => p.number)).toEqual([11, 10, 9, 7, 13, 5, 6, 3, 4]); const ten = report.pulls.find((p) => p.number === 10)!; expect(ten.sessions.map((s) => [s.harness, s.session, s.role, s.effort])).toEqual([ ["claude", CLAUDE, "implementing", "high"], @@ -561,9 +689,31 @@ describe("usage", () => { ["L", "reviewing", "codex", 1], ["L", "reviewing", "opencode", 4], ]); - expect(report.accuracy.map((a) => [a.size, a.model, a.pulls, a.reviewed, a.score, a.rounds, a.fixes])).toEqual([ - ["S", "gpt-6.1-sol", 1, 1, 17, 2, 1], - ["L", "claude-opus-5-5", 1, 1, 17, 2, 1], + const tally = { + rounds: 2, + findings: { high: 1, medium: 1, low: 1 }, + fixed: 2, + refuted: 1, + settled: 0, + open: 0, + }; + // #10's, #7's, each deleted fork's, and #3's. + const five = { + rounds: 11, + findings: { high: 5, medium: 6, low: 4 }, + fixed: 10, + refuted: 3, + settled: 1, + open: 1, + }; + const codex = (n: number) => report.pulls.find((p) => p.number === n)!.review[0]!.tally; + expect([codex(5), codex(6)].map((t) => [t!.settled, t!.open])).toEqual([ + [1, 0], + [0, 1], + ]); + expect(report.accuracy.map((a) => [a.size, a.model, a.pulls, a.reviewed, a.score, a.tally, a.fixes])).toEqual([ + ["S", "gpt-6.1-sol", 1, 1, 17, tally, 1], + ["L", "claude-opus-5-5", 7, 5, 85, five, 1], ]); expect(ten.online).toEqual([ { @@ -576,11 +726,11 @@ describe("usage", () => { unanswered: 0, }, ]); - // #11 and #10 merged; #9 closed and #7 is open, so neither counts. + // #11, #10, #5, and #6 merged; #9 closed and #7 is open, so neither counts. expect(report.online).toEqual([ { reviewer: "rabbit[bot]", - merged: 2, + merged: 4, reviewed: 1, finalHead: 1, findings: 1, @@ -591,12 +741,17 @@ describe("usage", () => { }, ]); const text = usageLines(report).join("\n"); + expect(text).toContain( + " review: Claude Code, claude-opus-5-5, high effort; not in the latest run; rounds 1, findings 1 (medium 1): fixed 1, refuted 0, open 0\n", + ); + expect(text).toContain("rounds 2.0, findings 3 (high 1, medium 1, low 1): fixed 2, refuted 1, open 0;"); expect(text).toContain("#10 L merged: PR 10 (quest/m0/a)"); + expect(text).toContain("findings 2 (high 1, low 1): fixed 1, refuted 0, settled 1, open 0\n"); expect(text).toContain( "online review: rabbit[bot], at the final head; 1 finding: 1 accepted, 0 declined, 0 unclassified, 0 unanswered", ); expect(text).toContain( - "rabbit[bot] reviewed 1/2 (50%), final head 1 (50%), 1.0 findings per PR reviewed, 100% accepted;", + "rabbit[bot] reviewed 1/4 (25%), final head 1 (25%), 1.0 findings per PR reviewed, 100% accepted;", ); expect(text).toContain("fixed later by: #11"); expect(text).not.toContain("SECRET"); @@ -618,7 +773,10 @@ describe("usage", () => { [10, 0], [9, 0], [7, 0], + [13, 0], + [5, 0], + [6, 0], ]); - expect(quest.pulls[0]!.review.map((r) => r.score)).toEqual([9, 8]); + expect(quest.pulls[0]!.review.map((r) => r.score)).toEqual([9, 8, null]); }); });