From 8db32008981165c62727ad3d098fbbf86178253f Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 11 Nov 2026 17:18:48 -0800 Subject: [PATCH 1/5] quest: plan cost-against-quality charts for the model check and the routing readout Co-Authored-By: Claude Opus 5.5 --- quest/m0/routing-evidence/README.md | 3 ++ quest/m0/routing-evidence/charts.md | 68 +++++++++++++++++++++++++++++ 2 files changed, 71 insertions(+) create mode 100644 quest/m0/routing-evidence/charts.md diff --git a/quest/m0/routing-evidence/README.md b/quest/m0/routing-evidence/README.md index 997be260..4ce51b82 100644 --- a/quest/m0/routing-evidence/README.md +++ b/quest/m0/routing-evidence/README.md @@ -54,6 +54,9 @@ Readout, the agent's choices for the implementer to revisit: chooses to. - Carry the tallies into [the model check](/quest/m1/model-check.md)'s baseline. +- Publish the arms' cost and quality as the charts + [cost against quality](/quest/m0/routing-evidence/charts.md) describes, + and link them with the recommendations. Baseline from the 2026-11-11 audit, at API list price: diff --git a/quest/m0/routing-evidence/charts.md b/quest/m0/routing-evidence/charts.md new file mode 100644 index 00000000..6635c28d --- /dev/null +++ b/quest/m0/routing-evidence/charts.md @@ -0,0 +1,68 @@ +# [M] Chart cost against quality for every model and effort + +## Goal + +Each [model check](/quest/m1/model-check.md) run, and this epic's +readout, publishes charts of cost against quality for every writer and +reviewer model and effort `quest` runs, and links them in its PR, so a +routing change is read off the cost/quality frontier rather than tables: + +- Writers: cost per merged PR against first-round findings, review rounds, + hand-backs, and later `## Caused by` escapes, one point per size, model, + and effort. +- Reviewers: cost per review against findings raised and the share fixed or + accepted, for the local reviewers and the online ones, which `quest usage` + reports since [moq-dev/quest#642](https://github.com/moq-dev/quest/pull/642). +- Beside our points, the published benchmark points for the same models and + efforts, so a cell where our work departs from the benchmarks stands out. + +Non-goal: a live dashboard. No chart code ships in `quest`. + +## Plan + +Decided 2026-11-11 (user, /quest-plan in a /quest-run session, when picking +[moq-dev/quest#642](https://github.com/moq-dev/quest/pull/642) to merge): + +- Goal? ✅ Charts for the model check and the readout, the two places + routing changes (rejected: a live dashboard; benchmarks only) +- Which data? ✅ Our measured points from `quest usage --json` with published + benchmark points beside them (rejected: ours only; benchmarks only) +- Where do they live? ✅ An artifact per run: the agent running the check or + the readout renders a private claude.ai page from `quest usage --json` and + the benchmark table, and links it in its PR, so the core stays + independent of a harness (rejected: a `quest usage --html` report, which + adds chart code and tests to the CLI; token-max, which knows no PR + quality signals) +- Milestone? ✅ m0, in this epic (rejected: m1) + +The benchmarks behind the XS writer change, for the first table +([writer A/B](/quest/m0/routing-evidence/writer-ab.md) records the decision): + +- Anthropic's + [cost guide](https://platform.claude.com/docs/en/about-claude/models/optimizing-for-cost-and-intelligence), + SWE-bench Pro subset: Opus 5.5 low 87.4% and medium 92.8%, Sonnet 5.5 + high 87.6% with medium about 7 points lower, at about $0.11 per solved + task for Opus low and Sonnet high. +- [Artificial Analysis](https://artificialanalysis.ai/models/comparisons/claude-sonnet-5-5-high-vs-claude-opus-5-5-low), + Terminal-Bench 4.0 and cost per task: Opus 5.5 low 31.3%, $0.98; Sonnet + 5.5 high 43.9%, $1.64; Sonnet 5.5 medium 29.8%, $0.78; Haiku 5.5 xhigh + 29.3%, $0.25. + +Agent's choices, for the implementer to revisit: + +- Keep the benchmark points as a table in the model check's Plan, where its + existing benchmark notes already live, with each row's source, effort, and + date; each run refreshes it. A row without a stated effort stays out. +- Write the chart procedure into the model check's Plan and the epic's + readout: which `quest usage --json` fields feed each axis, how cells with + too few PRs are marked rather than dropped, and that each chart names its + date range and PR counts. +- Make the first charts from the data on main when this lands, link them in + this quest's PR, and say which cells are still too thin. + +## Required + +- [Review tallies](/quest/m0/routing-evidence/review-tallies.md) - the + findings each reviewer raised across rounds +- [usage through token-max](/quest/m0/routing-evidence/usage-token-max.md) - + the cost axis, which today misses sub-agent transcripts, 72% of host spend From 6c3c447cdb69686f5adaaf07f8a8cfaedbd87ef0 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 11 Nov 2026 17:29:49 -0800 Subject: [PATCH 2/5] quest: chart axes the data has, and benchmarks on charts of their own Co-Authored-By: Claude Opus 5.5 --- quest/m0/routing-evidence/README.md | 4 ++-- quest/m0/routing-evidence/charts.md | 29 ++++++++++++++++++++--------- 2 files changed, 22 insertions(+), 11 deletions(-) diff --git a/quest/m0/routing-evidence/README.md b/quest/m0/routing-evidence/README.md index 4ce51b82..ada1c297 100644 --- a/quest/m0/routing-evidence/README.md +++ b/quest/m0/routing-evidence/README.md @@ -54,8 +54,8 @@ Readout, the agent's choices for the implementer to revisit: chooses to. - Carry the tallies into [the model check](/quest/m1/model-check.md)'s baseline. -- Publish the arms' cost and quality as the charts - [cost against quality](/quest/m0/routing-evidence/charts.md) describes, +- Publish the arms' cost and quality as + [the cost-against-quality charts](/quest/m0/routing-evidence/charts.md) quest describes, and link them with the recommendations. Baseline from the 2026-11-11 audit, at API list price: diff --git a/quest/m0/routing-evidence/charts.md b/quest/m0/routing-evidence/charts.md index 6635c28d..ede2abe7 100644 --- a/quest/m0/routing-evidence/charts.md +++ b/quest/m0/routing-evidence/charts.md @@ -7,14 +7,20 @@ readout, publishes charts of cost against quality for every writer and reviewer model and effort `quest` runs, and links them in its PR, so a routing change is read off the cost/quality frontier rather than tables: -- Writers: cost per merged PR against first-round findings, review rounds, +- Writers: cost per merged PR against the findings raised across its review + rounds (the review tallies' per-reviewer totals), review rounds, hand-backs, and later `## Caused by` escapes, one point per size, model, and effort. -- Reviewers: cost per review against findings raised and the share fixed or - accepted, for the local reviewers and the online ones, which `quest usage` - reports since [moq-dev/quest#642](https://github.com/moq-dev/quest/pull/642). -- Beside our points, the published benchmark points for the same models and - efforts, so a cell where our work departs from the benchmarks stands out. +- Local reviewers: cost per review against findings raised and the share + fixed. +- Online reviewers, which run on flat plans and have no cost per review: + coverage against findings per PR and acceptance, which `quest usage` + reports since [moq-dev/quest#642](https://github.com/moq-dev/quest/pull/642), + on a chart of their own. +- Published benchmarks: one chart per benchmark suite, score against its + own cost per task, for the same models and efforts. Benchmark and PR + metrics differ, so compare the rank order of cells across the charts, + not their values; a cell whose rank differs stands out. Non-goal: a live dashboard. No chart code ships in `quest`. @@ -30,13 +36,18 @@ Decided 2026-11-11 (user, /quest-plan in a /quest-run session, when picking - Where do they live? ✅ An artifact per run: the agent running the check or the readout renders a private claude.ai page from `quest usage --json` and the benchmark table, and links it in its PR, so the core stays - independent of a harness (rejected: a `quest usage --html` report, which + independent of a harness. The page is for the maintainer, who picks + routing; the PR body also carries the plotted numbers as tables, so a + reviewer who cannot open the page can check them (rejected: a `quest usage --html` report, which adds chart code and tests to the CLI; token-max, which knows no PR quality signals) - Milestone? ✅ m0, in this epic (rejected: m1) -The benchmarks behind the XS writer change, for the first table -([writer A/B](/quest/m0/routing-evidence/writer-ab.md) records the decision): +The benchmarks behind the XS writer change to Sonnet 5.5 high +([moq-dev/quest#646](https://github.com/moq-dev/quest/pull/646)) and the XS +arm of the [writer A/B](/quest/m0/routing-evidence/writer-ab.md) +([moq-dev/quest#644](https://github.com/moq-dev/quest/pull/644)), both +decided on 2026-11-11, for the first table: - Anthropic's [cost guide](https://platform.claude.com/docs/en/about-claude/models/optimizing-for-cost-and-intelligence), From 6640aad2670cb45562ba8af3d6154bba028d0997 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 11 Nov 2026 17:37:18 -0800 Subject: [PATCH 3/5] quest: chart the readout's own measures, and define the rank comparison Co-Authored-By: Claude Opus 5.5 --- quest/m0/routing-evidence/charts.md | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/quest/m0/routing-evidence/charts.md b/quest/m0/routing-evidence/charts.md index ede2abe7..394ce359 100644 --- a/quest/m0/routing-evidence/charts.md +++ b/quest/m0/routing-evidence/charts.md @@ -1,4 +1,4 @@ -# [M] Chart cost against quality for every model and effort +# [M] Chart routing's cost and quality for every model and effort ## Goal @@ -7,10 +7,10 @@ readout, publishes charts of cost against quality for every writer and reviewer model and effort `quest` runs, and links them in its PR, so a routing change is read off the cost/quality frontier rather than tables: -- Writers: cost per merged PR against the findings raised across its review - rounds (the review tallies' per-reviewer totals), review rounds, - hand-backs, and later `## Caused by` escapes, one point per size, model, - and effort. +- Writers: cost per merged PR against the quality measures the epic's + readout compares for the writer arms (findings, review rounds, + hand-backs, and later `## Caused by` escapes), read from the review + tallies, one point per size, model, and effort. - Local reviewers: cost per review against findings raised and the share fixed. - Online reviewers, which run on flat plans and have no cost per review: @@ -19,8 +19,10 @@ routing change is read off the cost/quality frontier rather than tables: on a chart of their own. - Published benchmarks: one chart per benchmark suite, score against its own cost per task, for the same models and efforts. Benchmark and PR - metrics differ, so compare the rank order of cells across the charts, - not their values; a cell whose rank differs stands out. + metrics differ, so the comparison is a table beside the charts: each + model and effort cell's rank by quality per dollar on each benchmark and + on our PRs, for the cells both cover. A cell whose rank on our PRs + differs from its benchmark ranks stands out. Non-goal: a live dashboard. No chart code ships in `quest`. From ecce4dcd4893d793134ffa8df7566d9b9b8cacfd Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 11 Nov 2026 17:40:32 -0800 Subject: [PATCH 4/5] quest: match the readout's first-round measure and rank per size Co-Authored-By: Claude Opus 5.5 --- quest/m0/routing-evidence/charts.md | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/quest/m0/routing-evidence/charts.md b/quest/m0/routing-evidence/charts.md index 394ce359..864d406a 100644 --- a/quest/m0/routing-evidence/charts.md +++ b/quest/m0/routing-evidence/charts.md @@ -8,9 +8,10 @@ reviewer model and effort `quest` runs, and links them in its PR, so a routing change is read off the cost/quality frontier rather than tables: - Writers: cost per merged PR against the quality measures the epic's - readout compares for the writer arms (findings, review rounds, - hand-backs, and later `## Caused by` escapes), read from the review - tallies, one point per size, model, and effort. + readout compares for the writer arms (first-round findings, review + rounds, hand-backs, and later `## Caused by` escapes), from the same + `quest usage --json` records the readout reads, one point per size, + model, and effort. - Local reviewers: cost per review against findings raised and the share fixed. - Online reviewers, which run on flat plans and have no cost per review: @@ -19,9 +20,10 @@ routing change is read off the cost/quality frontier rather than tables: on a chart of their own. - Published benchmarks: one chart per benchmark suite, score against its own cost per task, for the same models and efforts. Benchmark and PR - metrics differ, so the comparison is a table beside the charts: each - model and effort cell's rank by quality per dollar on each benchmark and - on our PRs, for the cells both cover. A cell whose rank on our PRs + metrics differ, so the comparison is a table beside the charts, per PR + size: each model and effort cell's rank on each benchmark by score per + dollar, and on our PRs by first-round findings per dollar, for the cells + both cover. A cell whose rank on our PRs differs from its benchmark ranks stands out. Non-goal: a live dashboard. No chart code ships in `quest`. From 4994823f36a906d926f05338752bd94d3d11e619 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 11 Nov 2026 17:41:30 -0800 Subject: [PATCH 5/5] quest: separate cost and quality axes, and source the first-round findings Co-Authored-By: Claude Opus 5.5 --- quest/m0/routing-evidence/charts.md | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/quest/m0/routing-evidence/charts.md b/quest/m0/routing-evidence/charts.md index 864d406a..6ccfed08 100644 --- a/quest/m0/routing-evidence/charts.md +++ b/quest/m0/routing-evidence/charts.md @@ -7,11 +7,10 @@ readout, publishes charts of cost against quality for every writer and reviewer model and effort `quest` runs, and links them in its PR, so a routing change is read off the cost/quality frontier rather than tables: -- Writers: cost per merged PR against the quality measures the epic's +- Writers: cost per merged PR against each quality measure the epic's readout compares for the writer arms (first-round findings, review - rounds, hand-backs, and later `## Caused by` escapes), from the same - `quest usage --json` records the readout reads, one point per size, - model, and effort. + rounds, hand-backs, and later `## Caused by` escapes), one point per + size, model, and effort, with cost and quality as separate axes. - Local reviewers: cost per review against findings raised and the share fixed. - Online reviewers, which run on flat plans and have no cost per review: @@ -20,11 +19,9 @@ routing change is read off the cost/quality frontier rather than tables: on a chart of their own. - Published benchmarks: one chart per benchmark suite, score against its own cost per task, for the same models and efforts. Benchmark and PR - metrics differ, so the comparison is a table beside the charts, per PR - size: each model and effort cell's rank on each benchmark by score per - dollar, and on our PRs by first-round findings per dollar, for the cells - both cover. A cell whose rank on our PRs - differs from its benchmark ranks stands out. + metrics differ, so compare where each cell sits on each chart's + frontier rather than its values; a cell on the frontier for the + benchmarks but not for our PRs, or the reverse, stands out. Non-goal: a live dashboard. No chart code ships in `quest`. @@ -72,6 +69,10 @@ Agent's choices, for the implementer to revisit: readout: which `quest usage --json` fields feed each axis, how cells with too few PRs are marked rather than dropped, and that each chart names its date range and PR counts. +- `quest usage --json` reports each reviewer's latest round, and + [review tallies](/quest/m0/routing-evidence/review-tallies.md) add totals + across rounds. If the tally line does not also record the first round's + findings, which the readout compares, add them there. - Make the first charts from the data on main when this lands, link them in this quest's PR, and say which cells are still too thin.