Repository navigation
quest: plan cost-against-quality charts for the model check and the routing readout #647
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
8db3200
6c3c447
6640aad
ecce4dc
4994823
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,84 @@ | ||
| # [M] Chart routing's cost and quality for every model and effort | ||
|
|
||
| ## Goal | ||
|
|
||
| Each [model check](/quest/m1/model-check.md) run, and this epic's | ||
| readout, publishes charts of cost against quality for every writer and | ||
| reviewer model and effort `quest` runs, and links them in its PR, so a | ||
| routing change is read off the cost/quality frontier rather than tables: | ||
|
|
||
| - Writers: cost per merged PR against each quality measure the epic's | ||
| readout compares for the writer arms (first-round findings, review | ||
| rounds, hand-backs, and later `## Caused by` escapes), one point per | ||
| size, model, and effort, with cost and quality as separate axes. | ||
| - Local reviewers: cost per review against findings raised and the share | ||
| fixed. | ||
| - Online reviewers, which run on flat plans and have no cost per review: | ||
| coverage against findings per PR and acceptance, which `quest usage` | ||
| reports since [moq-dev/quest#642](https://github.com/moq-dev/quest/pull/642), | ||
| on a chart of their own. | ||
| - Published benchmarks: one chart per benchmark suite, score against its | ||
| own cost per task, for the same models and efforts. Benchmark and PR | ||
| metrics differ, so compare where each cell sits on each chart's | ||
| frontier rather than its values; a cell on the frontier for the | ||
| benchmarks but not for our PRs, or the reverse, stands out. | ||
|
|
||
| Non-goal: a live dashboard. No chart code ships in `quest`. | ||
|
|
||
| ## Plan | ||
|
|
||
| Decided 2026-11-11 (user, /quest-plan in a /quest-run session, when picking | ||
| [moq-dev/quest#642](https://github.com/moq-dev/quest/pull/642) to merge): | ||
|
|
||
| - Goal? ✅ Charts for the model check and the readout, the two places | ||
| routing changes (rejected: a live dashboard; benchmarks only) | ||
| - Which data? ✅ Our measured points from `quest usage --json` with published | ||
| benchmark points beside them (rejected: ours only; benchmarks only) | ||
| - Where do they live? ✅ An artifact per run: the agent running the check or | ||
| the readout renders a private claude.ai page from `quest usage --json` and | ||
| the benchmark table, and links it in its PR, so the core stays | ||
| independent of a harness. The page is for the maintainer, who picks | ||
| routing; the PR body also carries the plotted numbers as tables, so a | ||
| reviewer who cannot open the page can check them (rejected: a `quest usage --html` report, which | ||
| adds chart code and tests to the CLI; token-max, which knows no PR | ||
| quality signals) | ||
| - Milestone? ✅ m0, in this epic (rejected: m1) | ||
|
|
||
| The benchmarks behind the XS writer change to Sonnet 5.5 high | ||
| ([moq-dev/quest#646](https://github.com/moq-dev/quest/pull/646)) and the XS | ||
| arm of the [writer A/B](/quest/m0/routing-evidence/writer-ab.md) | ||
| ([moq-dev/quest#644](https://github.com/moq-dev/quest/pull/644)), both | ||
| decided on 2026-11-11, for the first table: | ||
|
|
||
| - Anthropic's | ||
| [cost guide](https://platform.claude.com/docs/en/about-claude/models/optimizing-for-cost-and-intelligence), | ||
| SWE-bench Pro subset: Opus 5.5 low 87.4% and medium 92.8%, Sonnet 5.5 | ||
| high 87.6% with medium about 7 points lower, at about $0.11 per solved | ||
| task for Opus low and Sonnet high. | ||
| - [Artificial Analysis](https://artificialanalysis.ai/models/comparisons/claude-sonnet-5-5-high-vs-claude-opus-5-5-low), | ||
| Terminal-Bench 4.0 and cost per task: Opus 5.5 low 31.3%, $0.98; Sonnet | ||
| 5.5 high 43.9%, $1.64; Sonnet 5.5 medium 29.8%, $0.78; Haiku 5.5 xhigh | ||
| 29.3%, $0.25. | ||
|
|
||
| Agent's choices, for the implementer to revisit: | ||
|
|
||
| - Keep the benchmark points as a table in the model check's Plan, where its | ||
| existing benchmark notes already live, with each row's source, effort, and | ||
| date; each run refreshes it. A row without a stated effort stays out. | ||
| - Write the chart procedure into the model check's Plan and the epic's | ||
| readout: which `quest usage --json` fields feed each axis, how cells with | ||
|
Comment on lines
+68
to
+69
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Carry the chart procedure into the next model-check run. The next model-check run starts from the carry-forward list in Add the procedure to that list: Suggested fix
|
||
| too few PRs are marked rather than dropped, and that each chart names its | ||
| date range and PR counts. | ||
| - `quest usage --json` reports each reviewer's latest round, and | ||
| [review tallies](/quest/m0/routing-evidence/review-tallies.md) add totals | ||
| across rounds. If the tally line does not also record the first round's | ||
| findings, which the readout compares, add them there. | ||
| - Make the first charts from the data on main when this lands, link them in | ||
| this quest's PR, and say which cells are still too thin. | ||
|
|
||
| ## Required | ||
|
|
||
| - [Review tallies](/quest/m0/routing-evidence/review-tallies.md) - the | ||
| findings each reviewer raised across rounds | ||
| - [usage through token-max](/quest/m0/routing-evidence/usage-token-max.md) - | ||
| the cost axis, which today misses sub-agent transcripts, 72% of host spend | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Scope the writer frontier to each size.
The writer chart places every size on the same axes, so the frontier read from it spans all sizes. PRs of different sizes differ in cost and finding counts. A cross-size frontier can therefore mark an XL arm as dominated by an XS arm without showing that the XL routing choice is worse. Define the frontier within each size before comparing it with the benchmark rankings.
📝 Suggested fix
Committable suggestion