From 51592fca2bcbd423aa0000244215c9f2662b97ff Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Mon, 9 Nov 2026 09:15:18 -0800 Subject: [PATCH 1/3] chore(quest): claim quest/m0/model-check.md Quest-Claim: 18ff5461de8e9b21 From f25685fcb0f374031880cf0d60dd95e53d03398b Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Mon, 9 Nov 2026 09:16:20 -0800 Subject: [PATCH 2/3] quest: finish m0's model check, carrying its routing tallies to m1 No table changed: 4-7 routed PRs per size are too few to retune. Co-Authored-By: Claude Opus 5.5 --- quest/m0/model-check.md | 31 ------------------------------- quest/m0/token-waste/README.md | 4 ---- quest/m1/model-check.md | 23 +++++++++++++++++++++-- 3 files changed, 21 insertions(+), 37 deletions(-) delete mode 100644 quest/m0/model-check.md diff --git a/quest/m0/model-check.md b/quest/m0/model-check.md deleted file mode 100644 index 51cf695e6..000000000 --- a/quest/m0/model-check.md +++ /dev/null @@ -1,31 +0,0 @@ -# [S] Model check - -## Goal - -Model routing's first audit checks how the routed PRs went and retunes -`quest`'s built-in size tables, as each milestone's [model check](/quest/m1/model-check.md) does -after it. - -## Plan - -Recurring, decided 2026-11-04 (user, /quest-plan follow-up of -[moq-dev/quest#487](https://github.com/moq-dev/quest/pull/487)): the first -run is in m0 once routing lands, then each milestone. - -- This run audits the routing itself, here in m0, following the steps - [m1's run](/quest/m1/model-check.md) lists, over the PRs merged since the - routing epic's last quest landed. It skips m1's model id comparison and - sandbox test, which m0 ran in - [moq-dev/quest#426](https://github.com/moq-dev/quest/pull/426). -- Review efforts changed 2026-11-05 (user, the - [token waste](/quest/m0/token-waste/README.md) audit): Sol reviews M at - medium and L at high, replacing high and xhigh from - [moq-dev/quest#493](https://github.com/moq-dev/quest/pull/493). xhigh cost - 2-6× more per review, and only 4 large xhigh reviews existed, so nothing - showed it finds more. Reviews are single-pass since the same change, so - compare findings per round by effort across the routed PRs, with the - misses later rounds, CI, or follow-ups found, and recommend restoring a - higher effort where a lower one misses findings. -- m1's run already exists, listed in [m1](/quest/m1/README.md)'s `Required` - and requiring m0. The PR that finishes this run creates no new run: it - carries the tallies and any table changes into m1's Plan as its baseline. diff --git a/quest/m0/token-waste/README.md b/quest/m0/token-waste/README.md index ea5fa14ba..497a9d59d 100644 --- a/quest/m0/token-waste/README.md +++ b/quest/m0/token-waste/README.md @@ -65,7 +65,3 @@ Decision prompts, chosen options marked ✅: - The milestone re-cut the maintainer raised (m1 additive improvements, m2 semver-breaking features) is moq-dev/moq's own plan: ✅ re-plan moq separately (rejected: make it quest's convention; not now). - -## Related - -- [Model check](/quest/m0/model-check.md) - audits the size tables the review quest changes diff --git a/quest/m1/model-check.md b/quest/m1/model-check.md index d13fcd419..2e87247bb 100644 --- a/quest/m1/model-check.md +++ b/quest/m1/model-check.md @@ -22,8 +22,8 @@ the size tables of model routing (`src/cli/models.ts`): - A separate audit owning the tables - A separate audit of efforts only - When does it run? ✅ First in m0 once routing lands - ([m0's run](/quest/m0/model-check.md)), then each milestone (rejected: - each milestone only) + ([moq-dev/quest#584](https://github.com/moq-dev/quest/pull/584)), then + each milestone (rejected: each milestone only) Audit the routing from the records of the PRs merged since the last run, with `quest usage --since `, whose accuracy rollup reads each body's @@ -49,6 +49,25 @@ checkpoints and outages to count by hand: it unasked. Replacing a stale model id with the same vendor's newer one, below, needs no decision, as this check always did. +- Baseline from m0's routing audit + ([moq-dev/quest#584](https://github.com/moq-dev/quest/pull/584)), + `quest usage --since 2026-11-07`, after routing landed in + [moq-dev/quest#558](https://github.com/moq-dev/quest/pull/558); writers + on their table entry: + - XS Haiku high: 4 PRs, 0 handed back, score 9.7, 1.0 rounds, 0 fixed. + - S Opus low: 4 PRs, 0 handed back, score 10.0, 1.3 rounds, 1 fixed. + - M Opus medium: 5 PRs, 1 handed back, score 9.2, 2.2 rounds, 5 fixed, + 3 open; the hand-back, + [moq-dev/quest#553](https://github.com/moq-dev/quest/pull/553), was + iterated at low. + - L Opus high: 7 PRs, 0 handed back, score 10.0, 2.6 rounds, 10 fixed. + - Sol reviews, latest round's fixes: low 4 PRs, 0 fixed; medium 25 PRs, + 4 with fixes, 4 fixed; high 9 PRs, 6 with fixes, 13 fixed; xhigh 2 PRs, + 1 fixed. No PR names a routed PR under `## Caused by`, so no review + miss showed, and no outage skipped a quest. + - Too thin (4-7 PRs per cell) to move a size; no table changed. Watch M + at medium, iterates below the size's effort, and `Implementer:` lines + without an effort (about 9 said `unknown`). - Compare both tables' models (in `src/cli/models.ts`) with each vendor's current models: the newest general model. Bump any that are stale, and check that both CLIs accept the new ids and the pinned efforts. `--deep` and its strongest-model From fecf566d98ea022afd2de711ebcc15f51a788d8f Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Mon, 9 Nov 2026 10:00:33 -0800 Subject: [PATCH 3/3] quest: baseline m1's model check on merged PRs only Co-Authored-By: Claude Opus 5.5 --- quest/m1/model-check.md | 39 +++++++++++++++++++++++---------------- 1 file changed, 23 insertions(+), 16 deletions(-) diff --git a/quest/m1/model-check.md b/quest/m1/model-check.md index 2e87247bb..50d04740f 100644 --- a/quest/m1/model-check.md +++ b/quest/m1/model-check.md @@ -50,24 +50,31 @@ checkpoints and outages to count by hand: below, needs no decision, as this check always did. - Baseline from m0's routing audit - ([moq-dev/quest#584](https://github.com/moq-dev/quest/pull/584)), - `quest usage --since 2026-11-07`, after routing landed in - [moq-dev/quest#558](https://github.com/moq-dev/quest/pull/558); writers - on their table entry: - - XS Haiku high: 4 PRs, 0 handed back, score 9.7, 1.0 rounds, 0 fixed. + ([moq-dev/quest#584](https://github.com/moq-dev/quest/pull/584)): the 25 + PRs merged from 2026-11-07T22:30:45Z, when routing landed in + [moq-dev/quest#558](https://github.com/moq-dev/quest/pull/558), to + 2026-11-09T17:00Z, by GitHub's `merged_at`, with their records from + `quest usage --since 2026-11-06 --json`. `quest usage` selects by update + time with open PRs included, so filter its PRs by merge time. Writers on + their table entry: + - XS Haiku high: none; both merged XS PRs ran another model. - S Opus low: 4 PRs, 0 handed back, score 10.0, 1.3 rounds, 1 fixed. - - M Opus medium: 5 PRs, 1 handed back, score 9.2, 2.2 rounds, 5 fixed, - 3 open; the hand-back, - [moq-dev/quest#553](https://github.com/moq-dev/quest/pull/553), was - iterated at low. - - L Opus high: 7 PRs, 0 handed back, score 10.0, 2.6 rounds, 10 fixed. - - Sol reviews, latest round's fixes: low 4 PRs, 0 fixed; medium 25 PRs, - 4 with fixes, 4 fixed; high 9 PRs, 6 with fixes, 13 fixed; xhigh 2 PRs, - 1 fixed. No PR names a routed PR under `## Caused by`, so no review + - M Opus medium: 2 PRs, 0 handed back, score 9.5, 2.0 rounds, 0 fixed. + - L Opus high: 3 PRs, 0 handed back, score 10.0, 3.0 rounds, 4 fixed. + - Sol reviews, latest round's fixes: low 1 PR, 0 fixed; medium 17 PRs, + 4 with fixes, 4 fixed; high 3 PRs, 2 with fixes, 4 fixed; xhigh 1 PR, + 0 fixed. No PR names a routed PR under `## Caused by`, so no review miss showed, and no outage skipped a quest. - - Too thin (4-7 PRs per cell) to move a size; no table changed. Watch M - at medium, iterates below the size's effort, and `Implementer:` lines - without an effort (about 9 said `unknown`). + - Unfinished then, not counted: open + [moq-dev/quest#583](https://github.com/moq-dev/quest/pull/583), + [moq-dev/quest#586](https://github.com/moq-dev/quest/pull/586), and + [moq-dev/quest#588](https://github.com/moq-dev/quest/pull/588), and + [moq-dev/quest#553](https://github.com/moq-dev/quest/pull/553) (M, + merged 2026-11-09T17:33Z after a hand-back). Count those that merged + in m1's run. + - Too thin (0-4 PRs per cell) to move a size; no table changed. Watch + iterates below the size's effort, and `Implementer:` lines without an + effort (6 of the 20 lines said `unknown`). - Compare both tables' models (in `src/cli/models.ts`) with each vendor's current models: the newest general model. Bump any that are stale, and check that both CLIs accept the new ids and the pinned efforts. `--deep` and its strongest-model