From 2055608cc9de4a13fd90fab342a11a98ec7ec697 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 17 Nov 2026 02:58:15 +0000 Subject: [PATCH 1/3] quest: plan a Cursor CLI reviewer and quota fallthrough for quest review Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01DFbhGvB8xkG1Ev8Rtazetu --- quest/m0/review-quota-fallthrough.md | 27 ++++++++++++++++++++ quest/m1/cursor-reviewer.md | 37 ++++++++++++++++++++++++++++ 2 files changed, 64 insertions(+) create mode 100644 quest/m0/review-quota-fallthrough.md create mode 100644 quest/m1/cursor-reviewer.md diff --git a/quest/m0/review-quota-fallthrough.md b/quest/m0/review-quota-fallthrough.md new file mode 100644 index 00000000..3e80c3f9 --- /dev/null +++ b/quest/m0/review-quota-fallthrough.md @@ -0,0 +1,27 @@ +# [S] quest review falls through past a reviewer at its usage limit + +## Goal + +When the reviewer `quest review` picks is at its usage limit or logged out, the round runs on the next reviewer in the size's ranking that the latest writer does not avoid, instead of exiting 3 and leaving the PR a draft until the limit resets. The `## Review` record names the reviewer that ran and the one it skipped, with the reason and reset time. `quest review` stops as an outage only when every eligible ranked reviewer is out. + +## Plan + +Decided 2026-11-16 (user, /quest-plan), after their Codex hit its usage limit and `quest review` on a PR in kixel-ai/rind could not run, since Claude Opus 5.5 wrote it and Opus is the only other reviewer for its size: + +- When the picked reviewer is out of quota or logged out: + - ✅ fall through to the next ranked reviewer the writer does not avoid, and record which ran and why (recommended: the classifiers and splitters in `src/cli/models.ts` already run "the first that can run, past any logged out or at its usage limit") + - keep stopping the review as an outage until it resets (rejected: a single vendor's quota then blocks every PR it would review, which is how opencode's quota failed every `quest review` on 2026-11-13, per [token waste](/quest/m0/token-waste/README.md)) +- ✅ m0 (rejected: m1, m2): outages already block moq-dev's and Kixel's PRs today, on the maintainer's machines and contributors' forks alike. + +Guidance: + +- `UsageLimitError` (`src/cli/review.ts`) and the login checks (`LOGINS`, `src/cli/agent.ts`) already detect both cases. Detect them before any finding is saved, retry the round on the next eligible entry, and keep the exit-3 path for the last one. +- A reviewer that fell through settles the open findings of the one it replaces, as a writer change already does in `quest review`. +- Decide in the PR whether the shadow reviewer follows the fallback (its model one effort lower) or is skipped for that round, and record why. +- This quest authorizes the matching edit to `assets/skills/review.md`'s outage paragraph ("A reviewer out of quota or logged out is an outage, not a skip"), which `AGENTS.md` otherwise protects: it becomes an outage only when every eligible reviewer is out. +- The fallback order is the size's existing `REVIEWERS` list; [the Cursor reviewer](/quest/m1/cursor-reviewer.md) adds entries to it, so the two compose without either waiting on the other. + +## Related + +- [Cursor reviewer](/quest/m1/cursor-reviewer.md) - adds a third harness, so Opus-written work has a fallback beyond Codex +- [Model check](/quest/m1/model-check.md) - audits the ranking this falls through diff --git a/quest/m1/cursor-reviewer.md b/quest/m1/cursor-reviewer.md new file mode 100644 index 00000000..a6c5a993 --- /dev/null +++ b/quest/m1/cursor-reviewer.md @@ -0,0 +1,37 @@ +# [L] quest review runs reviewers through the Cursor CLI + +## Goal + +`quest review --harness cursor --model ` reviews a branch through Cursor's CLI (`agent -p`), read-only, on the user's own Cursor login, so a contributor with a Cursor plan can review with Gemini 4 Carbon or any other model Cursor serves. The PR scores Gemini 4 Carbon through Cursor with `quest eval run` against the current reviewers, and moves it ahead of Codex Sol in each PR size's `REVIEWERS` ranking where it catches more culprits per dollar. + +Non-goals: Cursor as a writer, iterate, or land-fixer harness; a Grok or other non-Gemini entry in the ranking, which [the model check](/quest/m1/model-check.md) can add from its own evidence once the harness exists. + +## Plan + +Decided 2026-11-16 (user, /quest-plan). The user reports Gemini 4 Carbon, run through Cursor, catching bugs that Claude Opus 5.5 at medium effort missed in their own work, at a lower price. + +- What should the quest promise? + - ✅ an opt-in reviewer harness, scored with `quest eval run` before any ranking change (recommended) + - Gemini in the default ranking right away (rejected: no eval, which `AGENTS.md` requires for a change to `src/cli/models.ts`) + - a generic any-model harness again (rejected: [moq-dev/quest#701](https://github.com/moq-dev/quest/pull/701) removed opencode's on 2026-11-13; Cursor reaches the same models through one subscription the user already has) +- Which harness? + - ✅ the Cursor CLI (recommended): headless print mode, `--model`, `--resume `, `--output-format stream-json`, and a read-only `--mode ask`; one login reaches Gemini, Grok, GPT, and Claude models + - Google's Gemini CLI (rejected: a second login or subscription, and Gemini only) +- If Gemini 4 Carbon scores better: + - ✅ the eval decides and this PR moves it ahead of Codex Sol where it wins (recommended: the question is which model should review Opus-written work, and the eval is the evidence the ranking change needs; the maintainer still judges it in review) + - post the scores and leave the ranking to a `Maintainer:` decision (rejected) +- ✅ m1 (rejected: m0, m2): the public release promises contributors can run Quest on their forks, and one whose only plan is Cursor cannot run `quest review` today. + +Guidance: + +- Add `cursor` to `HARNESSES` and `HARNESS` in `src/cli/agent.ts`, and its login check (`agent status`; the login command is `agent login`) to `LOGINS`, using the CLI's stored login, not `CURSOR_API_KEY`, as [moq-dev/quest#492](https://github.com/moq-dev/quest/pull/492) settled for the other harnesses. +- The reviewers run nothing from a reviewed head. Keep that for Cursor: read-only mode, no MCP servers or hooks, and none of the checkout's own `.cursor/` rules, settings, or MCP config, the way `quest review` runs Codex with `--ignore-user-config`. Verify it with a test like `test/live/review-sandbox.test.ts`'s and, if the CLI offers no supported way to skip project config, record that under `## Waiting` instead of shipping the harness. +- Detect Cursor's usage-limit and logged-out errors so [the quota fallthrough](/quest/m0/review-quota-fallthrough.md) treats Cursor like the other harnesses. +- Look up Gemini 4 Carbon's Cursor model id with the CLI's model list rather than guessing it; record the CLI version checked. +- Run `quest eval run` with the base's build and the head's, per `AGENTS.md`, and put both scores, the per-size catch rates, and their cost in the PR's `## Validation`. Compare Carbon with Sol for each size, and with Opus where Opus reviews. +- Update `README.md`'s reviewer list and `quest review --help` to name the new harness. + +## Related + +- [Quota fallthrough](/quest/m0/review-quota-fallthrough.md) - falls through the ranking this adds to +- [Model check](/quest/m1/model-check.md) - audits each milestone's reviewer ranking From 9c372e49c743339e2bebd090bc8fe89deaf23d2c Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 17 Nov 2026 00:29:33 -0800 Subject: [PATCH 2/3] quest: fold the maintainer decision and both reviews into the reviewer plans Co-Authored-By: Claude Opus 5.5 --- quest/m0/review-quota-fallthrough.md | 21 +++++++++++++++------ quest/m1/cursor-reviewer.md | 13 +++++++++---- 2 files changed, 24 insertions(+), 10 deletions(-) diff --git a/quest/m0/review-quota-fallthrough.md b/quest/m0/review-quota-fallthrough.md index 3e80c3f9..a6d9eb5d 100644 --- a/quest/m0/review-quota-fallthrough.md +++ b/quest/m0/review-quota-fallthrough.md @@ -1,8 +1,8 @@ -# [S] quest review falls through past a reviewer at its usage limit +# [M] quest review falls through past a reviewer at its usage limit ## Goal -When the reviewer `quest review` picks is at its usage limit or logged out, the round runs on the next reviewer in the size's ranking that the latest writer does not avoid, instead of exiting 3 and leaving the PR a draft until the limit resets. The `## Review` record names the reviewer that ran and the one it skipped, with the reason and reset time. `quest review` stops as an outage only when every eligible ranked reviewer is out. +When the reviewer `quest review` picks is at its usage limit, logged out, or not installed, the round runs on the next reviewer in the size's ranking that the latest writer does not avoid, instead of exiting 3 and leaving the PR a draft until the limit resets. Each size ranks a third reviewer, so Opus-written and Sol-written work both have one to fall to. The `## Review` record names the reviewer that ran and the one it skipped, with the reason and reset time. `quest review` stops as an outage only when every eligible ranked reviewer is out. ## Plan @@ -11,17 +11,26 @@ Decided 2026-11-16 (user, /quest-plan), after their Codex hit its usage limit an - When the picked reviewer is out of quota or logged out: - ✅ fall through to the next ranked reviewer the writer does not avoid, and record which ran and why (recommended: the classifiers and splitters in `src/cli/models.ts` already run "the first that can run, past any logged out or at its usage limit") - keep stopping the review as an outage until it resets (rejected: a single vendor's quota then blocks every PR it would review, which is how opencode's quota failed every `quest review` on 2026-11-13, per [token waste](/quest/m0/token-waste/README.md)) -- ✅ m0 (rejected: m1, m2): outages already block moq-dev's and Kixel's PRs today, on the maintainer's machines and contributors' forks alike. + +Decided 2026-11-17 (maintainer, [moq-dev/quest#761](https://github.com/moq-dev/quest/pull/761#issuecomment-6942626871)): + +- ✅ stays in m0 and adds a third ranked reviewer per size from a model already supported, such as Claude Sonnet 5.5, scored with `quest eval run` as `AGENTS.md` requires (rejected: m1 behind the Cursor reviewer; keep as planned; pull the Cursor reviewer into m0). Today `REVIEWERS` in `src/cli/models.ts` lists only Codex Sol and Claude Opus per size, so for Opus-written work the fallthrough had nothing to fall to: that is the rind stall above. Guidance: -- `UsageLimitError` (`src/cli/review.ts`) and the login checks (`LOGINS`, `src/cli/agent.ts`) already detect both cases. Detect them before any finding is saved, retry the round on the next eligible entry, and keep the exit-3 path for the last one. +- Third reviewer: append Claude Sonnet 5.5 to each size's `REVIEWERS`, last, at the effort the eval supports (start from one step below the size's Opus entry, never below low). Run `quest eval run` with the base's build and the head's, and put both scores, the per-size catch rates, and their cost in the PR's `## Validation`. If Sonnet catches too little at some size to be worth a review, say so there and recommend another supported model instead of shipping a reviewer that rubber-stamps. +- Fall through only on a failure the provider or launcher reports as a usage limit or a missing login (`ReviewLimitError` and the `unauthorized` patterns in `src/cli/review.ts`, the login checks in `LOGINS` in `src/cli/agent.ts`), or a reviewer CLI that is not installed, all detected before any finding is saved. Text in the diff or in a reviewer's response, such as "usage limit", never changes the reviewer: match only the launcher's exit status, the CLI's structured error, or its login check. +- Separate reviewer unavailability from input errors: today `prepare()` and `turn()` throw `ReviewInputError` for auth failures and a missing CLI (`requireProgram`) alongside a dirty tree, a bad `--branch`, or a malformed `RISK.md`. Give unavailability its own error type, so only it falls through and input errors still exit 2. +- An explicit `--harness` or `--model` names the reviewer: it never falls through, and fails as an outage, so a controlled run such as `quest eval run` cannot silently measure a different model. +- Detect the outage before the round starts where the login check allows, and retry the round on the next eligible entry; keep the exit-3 path for the last one. - A reviewer that fell through settles the open findings of the one it replaces, as a writer change already does in `quest review`. +- Record the skipped reviewer outside the `- Reviewer:` blocks of `## Review`, such as a `- Skipped: , , effort: , resets