Repository navigation
quest: plan a Cursor CLI reviewer and quota fallthrough for quest review #761
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
moq-bot
merged 3 commits into
moq-dev:main
from
steelhead99x:quest-plan/cursor-reviewer
Nov 17, 2026
Merged
Changes from all commits
Commits
Show all changes
3 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,36 @@ | ||
| # [M] quest review falls through past a reviewer at its usage limit | ||
|
|
||
| ## Goal | ||
|
|
||
| When the reviewer `quest review` picks is at its usage limit, logged out, or not installed, the round runs on the next reviewer in the size's ranking that the latest writer does not avoid, instead of exiting 3 and leaving the PR a draft until the limit resets. Each size ranks a third reviewer, so Opus-written and Sol-written work both have one to fall to. The `## Review` record names the reviewer that ran and the one it skipped, with the reason and reset time. `quest review` stops as an outage only when every eligible ranked reviewer is out. | ||
|
|
||
| ## Plan | ||
|
|
||
| Decided 2026-11-16 (user, /quest-plan), after their Codex hit its usage limit and `quest review` on a PR in kixel-ai/rind could not run, since Claude Opus 5.5 wrote it and Opus is the only other reviewer for its size: | ||
|
|
||
| - When the picked reviewer is out of quota or logged out: | ||
| - ✅ fall through to the next ranked reviewer the writer does not avoid, and record which ran and why (recommended: the classifiers and splitters in `src/cli/models.ts` already run "the first that can run, past any logged out or at its usage limit") | ||
| - keep stopping the review as an outage until it resets (rejected: a single vendor's quota then blocks every PR it would review, which is how opencode's quota failed every `quest review` on 2026-11-13, per [token waste](/quest/m0/token-waste/README.md)) | ||
|
|
||
| Decided 2026-11-17 (maintainer, [moq-dev/quest#761](https://github.com/moq-dev/quest/pull/761#issuecomment-6942626871)): | ||
|
|
||
| - ✅ stays in m0 and adds a third ranked reviewer per size from a model already supported, such as Claude Sonnet 5.5, scored with `quest eval run` as `AGENTS.md` requires (rejected: m1 behind the Cursor reviewer; keep as planned; pull the Cursor reviewer into m0). Today `REVIEWERS` in `src/cli/models.ts` lists only Codex Sol and Claude Opus per size, so for Opus-written work the fallthrough had nothing to fall to: that is the rind stall above. | ||
|
|
||
| Guidance: | ||
|
|
||
| - Third reviewer: append Claude Sonnet 5.5 to each size's `REVIEWERS`, last, at the effort the eval supports (start from one step below the size's Opus entry, never below low). Run `quest eval run` with the base's build and the head's, and put both scores, the per-size catch rates, and their cost in the PR's `## Validation`. If Sonnet catches too little at some size to be worth a review, say so there and recommend another supported model instead of shipping a reviewer that rubber-stamps. | ||
| - Fall through only on a failure the provider or launcher reports as a usage limit or a missing login (`ReviewLimitError` and the `unauthorized` patterns in `src/cli/review.ts`, the login checks in `LOGINS` in `src/cli/agent.ts`), or a reviewer CLI that is not installed, all detected before any finding is saved. Text in the diff or in a reviewer's response, such as "usage limit", never changes the reviewer: match only the launcher's exit status, the CLI's structured error, its login check, or, on a failed turn, the CLI's stderr against the `unauthorized` patterns and, for a Codex gateway, `RETRIED_429`. | ||
| - Separate reviewer unavailability from input errors: today `prepare()` and `turn()` throw `ReviewInputError` for auth failures and a missing CLI (`requireProgram`) alongside a dirty tree, a bad `--branch`, or a malformed `RISK.md`. Give unavailability its own error type, so only it falls through and input errors still exit 2. | ||
| - An explicit `--harness` or `--model` names the reviewer: it never falls through, and fails as an outage, so a controlled run such as `quest eval run` cannot silently measure a different model. | ||
| - Detect the outage before the round starts where the login check allows, and retry the round on the next eligible entry; keep the exit-3 path for the last one. | ||
| - A reviewer that fell through settles the open findings of the one it replaces, as a writer change already does in `quest review`. | ||
| - Record the skipped reviewer outside the `- Reviewer:` blocks of `## Review`, such as a `- Skipped: <harness>, <model>, <effort> effort: <reason>, resets <time>` line in the header block, since `parseReview()` in `src/cli/review-record.ts` splits on `- Reviewer:` and throws on a block without a session. Update `parseReview()` and `quest usage` together if the format needs them to read it. | ||
| - Decide in the PR whether the shadow reviewer follows the fallback (its model one effort lower) or is skipped for that round, and record why. | ||
| - This quest authorizes the matching edit to `assets/skills/review.md`'s outage paragraph ("A reviewer out of quota or logged out is an outage, not a skip"), which `AGENTS.md` otherwise protects: it becomes an outage only when every eligible reviewer is out. | ||
| - Update `quest review --help` and the README's reviewer list for the third entry and the fallthrough. | ||
| - The fallback order is the size's `REVIEWERS` list; [the Cursor reviewer](/quest/m1/cursor-reviewer.md) may add entries to it, so the two compose without either waiting on the other. | ||
|
|
||
| ## Related | ||
|
|
||
| - [Cursor reviewer](/quest/m1/cursor-reviewer.md) - adds a harness whose models may join the ranking | ||
| - [Model check](/quest/m1/model-check.md) - audits the ranking this falls through |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,43 @@ | ||
| # [L] quest review runs reviewers through the Cursor CLI | ||
|
|
||
| ## Goal | ||
|
|
||
| `quest review --harness cursor --model <id> [--effort <level>]` reviews a branch through Cursor's CLI (`agent -p`), read-only, on the user's own Cursor login, so a contributor with a Cursor plan can review with Gemini 4 Carbon or any other model Cursor serves. The PR scores Gemini 4 Carbon through Cursor with `quest eval run` against the current reviewers, and moves it ahead of Codex Sol in each PR size's `REVIEWERS` ranking where it catches more culprits per dollar. | ||
|
|
||
| Non-goals: Cursor as a writer, iterate, or land-fixer harness; a Grok or other non-Gemini entry in the ranking, which [the model check](/quest/m1/model-check.md) can add from its own evidence once the harness exists. | ||
|
|
||
| ## Plan | ||
|
|
||
| Decided 2026-11-16 (user, /quest-plan). The user reports Gemini 4 Carbon, run through Cursor, catching bugs that Claude Opus 5.5 at medium effort missed in their own work, at a lower price. | ||
|
|
||
| - What should the quest promise? | ||
| - ✅ an opt-in reviewer harness, scored with `quest eval run` before any ranking change (recommended) | ||
| - Gemini in the default ranking right away (rejected: no eval, which `AGENTS.md` requires for a change to `src/cli/models.ts`) | ||
| - a generic any-model harness again (rejected: [moq-dev/quest#701](https://github.com/moq-dev/quest/pull/701) removed opencode's on 2026-11-13; Cursor reaches the same models through one subscription the user already has) | ||
| - Which harness? | ||
| - ✅ the Cursor CLI (recommended): headless print mode, `--model`, `--resume <chatId>`, `--output-format stream-json`, and a read-only `--mode ask`; one login reaches Gemini, Grok, GPT, and Claude models | ||
| - Google's Gemini CLI (rejected: a second login or subscription, and Gemini only) | ||
| - If Gemini 4 Carbon scores better: | ||
| - ✅ the eval decides and this PR moves it ahead of Codex Sol where it wins (recommended: the question is which model should review Opus-written work, and the eval is the evidence the ranking change needs; the maintainer still judges it in review) | ||
| - post the scores and leave the ranking to a `Maintainer:` decision (rejected) | ||
| - ✅ m1 (rejected: m0, m2): the public release promises contributors can run Quest on their forks, and one whose only plan is Cursor cannot run `quest review` today. | ||
|
|
||
| Guidance: | ||
|
|
||
| - Keep Cursor a reviewer-only harness, per the Non-goal: give `quest review` its own reviewer harness list (a `ReviewHarness` union beside `HARNESSES`, as `review.ts` and `review-record.ts` had before [moq-dev/quest#701](https://github.com/moq-dev/quest/pull/701)) rather than adding `cursor` to `HARNESSES` in `src/cli/agent.ts`, which would also reach `quest agent start --harness` and `transcriptFacts()`. | ||
| - Use the CLI's stored login (`agent login`), not `CURSOR_API_KEY`, as [moq-dev/quest#492](https://github.com/moq-dev/quest/pull/492) settled for the other harnesses. `agent status` exits 0 when logged out (`Not logged in`) and with expired credentials (`Logged in (unable to fetch user details)`), per Cursor CLI `2026.11.13-eacade1`, so the login check reads its output, not only its exit code, and a turn's `Authentication failed` maps to logged out. | ||
| - Strip Cursor's credential and model overrides from the reviewer's environment (`CURSOR_AGENT`, `CURSOR_API_KEY`, `CURSOR_AUTH_TOKEN*`, `CURSOR_MODEL`), and make `runningHarness()` recognize `CURSOR_AGENT`, so `quest review` from a Cursor session avoids the right writer. | ||
| - The reviewers run nothing from a reviewed head. Cursor's read-only `--mode ask` is necessary but not the isolation boundary: the reviewer must also load none of the user's or the checkout's rules, skills, MCP servers, hooks, or `.cursor/` config, the way `quest review` runs Codex with `--ignore-user-config`. The public `agent --help` has no flag for that; `2026.11.13-eacade1` has hidden `--disable-project-configs` (`.cursor/cli.json` only), `--exclude-workspace-context`, and `--disable-indexing`, while `.cursor/mcp.json` and `~/.cursor/mcp.json` still load unless `CURSOR_CONFIG_DIR` and `CURSOR_DATA_DIR` point at a clean directory and `--sandbox enabled` is set. Verify it with a test like `test/live/review-sandbox.test.ts`'s: an untrusted checkout whose rules, MCP config, and hooks try to read outside the review snapshot or run a process, on Linux and macOS (keychain login via `AGENT_CLI_CREDENTIAL_STORE`). If the pinned CLI cannot reliably disable both inherited user config and checkout config, record that under `## Waiting` instead of shipping the harness. | ||
| - `agent` has no `--effort` flag (effort rides on the model, as `model[effort=high]`, only for models that take it) and no output-schema flag. Map the effort ladder onto that syntax where the model takes it; where it does not, run the model at its one effort and skip the shadow, as `quest review` does for an effort off the ladder. With no `REVIEWERS` entry to supply one, `--effort` stays required for a Cursor model that takes an effort, as `pickReviewer()` requires it today, and is not needed for one that takes none. Parse the verdict from the response text as `verdict()` already does. | ||
| - `quest review` classifies the head's risk with the first `CLASSIFIERS` entry that can run before any reviewer starts, and today that list holds only Claude Haiku and Codex Sol, so a Cursor-only contributor would stop there. Append a cheap Cursor-served classifier last (its id from the CLI's model list), so classification reaches Cursor only when neither of the others can run. | ||
| - Detect Cursor's usage-limit and logged-out errors from its structured output so [the quota fallthrough](/quest/m0/review-quota-fallthrough.md) treats Cursor like the other harnesses. | ||
| - Look up Gemini 4 Carbon's Cursor model id with the CLI's model list rather than guessing it; record the CLI version checked. | ||
| - `quest eval run` prices sessions through [kixel-ai/token-max](https://github.com/kixel-ai/token-max), which reads only Claude Code and Codex transcripts, so a Cursor run would report a lower-bound `$0.00`. Recommended: add Cursor transcripts to token-max in its own PR first, since `quest eval` already delegates all pricing there; else price Cursor seats in `quest eval` from the usage its stream reports. The culprits-per-dollar comparison needs one of them before the ranking change. | ||
| - Run `quest eval run` with the base's build and the head's, per `AGENTS.md`, and put both scores, the per-size catch rates, and their cost in the PR's `## Validation`. Compare Carbon with Sol for each size, and with Opus and the third reviewer [the quota fallthrough](/quest/m0/review-quota-fallthrough.md) adds where they review. | ||
| - Keep the harness opt-in (`--harness cursor`) until the eval moves a Cursor model into `REVIEWERS`; contributors without Cursor installed then fall through past it, per the quota fallthrough. | ||
| - Update `README.md`'s reviewer list and `quest review --help` to name the new harness. | ||
|
|
||
| ## Related | ||
|
|
||
| - [Quota fallthrough](/quest/m0/review-quota-fallthrough.md) - falls through the ranking this adds to | ||
| - [Model check](/quest/m1/model-check.md) - audits each milestone's reviewer ranking | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.