Skip to content

fix(cursor-review): collapse catalog-drift review-me list to one row per family (BE-6911) - #147

Open
mattmillerai wants to merge 1 commit into
mainfrom
matt/be-6911-catalog-drift-tier-collapse
Open

fix(cursor-review): collapse catalog-drift review-me list to one row per family (BE-6911)#147
mattmillerai wants to merge 1 commit into
mainfrom
matt/be-6911-catalog-drift-tier-collapse

Conversation

@mattmillerai

Copy link
Copy Markdown
Contributor

ELI-5

Every Monday a robot compares the four AI models our PR-review panel uses against Cursor's live model list, and files a report saying "here's everything you're not using — worth a look?". Cursor sells each model at half a dozen effort levels (-low, -high, -max, each optionally -fast), and we pin exactly one level per model — so the report was listing every other effort level of every model we already use, forever. Last Monday that came to 178 lines, cut off at 25 per vendor in whatever order Cursor happened to print them. The two brand-new models that actually shipped that week were on lines 26+, invisible. This groups the effort levels back into one line per model and puts the newest models first, so the report is 37 lines and the new stuff is at the top.

What changed

.github/cursor-review/catalog-drift.py, rendering only:

  • Tier collapse. New module-level TIER_SUFFIXES (none/minimal/low/medium/high/extra-high/xhigh/max, each optionally -fast, plus a bare -fast) and family_of() split an id into (family, tier). The same-lab review-me list now renders one row per (lab, family): the family, how many tiers the catalog offers and which, and one member's catalog note reproduced verbatim.
  • Signal ordering. collapse_tiers() sorts rows by _family_version() descending (every digit run in the family id, in order — labs write versions as 5.6, 4-8 and k3, so anything smarter would need a per-lab table), ties broken by catalog order. MAX_FAMILY_IDS now drops the oldest families instead of an arbitrary tail of Cursor's print order.
  • Bounds. New MAX_TIERS_PER_ROW = 8 keeps a collapsed row at roughly the char cost of the single id row it replaces, so the section's worst-case budget is unchanged and the raw catalog fold keeps its room. A truncated tier list still states the full count and drops the weakest tiers.
  • The stale MAX_FAMILY_IDS comment (the one claiming "a few dozen models" and "every list leads with its highest-signal rows" — neither of which was true of this list) is rewritten to describe the shipped behaviour.
  • README.md's catalog row gains a clause on the per-family rendering.

Untouched on purpose: analyze(), the JSON report, summary_line(), urgent / has_findings, the NO-ZDR handling on pins, and the unpinned-families fold (still one row per id in catalog order — it's short and already collapsed).

Result against the 2026-08-10 catalog (#144)

178 unpinned same-lab ids → 37 rows. The gpt group goes from 25 shown + "… and 57 more" to 11 rows with nothing hidden, and reads:

- `gpt-5.6-sol-*`   — 11 tiers …  · _panel already pins `gpt-5.6-sol-max`_
- `gpt-5.6-terra-*` — 12 tiers (`max`, `max-fast`, `xhigh`, …); top `gpt-5.6-terra-max` — `- GPT-5.6 Terra 1M Max`
- `gpt-5.6-luna-*`  — 12 tiers (`max`, `max-fast`, `xhigh`, …); top `gpt-5.6-luna-max`  — `- GPT-5.6 Luna 1M Max`
- `gpt-5.5-*` … `gpt-5.4-*` … `gpt-5.3-codex-*` … `gpt-5.2-*` … `gpt-5.1-*` … `gpt-5-mini`

claude 87 → 21 rows, gemini 6 → 3, kimi 3 → 2. No group emits a "… and N more" line. Rendered body drops from ~44 KB to ~16 KB.

Tests

python3 -m unittest discover -s .github/cursor-review/tests -p 'test_*.py'175 pass (159 before + 16 new). The other three Python suites (agents-md-integrity, groom, refresh-reviewers) are green too.

New TierCollapseTest covers: the family/tier split (longest-tier-wins for extra-high, bare -fast, no-suffix ids joining their family), version ordering including the catalog-order tie-break and a version-less family, tier collapse in the rendered body, verbatim notes on a collapsed row, the capped-rows message, a synthetic brand-new family printed below the pre-change cut, the worst-case char budget, and the real #144 catalog.

The #144 catalog is committed verbatim as .github/cursor-review/tests/fixtures/catalog-2026-08-10.txt so the acceptance case is asserted against the real thing rather than a hand-built approximation. It's inside the suite's existing path filter.

Judgment calls

  • -thinking is deliberately NOT a collapsible suffix. It reads like a reasoning knob, but the panel pins on it (claude-opus-5-thinking-max vs claude-opus-5-max are different products for this purpose), so collapsing them would hide the distinction a promotion decision turns on. Consequence: claude-4.6-opus-high-thinking (suffixes in the reverse order) stays a row of its own. Cosmetic.
  • minimal and extra-high were added to the ticket's named suffix set — both appear in the live catalog (gemini-3.6-flash-minimal, gpt-5.5-extra-high) and are plainly the same kind of tier.
  • A collapsed row shows one member's note, not all of them. Notes are per-tier ("GPT-5.6 Terra 1M Max"), so a family row reproduces the strongest tier's verbatim and the section intro points at the raw fold for the rest. The trade: a NO-ZDR marker present on only a non-top tier of a family is no longer in the report body (it is still in the raw fold). Pre-change it was equally invisible for anything past row 25, and I did not add NO-ZDR detection to candidate rows because the acceptance criteria require notes stay verbatim and uninterpreted there — the NO-ZDR check on pins (the urgent path) is untouched.
  • Beyond scope, flagged not fixed: a delisted pin's "available for lab X" line still lists raw ids capped at MAX_FAMILY_IDS (catalog-drift.py line ~713). Same underlying noise, but it's the urgent path that gets read carefully and the ticket scopes only the review-me list. Happy to follow up.
  • Negative-claim falsification (self-review step e): trigger did not fire. The diff adds no "not supported"/"unavailable"/"STOP" string, no throw/deny dead-end, and flips no test to assert a dead-end. It renders strictly more families than before and points at the raw catalog fold for per-id detail; nothing is denied.

Not done (out of scope per the ticket)

No pin is bumped — all four were re-derived against this catalog and none warrants a change; that stays a human call. urgent / has_findings / sticky-issue lifecycle, the family→lab alias table (still derived from the pins), cursor-review.yml's pin list, and the unpinned-families fold are all untouched.

…per family (BE-6911)

The weekly catalog-drift check listed 178 unpinned same-lab ids and then
head-truncated each lab group at 25 rows in Cursor's raw print order, so the
two genuinely new families in the 2026-08-10 catalog — gpt-5.6-terra-* and
gpt-5.6-luna-* — landed inside the hidden "… and 57 more" while the visible
rows were spent on gpt-5.2 and eight gpt-5.3-codex-* ids that are strictly
older than the current pin.

The rows are noise because the panel pins ONE reasoning/speed tier per family,
so every other tier of every pinned family is an unpinned "candidate" forever.
Collapse them: one row per (lab, family) naming the family, how many tiers the
catalog offers and which (strongest first), plus one member's catalog note
reproduced verbatim. Order rows newest-family-version first, ties broken by
catalog order, so MAX_FAMILY_IDS drops the oldest families rather than an
arbitrary tail of the print order.

Against the catalog captured in #144 this renders 37 rows instead of 178, both
new families sit at the top of the gpt group, and no group needs a "… and N
more" line at all. Rendering only — analyze(), the JSON report, urgent /
has_findings, the NO-ZDR handling on pins, and the unpinned-families fold are
untouched.
@mattmillerai mattmillerai added cursor-review Multi-model cursor review agent-coded Authored by the agent-work loop labels Aug 10, 2026
@coderabbitai

coderabbitai Bot commented Aug 10, 2026

Copy link
Copy Markdown

Warning

Review limit reached

You’ve reached a temporary PR review limit under our Fair Usage Limits Policy.

Your recent review volume is higher than typical usage, so adaptive limits are currently applied.

Next review available in: 31 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 0e93d1d7-04dc-441e-84bd-10c26fb01b34

📥 Commits

Reviewing files that changed from the base of the PR and between 935077f and 5281099.

📒 Files selected for processing (4)
  • .github/cursor-review/catalog-drift.py
  • .github/cursor-review/tests/fixtures/catalog-2026-08-10.txt
  • .github/cursor-review/tests/test_catalog_drift.py
  • README.md

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 Cursor Review — Consolidated panel

Triggered by @mattmillerai.

⚠️ Panel did not produce any findings.

Every reviewer in the matrix failed to contribute — see the panel summary for which cells errored, and the run logs for the underlying cause.

Panel: 0/8 reviewers contributed findings.

Reviewers that did not contribute: claude-opus-5-thinking-max:adversarial (error), gemini-3.1-pro:adversarial (error), gpt-5.6-sol-max:adversarial (error), kimi-k3-max:adversarial (error), claude-opus-5-thinking-max:edge-case (error), gemini-3.1-pro:edge-case (error), gpt-5.6-sol-max:edge-case (error), kimi-k3-max:edge-case (error)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent-coded Authored by the agent-work loop cursor-review Multi-model cursor review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants