[AntTheLimey]

What changed

infra_error matched quota anywhere in the stderr-plus-answer text
of a run with no tool calls. A Codex Luna answer to task 6 ("...unless
that service or repository has its own quota") was filed as infra, and
the runner stopped the release batch after 34 of 180 runs. A bare
quota now counts only in stderr; the other limit phrases still match
either.

cross_arm_problems now ignores MCP:cursor. Cursor's built-in
server was listed by the with arm in 2 of 3 control runs and by the
without arm in the third, so the check failed at random.

Checks

make test-evals: 75 tests pass. The new
test_quota_in_an_answer_is_not_infra fails with the run.py change
reverted, and the added cross_arm_problems assertion fails with
that change reverted.

Summary by CodeRabbit

• Bug Fixes
  • Successful answers mentioning “quota” are no longer incorrectly reported as infrastructure errors. Quota errors reported in stderr continue to be identified.
  • MCP:cursor is no longer reported as an extra item in comparisons between evaluation arms.
• Tests
  • Added coverage for quota error detection and evaluation-arm comparisons.

[coderabbitai]

No actionable comments were generated in the recent review.

Walkthrough

The evaluation runner changes how it detects quota-related infrastructure errors and how it identifies items reported only by the with arm. Tests cover quota wording in answers and stderr, and MCP:cursor in the with arm.

Changes: Quota classification

Layer / File(s)	Summary
Infrastructure-error check and test evals/run.py, evals/test_run.py	The general infrastructure-error match checks quota in stderr, not final-answer text. Tests cover quota wording in both sources. The checks still require that no calls were recorded.

Cross-arm filtering

Layer / File(s)	Summary
Cross-arm filtering and test evals/run.py, evals/test_run.py	cross_arm_problems excludes MCP:cursor, as well as SKILLS:pgedge*, from items listed only by the with arm. The test covers a with arm that reports only MCP:cursor and an empty without arm.

Estimated code review effort: 2 (Simple) | ~8 minutes

Change: Bug fix

Merge Risk: Minimal · up to 77cdb

No merge-blocking issue was established; normal checks remain appropriate.

Pre-merge checks: 5

Check name	Status	Explanation
Description Check		Check skipped - CodeRabbit’s high-level summary is enabled.
Title check		The title clearly identifies both main fixes: quota detection and Cursor MCP handling. It is concise and directly related to the changeset.
Docstring Coverage		Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentat…
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.

Full details: Docstring Coverage

Explanation

Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 2 functions across 2 files.

Finishing Touches

Generate docstrings

• Commit to this branch
• Create a new PR

Generate unit tests (beta)

• Commit to this branch
• Create a new PR

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Recent review info

Run configuration

• Source mode: Auto
• Repository: pgEdge/pgedge-cli
• Branch: fix/eval-quota-false-infra
• File: .coderabbit.yaml
• Commit: 77cdb59
• Layers: Repository YAML
• Review profile: CHILL
• Plan: Team
• Run ID: e2e371cf-f7bc-470c-8d27-76875acaf362

Commits

Reviewing files that changed from the base of the PR and between df30148 and 77cdb59.

Files selected for processing: 2

• evals/run.py
• evals/test_run.py

Included review availability: This review used your included allowance. 4 included reviews remain after this review. Your included PR review attempts over the past 7 days set your current allowance at 8 reviews per hour.