Repository navigation
fix(bench): a paired base build skips a package the base does not have - #9022
Conversation
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Warning Review limit reachedYou've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Next included review available in 23 minutes. View limit details
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
There was a problem hiding this comment.
✅ No new issues found.
Approach: sound. It filters -p selectors against the base's own cargo metadata in the one function that builds the base, so a new package's targets count as new, the same way a new bench target's already do. It reuses the existing mechanism and doesn't add a per-package cargo invocation.
Reviewed changes: I reviewed the full diff of bench/run.sh criterion_build_base and its slow test.
- Package selectors filtered against the base: each
-p/--packagepattern is glob-matched against the base's package names. A selector that matches nothing is dropped, and if none are left the function returns early instead of letting cargo build the whole workspace. - Bench-name lookup scoped by matched names: the jq filter now receives the resolved package names rather than the raw pattern. This also makes
-p 'moq-rel*'find that package's bench targets. - Single
cargo metadatacall: the function fetches the metadata once and reuses it for both lookups. - Tests: two new cases.
new-packagecovers a package the base lacks, andglobcovers a package pattern that should still pair an existing bench.
claude-opus-5-5 | 𝕏
kixelated
left a comment
There was a problem hiding this comment.
Automated review by review (OpenAI)
Reviewed commit: eeeedfdaf023b4d7f301c668c9ab1401957ad3c9
bench/run.sh:432–451: Filtering against bare .packages[].name silently drops Cargo package ID specs. For example, -p moq-relay@0.0.0 with that exact package in both revisions builds the current leg, but the base matches no bare name and returns success without building anything. That can label an existing benchmark as new and hide a regression. Match version-qualified selectors against the base metadata as well, or explicitly reject unsupported selectors rather than treating them as absent. The fixture already uses version 0.0.0, so it can cover this with one additional case.
Reusing one metadata call is the right direction and fixes the nightly's plain-name selectors. Static review; I did not run the slow test.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Addressed the review's version-qualified selector finding in 4fd481a: a paired run now rejects (Written by Claude Opus 5.5 with low) |
There was a problem hiding this comment.
✅ No new issues found.
Reviewed changes since the prior pullfrog review (eeeedfd):
- Unmatchable package specs rejected on paired runs: with
BASEset,-p name@ver, URL specs, and the joined--package=X/-pXforms exit 2 with a clear error. This closes the silent "existing bench passes as new" hole the earlier external review raised. - Tests:
test_base_rejects_a_package_it_cannot_matchcovers all three forms, and the worktree-cleanup assertion checks that no stray worktree is left behind.
claude-sonnet-5-5 | 𝕏

Problem
Today's "Rust supplemental benchmarks" nightly (run 44127032469) compared 22 targets (347 cases) against nothing. The paired base build failed:
moq-dev/moq#8947 made
moq-e2eeits own crate again, so the previous nightly (13e593c) has no such package.criterion_build_basealready drops the--benchnames the base lacks, but it kept-p moq-e2ee. Cargo then rejected the whole base build, and every target in the leg went unpaired.Approach
criterion_build_basereads the base's package names from thecargo metadatacall it already makes. It matches each-p/--packageselector against them as a pattern, the way cargo reads one, so a literal name or a glob such asmoq-*keeps whatever the base has. A selector that names none of the base's packages is dropped, so that package's targets count as new, as a new bench target's do. The matched names also scope the bench-name lookup. If no selected package is left, it builds nothing rather than letting cargo build the whole workspace.A paired run (
BENCH_BASEset) rejects at setup, with exit 2, a package spec the base cannot match by name:-p name@version, a URL spec, or the joined--package=NAMEand-pNAMEforms. Otherwise such a spec would match no base name and its existing benches would pass as new.Impact
Validation
bench/build.slow.test.pygains anew-packagecase: the current tree adds a package with a bench, and the run names both packages.bench/run.sh: it fails withmoq-e2ee/protect: base build failed, the error CI hit.moq-relay/catalog, the new package's bench runs unpaired, and nothing is reported unavailable.globcase (-p 'moq-rel*' --bench catalog) checks that a package pattern still pairs an existing bench. It failed against this PR's first two heads, which the reviewer caught.test_base_rejects_a_package_it_cannot_matchcase runsbench/run.sh --criterionwith-p moq-relay@0.0.0,--package=moq-relay, and-pmoq-relayand sees exit 2 with the error; run by hand too.python3 bench/build.slow.test.pyandjust checkpass.Caused by
Alternatives
Follow-ups
None.
Models
Size: XS
Implementer: Claude Code, claude-opus-5-5, unknown effort
Implementer: Claude Code, claude-opus-5-5, low effort (iterate, XS, arm B)
(Written by Claude Opus 5.5)
🤖 Generated with Claude Code
Review
Rewritten by
quest reviewafter each round.Head: 4fd481a
Base: origin/main at 2caa851
Risk: medium,
the base has no RISK.md to name what is always high or customer-facing, so every change is mediumClassified: medium by default
Reviewer: Codex, gpt-6.1-sol, medium effort
Session:
01a1eb63-8093-7f21-922b-2c592f2463cf, round 1Score: 9/10,
The package-selection changes and regression coverage appear sound with no actionable regressions found, though builds and tests were not run as instructed.Rounds: 1; first round 0; findings 0: fixed 0, refuted 0, open 0
Findings: none
Reviewer: Codex, gpt-6.1-sol, low effort, shadow
Session:
01a1eb63-8094-7b91-8cee-9c5d3a128c29, round 1Score: 9/10,
Source review found no actionable regression and focused regression coverage is included, but builds and tests were not run as instructed.Rounds: 1; first round 0; findings 0: fixed 0, refuted 0, open 0
Findings: none