Skip to content

fix(bench): a paired base build skips a package the base does not have - #9022

Merged
kixelated merged 4 commits into
mainfrom
fix/bench-base-missing-package
Nov 17, 2026
Merged

kixelated merged 4 commits into
mainfrom
fix/bench-base-missing-package

Conversation

@kixelated

@kixelated kixelated commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

Today's "Rust supplemental benchmarks" nightly (run 44127032469) compared 22 targets (347 cases) against nothing. The paired base build failed:

error: package ID specification `moq-e2ee` did not match any packages

moq-dev/moq#8947 made moq-e2ee its own crate again, so the previous nightly (13e593c) has no such package. criterion_build_base already drops the --bench names the base lacks, but it kept -p moq-e2ee. Cargo then rejected the whole base build, and every target in the leg went unpaired.

Approach

criterion_build_base reads the base's package names from the cargo metadata call it already makes. It matches each -p/--package selector against them as a pattern, the way cargo reads one, so a literal name or a glob such as moq-* keeps whatever the base has. A selector that names none of the base's packages is dropped, so that package's targets count as new, as a new bench target's do. The matched names also scope the bench-name lookup. If no selected package is left, it builds nothing rather than letting cargo build the whole workspace.

A paired run (BENCH_BASE set) rejects at setup, with exit 2, a package spec the base cannot match by name: -p name@version, a URL spec, or the joined --package=NAME and -pNAME forms. Otherwise such a spec would match no base name and its existing benches would pass as new.

Impact

  • No public API or wire change. This touches only the bench harness.

Validation

  • bench/build.slow.test.py gains a new-package case: the current tree adds a package with a bench, and the run names both packages.
    • Against the old bench/run.sh: it fails with moq-e2ee/protect: base build failed, the error CI hit.
    • With this change: the base builds and runs moq-relay/catalog, the new package's bench runs unpaired, and nothing is reported unavailable.
  • A glob case (-p 'moq-rel*' --bench catalog) checks that a package pattern still pairs an existing bench. It failed against this PR's first two heads, which the reviewer caught.
  • A test_base_rejects_a_package_it_cannot_match case runs bench/run.sh --criterion with -p moq-relay@0.0.0, --package=moq-relay, and -pmoq-relay and sees exit 2 with the error; run by hand too.
  • python3 bench/build.slow.test.py and just check pass.
  • Local review: Codex gpt-6.1-sol medium, no findings on 4fd481a.

Caused by

  • moq-dev/moq#6511: introduced the paired base build, filtering bench names but not packages, so a new package fails the whole base build (found by check: the nightly paired bench). moq-dev/moq#8947 only exposed it by adding a package.

Alternatives

  • Building each package's base separately. That costs a cargo invocation per package for a case that arises only when a package is added.

Follow-ups

None.

Models

Size: XS
Implementer: Claude Code, claude-opus-5-5, unknown effort
Implementer: Claude Code, claude-opus-5-5, low effort (iterate, XS, arm B)

(Written by Claude Opus 5.5)

🤖 Generated with Claude Code

Review

Rewritten by quest review after each round.

  • Head: 4fd481a

  • Base: origin/main at 2caa851

  • Risk: medium, the base has no RISK.md to name what is always high or customer-facing, so every change is medium

  • Classified: medium by default

  • Reviewer: Codex, gpt-6.1-sol, medium effort

  • Session: 01a1eb63-8093-7f21-922b-2c592f2463cf, round 1

  • Score: 9/10, The package-selection changes and regression coverage appear sound with no actionable regressions found, though builds and tests were not run as instructed.

  • Rounds: 1; first round 0; findings 0: fixed 0, refuted 0, open 0

  • Findings: none

  • Reviewer: Codex, gpt-6.1-sol, low effort, shadow

  • Session: 01a1eb63-8094-7b91-8cee-9c5d3a128c29, round 1

  • Score: 9/10, Source review found no actionable regression and focused regression coverage is included, but builds and tests were not run as instructed.

  • Rounds: 1; first round 0; findings 0: fixed 0, refuted 0, open 0

  • Findings: none

kixelated and others added 3 commits November 17, 2026 10:58
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated
kixelated marked this pull request as ready for review November 17, 2026 19:05
@coderabbitai

coderabbitai Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 23 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 4 included reviews currently available.

Learn how review limits work.

Review configuration:

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 9d034121-3fbe-4e9b-9695-eb0af23ed146
Commits

Reviewing files that changed from the base of the PR and between 2dcb5f8 and 4fd481a.

Files selected for processing: 2

  • bench/build.slow.test.py
  • bench/run.sh
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

pullfrog[bot]
pullfrog Bot previously approved these changes Nov 17, 2026

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Approach: sound. It filters -p selectors against the base's own cargo metadata in the one function that builds the base, so a new package's targets count as new, the same way a new bench target's already do. It reuses the existing mechanism and doesn't add a per-package cargo invocation.

Reviewed changes: I reviewed the full diff of bench/run.sh criterion_build_base and its slow test.

  • Package selectors filtered against the base: each -p/--package pattern is glob-matched against the base's package names. A selector that matches nothing is dropped, and if none are left the function returns early instead of letting cargo build the whole workspace.
  • Bench-name lookup scoped by matched names: the jq filter now receives the resolved package names rather than the raw pattern. This also makes -p 'moq-rel*' find that package's bench targets.
  • Single cargo metadata call: the function fetches the metadata once and reuses it for both lookups.
  • Tests: two new cases. new-package covers a package the base lacks, and glob covers a package pattern that should still pair an existing bench.

Pullfrog  | View workflow run | Using claude-opus-5-5 | 𝕏

@kixelated kixelated left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review by review (OpenAI)
Reviewed commit: eeeedfdaf023b4d7f301c668c9ab1401957ad3c9

bench/run.sh:432–451: Filtering against bare .packages[].name silently drops Cargo package ID specs. For example, -p moq-relay@0.0.0 with that exact package in both revisions builds the current leg, but the base matches no bare name and returns success without building anything. That can label an existing benchmark as new and hide a regression. Match version-qualified selectors against the base metadata as well, or explicitly reject unsupported selectors rather than treating them as absent. The fixture already uses version 0.0.0, so it can cover this with one additional case.

Reusing one metadata call is the right direction and fixes the nightly's plain-name selectors. Static review; I did not run the slow test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated

Copy link
Copy Markdown
Collaborator Author

Addressed the review's version-qualified selector finding in 4fd481a: a paired run now rejects -p name@version, URL specs, and the joined --package=NAME/-pNAME forms at setup (exit 2) instead of dropping them as absent, with a test for each. Local Codex review of 4fd481a found nothing.

(Written by Claude Opus 5.5 with low)

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes since the prior pullfrog review (eeeedfd):

  • Unmatchable package specs rejected on paired runs: with BASE set, -p name@ver, URL specs, and the joined --package=X / -pX forms exit 2 with a clear error. This closes the silent "existing bench passes as new" hole the earlier external review raised.
  • Tests: test_base_rejects_a_package_it_cannot_match covers all three forms, and the worktree-cleanup assertion checks that no stray worktree is left behind.

Pullfrog  | View workflow run | Using claude-sonnet-5-5 | 𝕏

@kixelated
kixelated added this pull request to the merge queue Nov 17, 2026
Merged via the queue into main with commit f88e8f5 Nov 17, 2026
7 checks passed
@kixelated
kixelated deleted the fix/bench-base-missing-package branch November 17, 2026 21:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant