Skip to content

Pin the rust-test macOS lane to Rust 1.97.1 with a self-expiring canary - #640

Open
ciaranra wants to merge 3 commits into
devfrom
fix-macos-rust-test-lane
Open

Pin the rust-test macOS lane to Rust 1.97.1 with a self-expiring canary#640
ciaranra wants to merge 3 commits into
devfrom
fix-macos-rust-test-lane

Conversation

@ciaranra

Copy link
Copy Markdown
Member

Summary

The rust-test macOS matrix lane fails on current stable: rustc 1.98.0 on aarch64-apple-darwin segfaults compiling ordinary dependency build scripts (libc, proc-macro2, quote) before any workspace crate builds. This is the same 1.98 aarch64-darwin crash that already pinned the python-compat-smoke macOS lane (issue #566); the rust-test lane was never pinned because its macOS job only runs on dispatch and pre-merge matrix runs.

Reproductions (identical signature, two dispatch runs on one branch plus dev itself):

Change

  • The Unix Rust setup step selects 1.97.1 on macos-latest and stable elsewhere (expression-level conditional; a matrix include would add a phantom macOS job on pull requests, where the matrix is ubuntu-only).
  • A self-expiring canary step (macOS, non-PR events only, never fails the job) probes latest stable with five plain cargo check runs — this lane's actual crashing invocation, unlike the smoke lane's maturin-link-args probe — and emits a workflow warning when the pin looks obsolete, mirroring the smoke-lane pattern and its unpin discipline (issue Unpin macOS python-compat-smoke Rust toolchain once the rustc 1.98 aarch64-darwin segfault is fixed #566).

The canary block intentionally duplicates the smoke lane's shape rather than extracting a shared script: the two probes differ in the invocation under test, and this keeps the change to one workflow.

Verification

Unpin path is unchanged and tracked in #566: remove the pin only after the canary warning persists across several PRs and an unpinned build passes on a draft PR.

@ciaranra

Copy link
Copy Markdown
Member Author

Live proof complete: the workflow_dispatch matrix run on this branch is fully green — rust-test (macos-latest) passes with the 1.97.1 pin (https://github.com/PECOS-packages/PECOS/actions/runs/33347969025), where the two unpinned runs cited in the description segfaulted in dependency build scripts. ubuntu and windows lanes unaffected and green.

@ciaranra

ciaranra commented Sep 8, 2026

Copy link
Copy Markdown
Member Author

The pin itself is well justified and the conditional is the right shape. One must-fix in the canary.

The canary cannot observe the crash it is probing for

The probe builds a bare project:

cargo +stable init --lib --name pin_probe "$probe_dir"
cargo +stable check --manifest-path "$probe_dir/Cargo.toml"

cargo init --lib emits a manifest with an empty [dependencies] section. I ran it: cargo check on that project compiles exactly one unit, the empty crate itself, and zero build scripts.

The failure this pin exists for is, in the PR's own words, rustc 1.98.0 on aarch64-apple-darwin segfaulting compiling ordinary dependency build scripts (libc, proc-macro2, quote). The probe compiles none of those, so it cannot reproduce the crash under any toolchain.

The consequence is the opposite of the stated intent. The step will record 0 crashes out of 5 on its very first run and emit:

Rust toolchain pin may be obsolete: rustc ... survived 5/5 cargo check probes on this runner.

while the pin is still genuinely required. A canary meant to stop the pin going stale silently instead argues for removing it from day one, and it will keep doing so no matter how broken stable is. The five repetitions and the rm -rf target between them do not help: five compiles of nothing are still nothing.

Worth noting because the "nondeterministic, so probe repeatedly" reasoning in the comment is sound, and it is what makes the empty probe look convincing rather than obviously wrong.

Fix

The probe needs to compile the crates whose build scripts crash. Either give the temporary project those dependencies:

[dependencies]
libc = "0.2"
proc-macro2 = "1"
quote = "1"

or point the probe at the real workspace with the pinned toolchain overridden, for example cargo +stable check -p pecos-core, which exercises the actual dependency graph and needs no synthetic manifest at all. The second is closer to "this lane's actual crashing invocation", which is what the PR says it is aiming for.

Whichever is chosen, the check worth doing before merge is the one I could not do from here: confirm on a macOS runner that the probe does report crashes against 1.98 stable. A canary that has never been seen to trip is indistinguishable from one that cannot.

The rest reads correctly

  • Selecting the toolchain by expression rather than a matrix include is right, and the reason given (a phantom macOS job on pull requests, where the matrix is ubuntu-only) is a real hazard.
  • Gating the canary on github.event_name != 'pull_request' and never failing the job is appropriate for an advisory probe.
  • Duplicating the smoke lane's shape rather than extracting a shared script is defensible when the invocations under test genuinely differ, which they do. That difference is exactly the point of this lane's probe being a plain cargo check rather than a maturin link-args probe -- which makes it more important, not less, that the plain cargo check actually builds something.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant