ci: self-heal cargo caches that kill build scripts with a bare ENOENT - #38404
Open
ggevay wants to merge 1 commit into
Open
ci: self-heal cargo caches that kill build scripts with a bare ENOENT#38404ggevay wants to merge 1 commit into
ggevay wants to merge 1 commit into
Conversation
ggevay
force-pushed
the
gabor/clear-corrupted-target-dir-enoent
branch
from
August 21, 2026 18:07
ec7c9e0 to
754cb23
Compare
A corrupted cached build-script binary fails with "failed to run custom build command" plus a bare "Error: No such file or directory (os error 2)", a signature clear-corrupted-cargo-target-dir did not recognize, so the corruption persisted across runs instead of being wiped and retried. On the single-agent merge-skew queue this took the check down for every PR until the agent was replaced by hand. Require both halves of the signature, with the ENOENT as the exact bare error line: the first half alone is any build-script bug, which a retry cannot fix, and a build script that fails on a missing file with an error context of its own reports the ENOENT in a "Caused by:" detail line instead. Mirror the signature in run_and_detect_retryable_build_failure, which the sync comments already tie to the shell script. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FJqg3FMKRiGYJ4owEwVAdC
ggevay
force-pushed
the
gabor/clear-corrupted-target-dir-enoent
branch
from
August 21, 2026 18:10
754cb23 to
cffdc3b
Compare
ggevay
changed the title
ggevay
marked this pull request as ready for review
August 21, 2026 18:24
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
On 2026-08-21 the merge-skew-cargo-check step failed org-wide for about an hour: a corrupted cached build-script binary on the single merge-skew agent died with a bare "No such file or directory", a signature
bin/clear-corrupted-cargo-target-dirdoes not recognize, so the self-heal never fired and every PR's check stayed red until the agent was replaced by hand. See for example https://buildkite.com/materialize/test/builds/132291#01a02515-b30e-4809-b330-98cdb7fbfe9d (test-pipeline builds 132290 through 132298 all failed identically, across unrelated branches).Description
Recognize the signature in
bin/clear-corrupted-cargo-target-dirand wipe-and-retry, mirrored inrun_and_detect_retryable_build_failureper the existing sync comments. Both halves of the signature are required, and the ENOENT must be the exact bare error line: "failed to run custom build command" alone is any build-script bug, which a retry cannot fix, and a build script that fails on a missing file with an error context of its own reports the ENOENT indented under "Caused by:" rather than on the bare "Error:" line, so genuine build-script bugs (which are user code, not cache corruption) do not trigger cache wipes.A follow-up PR (kept in draft for now) adds a warning annotation on every self-healing wipe so that recurring corruption stays observable instead of manifesting only as repeatedly cold builds.
Verification
The script detects the actual incident log from build 132298 (exit 199), detects a synthetic bare-pair log, and correctly ignores both a log carrying only the "failed to run custom build command" half and a log where the ENOENT appears in a "Caused by:" detail line of a contextful build-script error. The Python side raises
RustIncrementalBuildFailurefor the bare pair and falls through to a plainCalledProcessErrorfor the contextful case.