ci: annotate every self-healing cargo cache wipe - #38405
Draft
ggevay wants to merge 2 commits into
Draft
Conversation
A corrupted cached build-script binary fails with "failed to run custom build command" plus a bare "Error: No such file or directory (os error 2)", a signature clear-corrupted-cargo-target-dir did not recognize, so the corruption persisted across runs instead of being wiped and retried. On the single-agent merge-skew queue this took the check down for every PR until the agent was replaced by hand. Require both halves of the signature, with the ENOENT as the exact bare error line: the first half alone is any build-script bug, which a retry cannot fix, and a build script that fails on a missing file with an error context of its own reports the ENOENT in a "Caused by:" detail line instead. Mirror the signature in run_and_detect_retryable_build_failure, which the sync comments already tie to the shell script. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FJqg3FMKRiGYJ4owEwVAdC
The wipe-and-retry mechanism makes the build green, so if cache corruption ever became frequent its only symptom would be builds running cold over and over, attributed to nothing. Emit a warning annotation on every wipe, from the shell script and from both Python handlers, so each self-heal event stays visible on the build page and countable across builds, while the build still succeeds. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FJqg3FMKRiGYJ4owEwVAdC
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Follow-up to #38404 (whose commit this branch includes; rebase after it merges). The self-healing wipe-and-retry in
bin/clear-corrupted-cargo-target-dirand its Python counterparts ends in a green build, so if cargo cache corruption ever became frequent, nothing would fail: the only symptom would be builds repeatedly running cold, with the lost caches and the slowness attributed to nothing.Description
Emit a warning annotation on every self-healing cache wipe, covering the pre-existing signatures as well, from the shell script and from both Python handlers (
ci/test/build.pyand the cargo-test mzcompose workflow), via a sharedbuildkite.annotate_cache_wipehelper. Warning style renders as a yellow callout and does not affect build status; each wipe stays visible on the build page and countable across builds through the annotations API. The helpers are structurally unable to change the caller's exit code (env guards,|| true, try/except).A possible further step, deliberately left out for now, is feeding these events into the test-analytics database so they appear in
bin/ci-failuresand the CI dashboard's Failures tab; that touches the analytics write path and wants a closer look from CI folks first.Verification
Both annotation paths were exercised against a stubbed
buildkite-agenton PATH withBUILDKITE_JOB_IDset: called exactly once with the expected style, context, and message. With the CI variables absent, the helpers verifiably skip. The wipe paths themselves behave identically to #38404 (same smoke matrix incl. the real incident log).