Repository navigation
fix(ns8): tell a registry blip apart from a missing image in the manifest selftest - #265
Conversation
There was a problem hiding this comment.
Caution
The validate job on this PR is already red, and not for anything in the code path it fixes: three of the new comment lines put a lowercase word directly after buildah , which the workflow's subcommand validator reads as a buildah subcommand. It fails with ::error::buildah has no subcommand 'actually' used by the build scripts before either manifest selftest runs.
Approach: sound — buildah exits 125 for every failure, so its error text is the only signal available; classifying it, retrying only transport failures, and defaulting to fail-fast is the right shape, and the two messages are correctly distinct in what they tell the reader to do.
Reviewed changes
- Transport-only retry in
add_arch: the per-archbuildah manifest addnow gets up to three attempts with 5s/10s backoff, with the outcome reported back throughADD_KIND/ADD_OUTPUT. - Fail-fast classifier:
is_network_errormatches a specific list of transport and 5xx strings, so an unrecognised error — including ghcr's 403 for a nonexistent repository — fails on the first attempt rather than looping. - Case-specific messages: transport failures state the tag was never reported absent and say not to rebuild; genuine absences keep the old
build-images.sh … TARGETARCH=… PUSH=1advice, and both print the last lines of buildah's own output.
deepseek-v4.1-flash | 𝕏
| # | ||
| # So: classify, retry only transport failures, and let the message name the case. | ||
| retry_attempts="${RETRY_ATTEMPTS:-3}" | ||
| retry_delay="${RETRY_DELAY:-5}" |
There was a problem hiding this comment.
The header's Environment: block documents every knob this script reads, and these two are the only ones missing — someone tuning the retry budget or running the script by hand looks there first. Worth adding RETRY_ATTEMPTS (default 3) and RETRY_DELAY (default 5s, doubled per retry) next to SOURCE_PREFIX/PUSH_DESTINATION.
…fest selftest
The registry-backed selftest failed on a TCP reset while reading an arm64 config
blob, and reported:
Failed to add docker://ghcr.io/calnode/ns8-calnode:build-arm64 to ...
If that tag does not exist in the registry, run build-images.sh with
TARGETARCH=arm64 PUSH=1 first.
The image existed. One connection dropped. That advice sends the reader to
rebuild and push an artefact that is already published, which is real, pointless
work, and the red build says nothing about the network.
Both halves are fixed where the truth is, in push-manifest.sh, because that is
the code that has buildah's actual error in hand. The workflow step only sees an
exit code, so wrapping it there would have meant guessing from a bare status.
RETRY. add_arch() retries `buildah manifest add` only when the error looks like a
transport failure: connection reset or refused, i/o timeout, deadline exceeded,
name resolution, no such host, unreachable network, TLS handshake, unexpected
EOF, 502/503/504, bad gateway, service unavailable, gateway timeout, internal
server error. Three attempts, 5s then 10s backoff, both overridable.
The classifier defaults to "real failure". Anything unrecognised fails on the
first attempt, immediately, with the existing message. That matters more than the
retry: an unfamiliar buildah error must never become a loop that hides a broken
release, and a genuine 404 must stay unmistakable.
MESSAGE. On a transport failure the script now says the registry was unreachable,
states explicitly that the tag was never reported absent, says nothing is wrong
with the build, and says not to rebuild. On a real absence it keeps the old
advice word for word, because there it is correct. The last lines of buildah's
own output are printed in both cases so the classification can be judged.
PROVEN with the real script and real buildah (docker run --privileged
quay.io/buildah/stable), not by injecting a raise:
happy path, registry, both arches exit 0, "Published ... (amd64 arm64)"
missing, nonexistent repository exit 1, 2s, old advice, no retries
missing, real repo, bad arch tag exit 1, 3s, old advice, no retries
("manifest unknown" from ghcr)
unreachable, localhost:1 exit 1, 15s, 2 retry lines then
"registry was unreachable", never the
missing-tag advice
hermetic selftest (SOURCE_PREFIX="") exit 0, unchanged
Worth recording from the second case: a nonexistent ghcr repository answers 403,
not 404, because the bearer-token request fails before any manifest lookup. It is
classified as a real failure by default, which is the right outcome, and it is
why the default matters.
No workflow change, no pin change, no permission change, no reformatting. One
file.
ad3bab3 to
80f7503
Compare

One file:
packaging/ns8/push-manifest.sh. No workflow change, no pin change, no permission change, no reformatting.The message now lives where the truth is
I fixed it in
push-manifest.sh, not in the workflow step, because that is the code holding buildah's actual error text. The step only sees an exit code, so patching the message there would have meant guessing the cause from a bare status. The step is untouched and still blocking.What changed
Retry, transport failures only.
add_arch()retriesbuildah manifest addwhen, and only when, the error looks like a transport problem: connection reset or refused, i/o timeout, deadline exceeded, name resolution, no such host, unreachable network, TLS handshake, unexpected EOF, 502/503/504, bad gateway, service unavailable, gateway timeout, internal server error. Three attempts, 5s then 10s backoff, both overridable.The classifier defaults to "real failure". Anything unrecognised fails on the first attempt, immediately, with the existing message. That matters more than the retry itself: an unfamiliar buildah error must never turn into a loop that hides a broken release, and a genuine 404 has to stay unmistakable.
The two messages are now different.
Both print the last lines of buildah's own output, so the classification can be judged rather than trusted.
Evidence, from the real script and real buildah
Run in
docker run --privileged quay.io/buildah/stableagainst the actual source. Nothing was simulated by editing the script to raise.Published ... (amd64 arm64)localhost:1SOURCE_PREFIX="")The third row is the one that proves the boundary:
ghcr.io/calnode/ns8-calnode:build-no-such-archreturnsmanifest unknownand fails in 3 seconds with no retry lines, while the first arch in the same run succeeded.The unreachable case shows the bound explicitly:
15 seconds, exactly two retries, and the missing-tag advice never appears.
One thing worth knowing
A nonexistent ghcr repository answers 403, not 404, because the bearer-token request fails before any manifest lookup happens. It lands in the "real failure" branch by default, which is the right outcome, and it is the reason the default has to be fail-fast rather than retry-everything.
Not done
The step still depends on the live registry. That is a genuine design smell, but it is a much larger change and the check is useful as-is.