Skip to content

fix(ns8): tell a registry blip apart from a missing image in the manifest selftest - #265

Merged
shockalotti merged 1 commit into
mainfrom
fix/manifest-selftest-network-vs-missing
Nov 18, 2026
Merged

shockalotti merged 1 commit into
mainfrom
fix/manifest-selftest-network-vs-missing

Conversation

@shockalotti

Copy link
Copy Markdown
Contributor

One file: packaging/ns8/push-manifest.sh. No workflow change, no pin change, no permission change, no reformatting.

The message now lives where the truth is

I fixed it in push-manifest.sh, not in the workflow step, because that is the code holding buildah's actual error text. The step only sees an exit code, so patching the message there would have meant guessing the cause from a bare status. The step is untouched and still blocking.

What changed

Retry, transport failures only. add_arch() retries buildah manifest add when, and only when, the error looks like a transport problem: connection reset or refused, i/o timeout, deadline exceeded, name resolution, no such host, unreachable network, TLS handshake, unexpected EOF, 502/503/504, bad gateway, service unavailable, gateway timeout, internal server error. Three attempts, 5s then 10s backoff, both overridable.

The classifier defaults to "real failure". Anything unrecognised fails on the first attempt, immediately, with the existing message. That matters more than the retry itself: an unfamiliar buildah error must never turn into a loop that hides a broken release, and a genuine 404 has to stay unmistakable.

The two messages are now different.

  • Transport failure: says the registry was unreachable, says explicitly that the tag was never reported absent, says nothing is wrong with the build, and says do not rebuild.
  • Real absence: keeps the old advice word for word, because there it is correct.

Both print the last lines of buildah's own output, so the classification can be judged rather than trusted.

Evidence, from the real script and real buildah

Run in docker run --privileged quay.io/buildah/stable against the actual source. Nothing was simulated by editing the script to raise.

case result
happy path, live registry, both arches exit 0, Published ... (amd64 arm64)
missing: nonexistent repository exit 1, 2s, old advice, no retries
missing: real repo, bad arch tag exit 1, 3s, old advice, no retries
unreachable: localhost:1 exit 1, 15s, 2 retry lines, then "registry was unreachable"
hermetic selftest (SOURCE_PREFIX="") exit 0, unchanged

The third row is the one that proves the boundary: ghcr.io/calnode/ns8-calnode:build-no-such-arch returns manifest unknown and fails in 3 seconds with no retry lines, while the first arch in the same run succeeded.

The unreachable case shows the bound explicitly:

Registry unreachable (attempt 1/3), retrying in 5s: docker://localhost:1/...
Registry unreachable (attempt 2/3), retrying in 10s: docker://localhost:1/...
The registry was unreachable while reading docker://localhost:1/...
This is a TRANSPORT failure, not a missing image: ...

15 seconds, exactly two retries, and the missing-tag advice never appears.

One thing worth knowing

A nonexistent ghcr repository answers 403, not 404, because the bearer-token request fails before any manifest lookup happens. It lands in the "real failure" branch by default, which is the right outcome, and it is the reason the default has to be fail-fast rather than retry-everything.

Not done

The step still depends on the live registry. That is a genuine design smell, but it is a much larger change and the check is useful as-is.

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

The validate job on this PR is already red, and not for anything in the code path it fixes: three of the new comment lines put a lowercase word directly after buildah , which the workflow's subcommand validator reads as a buildah subcommand. It fails with ::error::buildah has no subcommand 'actually' used by the build scripts before either manifest selftest runs.

Approach: sound — buildah exits 125 for every failure, so its error text is the only signal available; classifying it, retrying only transport failures, and defaulting to fail-fast is the right shape, and the two messages are correctly distinct in what they tell the reader to do.

Reviewed changes

  • Transport-only retry in add_arch: the per-arch buildah manifest add now gets up to three attempts with 5s/10s backoff, with the outcome reported back through ADD_KIND/ADD_OUTPUT.
  • Fail-fast classifier: is_network_error matches a specific list of transport and 5xx strings, so an unrecognised error — including ghcr's 403 for a nonexistent repository — fails on the first attempt rather than looping.
  • Case-specific messages: transport failures state the tag was never reported absent and say not to rebuild; genuine absences keep the old build-images.sh … TARGETARCH=… PUSH=1 advice, and both print the last lines of buildah's own output.

Pullfrog  | Fix all ➔ | Fix 👍s ➔ | View workflow run | Using deepseek-v4.1-flash | 𝕏

Comment thread packaging/ns8/push-manifest.sh Resolved
#
# So: classify, retry only transport failures, and let the message name the case.
retry_attempts="${RETRY_ATTEMPTS:-3}"
retry_delay="${RETRY_DELAY:-5}"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The header's Environment: block documents every knob this script reads, and these two are the only ones missing — someone tuning the retry budget or running the script by hand looks there first. Worth adding RETRY_ATTEMPTS (default 3) and RETRY_DELAY (default 5s, doubled per retry) next to SOURCE_PREFIX/PUSH_DESTINATION.

…fest selftest

The registry-backed selftest failed on a TCP reset while reading an arm64 config
blob, and reported:

  Failed to add docker://ghcr.io/calnode/ns8-calnode:build-arm64 to ...
  If that tag does not exist in the registry, run build-images.sh with
  TARGETARCH=arm64 PUSH=1 first.

The image existed. One connection dropped. That advice sends the reader to
rebuild and push an artefact that is already published, which is real, pointless
work, and the red build says nothing about the network.

Both halves are fixed where the truth is, in push-manifest.sh, because that is
the code that has buildah's actual error in hand. The workflow step only sees an
exit code, so wrapping it there would have meant guessing from a bare status.

RETRY. add_arch() retries `buildah manifest add` only when the error looks like a
transport failure: connection reset or refused, i/o timeout, deadline exceeded,
name resolution, no such host, unreachable network, TLS handshake, unexpected
EOF, 502/503/504, bad gateway, service unavailable, gateway timeout, internal
server error. Three attempts, 5s then 10s backoff, both overridable.

The classifier defaults to "real failure". Anything unrecognised fails on the
first attempt, immediately, with the existing message. That matters more than the
retry: an unfamiliar buildah error must never become a loop that hides a broken
release, and a genuine 404 must stay unmistakable.

MESSAGE. On a transport failure the script now says the registry was unreachable,
states explicitly that the tag was never reported absent, says nothing is wrong
with the build, and says not to rebuild. On a real absence it keeps the old
advice word for word, because there it is correct. The last lines of buildah's
own output are printed in both cases so the classification can be judged.

PROVEN with the real script and real buildah (docker run --privileged
quay.io/buildah/stable), not by injecting a raise:

  happy path, registry, both arches     exit 0, "Published ... (amd64 arm64)"
  missing, nonexistent repository       exit 1, 2s, old advice, no retries
  missing, real repo, bad arch tag      exit 1, 3s, old advice, no retries
                                        ("manifest unknown" from ghcr)
  unreachable, localhost:1              exit 1, 15s, 2 retry lines then
                                        "registry was unreachable", never the
                                        missing-tag advice
  hermetic selftest (SOURCE_PREFIX="")  exit 0, unchanged

Worth recording from the second case: a nonexistent ghcr repository answers 403,
not 404, because the bearer-token request fails before any manifest lookup. It is
classified as a real failure by default, which is the right outcome, and it is
why the default matters.

No workflow change, no pin change, no permission change, no reformatting. One
file.
@shockalotti
shockalotti force-pushed the fix/manifest-selftest-network-vs-missing branch from ad3bab3 to 80f7503 Compare November 18, 2026 21:23
@shockalotti
shockalotti merged commit f7c3695 into main Nov 18, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant