ci: stop pushing a nightly image on every commit to main - #8
Merged
Conversation
GHCR refused a nightly push on 2026-08-07 with `403 permission_denied` and a secondary rate limit body, which reads like a permissions problem and is not one. Four full nightly pushes had run between 12:04 and 12:23, all triggered by documentation commits and all overwriting the same tag. A nightly is now pushed only when the push changed something the image contains. The new `changes` job decides that with a deny list rather than an allow list, because the Dockerfile copies the whole repository into its build stage: anything the list has not heard of counts as relevant, and a redundant nightly is cheaper than a stale one. Releases are never filtered. Three things reduce the pressure further: The workflow level concurrency group is gone. It serialised every run, so a queued nightly still ran its full push once the run ahead of it finished, turning a burst of commits into a burst of registry writes spread over an hour. Release Please keeps its own serialised group, and the container job cancels superseded nightlies while leaving releases alone. SBOM and provenance are release only. Each attestation is a further manifest pushed per platform, and on a tag overwritten several times a day the evidence is replaced before anyone reads it. Both pushes retry once after waiting out the registry. Buildx has no backoff of its own, so a single throttled manifest PUT used to lose an entire multi platform build; the retry finds every layer in the cache.
Test results 68 files 68 suites 9m 31s ⏱️ Results for commit f4b5d06. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
On 2026-08-07 the
Container imagejob failed like this:Worth knowing before anyone debugs this again: GHCR reports a throttled manifest
PUT as
403 permission_denied. It looks likepackages: writeis missing. It isnot.
What ran into the limit was volume. Every commit on main pushed both images for
two architectures, with an SBOM and
provenance: mode=maxon each. Four of thoseran between 12:04 and 12:23 that day, every one of them triggered by a
documentation commit, every one overwriting the same
nightlytag.What changed
A nightly is only pushed when the push changed something the image contains.
The new
changesjob diffs the pushed range against a deny list. Deny ratherthan allow, because
Dockerfile:24copies the whole repository into the buildstage: anything the list has not heard of counts as relevant. A redundant nightly
costs minutes, a missing one costs somebody an afternoon. Releases are never
filtered, and a force push or rewritten history falls through to a build.
Checked against real history on this repository:
779e9ebedocs: write CLAUDE.md in English6b51a2f9docs: write down the drop-in compatibility rule05bfbb5drefactor(ci): give the container checks a shared library1fe847d0fix(ci): read file ownership through a container47bd5a0bbuild(docker): build both architectures without emulatingAll four pushes from the 12:0x burst would have been skipped.
The workflow level concurrency group is gone. It had
cancel-in-progress: false, which serialised every run of the workflow. That sounds harmless and isnot: a queued nightly still runs its full push once the run ahead of it finishes,
so a burst of commits becomes a burst of registry writes spread over an hour.
Because runs never overlapped, a job level cancel could never have fired either.
Release Please now carries that serialised group itself, where it belongs, and
the container job has its own: nightlies share
container-nightly-<ref>andcancel each other, releases are keyed by version and cancel nothing.
SBOM and provenance are release only. Each attestation is a further manifest
pushed per platform, and on a tag overwritten several times a day it is replaced
before anyone reads it. Releases keep the full set.
Both pushes retry once, after waiting 180 seconds. Buildx has no backoff, so
one refused manifest PUT used to lose an entire two platform build. The retry is
cheap because every layer is still in the cache and only the push is replayed.
CLAUDE.mdis updated to match: it previously stated that every push carries anSBOM and provenance.
Reviewing this
The two
with:blocks per image are duplicated between the attempt and theretry, and there is a comment saying to keep them in sync. A composite action
would have removed the duplication, but whether
steps.<id>.outcomeis readableinside one is undocumented, and a retry that silently never fires is worse than
the redundancy.
actionlintincluding its shellcheck pass is clean on both workflow files.Not in this PR
Three further findings from the same investigation, none of which have caused a
failure yet:
so it evicts constantly and
cache-from: type=ghamisses more often than not.across three jobs, against a limit of 100 per 6 hours per shared runner IP.
CI run.
TRIVY_USERNAMEandTRIVY_PASSWORDare unset.Unrelated, found on the way: the Codecov upload fails with
{"message":"Repository not found"}and is hidden byfail_ci_if_error: false.Coverage reporting has not worked at all.