[deckhouse-cli] fix mirror pull silently loses image on fail#393
Open
Glitchy-Sheep wants to merge 2 commits into
Open
[deckhouse-cli] fix mirror pull silently loses image on fail#393Glitchy-Sheep wants to merge 2 commits into
Glitchy-Sheep wants to merge 2 commits into
Conversation
… retry AddImage recorded metaByTag before AppendImage. A transient network failure during blob download left the (tag, digest) pair in the map, so the idempotency guard treated the retry as already-done and returned nil without writing anything. The pull finished with exit 0 while the image was missing from the bundle, surfacing later as ImagePullBackOff in the cluster. Record metaByTag only after AppendImage succeeds and add regression tests covering both the unit-level retry and the full pull/push flow with an injected transient blob failure. Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
… layout - `PullImageSet` now cross-checks the pull plan against the layout: every planned image must be present in `index.json` under its short tag. - An incomplete bundle fails the pull with the list of missing references, while the user still has network access. - Matching by `short_tag` annotation avoids false positives on multi-arch digest rewrites; nil metas (AllowMissingTags) are skipped. - Tests cover both branches: a missing image trips the check with exact references, a healthy digest- and tag-style pull passes. Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
[mirror] Fix silent image loss on pull retry after failed write
Summary
Network failure during
mirror pullcould cause a silent drop of an image from bundle.But summary of the pull reports the success.
A cluster therefore could have ImagePullBackOff on the missing digest.
Problem
ImageLayout.AddImagerecordedmetaByTag[tag]beforeAppendImagewrotethe image.
When the blob download broke during write, the
(tag, digest)pairstayed in the map, so the retry hit the idempotency guard and returned
success without writing anything:
Fix
metaByTagonly afterAppendImagesucceeds: a failed write leavesno trace and the retry performs a real write.
PullImageSet, verify every planned image ispresent in the layout. An incomplete bundle now fails the pull with the
list of missing images instead of exiting 0.
Before / After
Before: the pull looks successful, the loss is invisible:
The mismatch surfaces only as
[362 / 362] Pushing ...vs 363 pulled, and dayslater in the air-gapped cluster: 404 error and ImagePullBackoff.
After: the same network blip heals itself - the retry re-downloads the
image and the bundle is complete:
If an image still fails to land for any reason, the pull fails on the spot,
while the user still has network access and can simply re-run it:
Tests
TestRepro_AddImage_RetryAfterFailedWrite_ImageMustLand- unit: failedwrite + retry must land the image (red before the fix).
TestRepro_AddImage_NoFailure_Control- guard still dedups repeated adds.TestRepro_MirrorPullThenPush_TransientBlobFailureLosesImage- e2e: pullthrough a registry that resets one blob download, push, verify every digest
resolves in the target registry.
TestVerifyPlannedImagesLanded_ReportsMissingImages- the completenesscheck reports exactly the missing references.
TestPullImageSet_CompletePullPassesVerification- a healthy pull ofdigest- and tag-style references does not trip the check.