The Sign stage intermittently fails when signing many images because an in-flight signature PUT is canceled by a 10s per-request timeout. It only passes when the whole stage is re-run.
Failing run: https://dev.azure.com/dnceng/internal/_build/results?buildId=3010858 (Sign stage)
Polly.Timeout.TimeoutRejectedException: The operation didn't complete within the allowed timeout of '00:00:10'. Execution Time: 10005ms
HTTP PUT .../dotnet/aspnet/manifests/sha256:abc123...
at OrasDotNetService.PushSignatureAsync ... line 132
at ImageSigningService.SignImagesAsync ... line 110
The uploads themselves during signing are small. Payload size is not the cause. The failure is a transient/slow registry response under load: the image-builder-oras-timeout handler in src/ImageBuilder/ImageBuilder.cs applies AddTimeout(TimeSpan.FromSeconds(10)), a total per-attempt timeout, and ImageSigningService.SignImagesAsync pushes all signatures concurrently via Parallel.ForEachAsync. When one PUT exceeds 10s, it is canceled.
The Polly retry does not recover it because retryOptions.DisableForUnsafeHttpMethods() disables retries for PUT/POST, so the request is logged as Handled: 'False' and the command fails outright, requiring manual re-run.
We should make signature uploads resilient to transient registry slowness so the stage does not depend on a re-run. Possible directions worth investigating (not prescriptive):
- bound parallelism during signing
- enabling retries for idempotent digest-addressed writes
- inactivity-based request timeout (allow unlimited timeout as long as bytes are actively being uploaded)
The
Signstage intermittently fails when signing many images because an in-flight signaturePUTis canceled by a 10s per-request timeout. It only passes when the whole stage is re-run.Failing run: https://dev.azure.com/dnceng/internal/_build/results?buildId=3010858 (
Signstage)The uploads themselves during signing are small. Payload size is not the cause. The failure is a transient/slow registry response under load: the
image-builder-oras-timeouthandler insrc/ImageBuilder/ImageBuilder.csappliesAddTimeout(TimeSpan.FromSeconds(10)), a total per-attempt timeout, andImageSigningService.SignImagesAsyncpushes all signatures concurrently viaParallel.ForEachAsync. When onePUTexceeds 10s, it is canceled.The Polly retry does not recover it because
retryOptions.DisableForUnsafeHttpMethods()disables retries forPUT/POST, so the request is logged asHandled: 'False'and the command fails outright, requiring manual re-run.We should make signature uploads resilient to transient registry slowness so the stage does not depend on a re-run. Possible directions worth investigating (not prescriptive):