Stream action archive downloads to disk instead of buffering in memory - #4588
Open
benoit-nexthop wants to merge 1 commit into
Open
Stream action archive downloads to disk instead of buffering in memory#4588benoit-nexthop wants to merge 1 commit into
benoit-nexthop wants to merge 1 commit into
Conversation
DownloadRepositoryArchive called HttpClient.GetAsync with the default HttpCompletionOption.ResponseContentRead, which buffers the entire archive in the managed heap before ReadAsStreamAsync returns, so the subsequent CopyToAsync to the FileStream never actually streamed from the network. For actions hosted in large repositories the tarball can be hundreds of MB, raising peak worker memory by the full archive size per download (and per retry) and throwing OutOfMemoryException on memory-constrained self-hosted runners. Pass HttpCompletionOption.ResponseHeadersRead (and the download cancellation token) so the body is copied to disk with a fixed-size buffer.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #4587
DownloadRepositoryArchivecallsHttpClient.GetAsync(downloadUrl)with the defaultHttpCompletionOption.ResponseContentRead, which buffers the entire archive in the managed heap beforeReadAsStreamAsync()returns — so theCopyToAsyncinto theFileStreamcopies from memory, not from the network. Peak worker memory grows by the full archive size per download (~73 MB in our case for an action hosted in a monorepo, since the codeload tarball is a snapshot of the whole source tree), and the retry loop re-buffers the entire body on each attempt. On memory-constrained self-hosted runners this throwsSystem.OutOfMemoryExceptionduring the "Download action repository" phase (full details and job logs in #4587).This change passes
HttpCompletionOption.ResponseHeadersReadso the body is streamed to disk with a fixed-size buffer, making peak memory independent of archive size. It also passes the existing download cancellation token toGetAsync, so the download timeout covers the connect/headers phase as well.We've been running this change built from source on our Kubernetes-hosted runners and it resolves the OOM.
🤖 Generated with Claude Code