fix: install CPU-only torch so the image can actually be deployed - #137
Merged
Conversation
The image is 14.1GB and `docker pull` now runs past the deploy step's 40 minute command_timeout, so deploys fail outright. That is what stopped the row-lock fix in #136 from reaching dev - it merged green and then could not be deployed. Where the size comes from, measured in the running container: 12.1GB RUN pip install -r requirements.txt 4.3GB site-packages/nvidia 1.7GB site-packages/torch 592MB site-packages/triton That is roughly 6.6GB of CUDA runtime on a 2-CPU EC2 instance with no GPU, which cannot execute any of it. Installing CPU-only torch first means the pinned torch==2.9.0 is already satisfied and pip never reaches for the CUDA build. PEP 440 treats the local version segment as compatible, so 2.9.0+cpu satisfies ==2.9.0 and requirements.txt needs no change. The wheel was confirmed to exist for this exact version and platform (torch-2.9.0+cpu-cp310-cp310-manylinux_2_28_x86_64.whl), so this changes the build of torch, not the version. --no-cache-dir on both installs drops the pip wheel cache from the layer. This is a prerequisite now rather than an optimisation: nothing else can deploy until the pull fits inside the timeout.
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
This was referenced Sep 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Blocking. Deploys are currently failing and this is why.
The problem
The image is 14.1GB.
docker pullnow runs past the deploy step's 40 minutecommand_timeout, soDeploy to EC2fails withRun Command Timeout. That is what stopped #136 (the row-lock fix) from reaching dev — it merged green and then could not be deployed. I confirmed the interrupted pull left nothing cached, so a re-run just restarts the same doomed download.Where the size comes from
Measured in the running container:
pip install -r requirements.txtsite-packages/nvidiasite-packages/torchsite-packages/tritonThat is ~6.6GB of CUDA runtime on a 2-CPU EC2 instance with no GPU, which cannot execute any of it.
The change
Install CPU-only torch before
requirements.txt, so the pinnedtorch==2.9.0is already satisfied and pip never reaches for the default CUDA build.PEP 440 treats the local version segment as compatible, so
2.9.0+cpusatisfies==2.9.0andrequirements.txtneeds no change. I confirmed the wheel exists for this exact version and platform:torch-2.9.0+cpu-cp310-cp310-manylinux_2_28_x86_64.whl. This changes the build of torch, not the version.--no-cache-diron both installs drops the pip wheel cache from the layer.Worth checking at review
If anything in the codebase assumes CUDA is available it will now find none — though on this box it never had a GPU, so runtime behaviour there should be unchanged. Inference via
transformers/torchwill run on CPU, as it already effectively did.Effect
Once merged, the next build from
devcarries both this and the #136 row-lock fix, so a single successful deploy lands both. Verification of the row-lock fix follows on #136 once it is actually running.