The backend image reached 14.1GB. docker pull then ran past the deploy step's 40 minute command_timeout, so Deploy to EC2 failed with Run Command Timeout and no indication of the cause.
This blocked #136 entirely: it merged green and then could not be deployed. The interrupted pull left nothing cached, so a re-run just restarted the same doomed download.
Where the size came from
Measured in the running container:
| Layer / package |
Size |
pip install -r requirements.txt |
12.1GB |
site-packages/nvidia |
4.3GB |
site-packages/torch |
1.7GB |
site-packages/triton |
592MB |
That is ~6.6GB of CUDA runtime on a 2-CPU EC2 instance with no GPU, which cannot execute any of it.
Fixed
Install CPU-only torch before requirements.txt, so the pinned torch==2.9.0 is already satisfied and pip never reaches for the CUDA build. PEP 440 treats the local version segment as compatible, so 2.9.0+cpu satisfies ==2.9.0 and requirements.txt needs no change — this changes the build of torch, not the version. Wheel confirmed to exist for the exact version and platform.
A build-time size ceiling (8GB) now fails the build rather than the deploy, so the feedback lands on the PR that causes it. Builds stayed green through this entire incident while the cost appeared only on the host, an hour later, on an environment that could no longer be deployed to.
Result
14.1GB → 3.31GB. Deploy pipeline green end to end; the guard reports Image size: 3.31 GB (ceiling 8 GB).
PRs: DataSpaceBackend#137, DataSpaceBackend#138
The backend image reached 14.1GB.
docker pullthen ran past the deploy step's 40 minutecommand_timeout, soDeploy to EC2failed withRun Command Timeoutand no indication of the cause.This blocked #136 entirely: it merged green and then could not be deployed. The interrupted pull left nothing cached, so a re-run just restarted the same doomed download.
Where the size came from
Measured in the running container:
pip install -r requirements.txtsite-packages/nvidiasite-packages/torchsite-packages/tritonThat is ~6.6GB of CUDA runtime on a 2-CPU EC2 instance with no GPU, which cannot execute any of it.
Fixed
Install CPU-only torch before
requirements.txt, so the pinnedtorch==2.9.0is already satisfied and pip never reaches for the CUDA build. PEP 440 treats the local version segment as compatible, so2.9.0+cpusatisfies==2.9.0andrequirements.txtneeds no change — this changes the build of torch, not the version. Wheel confirmed to exist for the exact version and platform.A build-time size ceiling (8GB) now fails the build rather than the deploy, so the feedback lands on the PR that causes it. Builds stayed green through this entire incident while the cost appeared only on the host, an hour later, on an environment that could no longer be deployed to.
Result
14.1GB → 3.31GB. Deploy pipeline green end to end; the guard reports
Image size: 3.31 GB (ceiling 8 GB).PRs: DataSpaceBackend#137, DataSpaceBackend#138