docker CI fails for every PR: poetry export emits two conflicting nvidia-cublas pins

[linhongyu510]

Bug

The docker workflow (pytest-api) now fails for every PR at the image build step, with a pip resolution error that has nothing to do with the PR's contents:

ERROR: Cannot install nvidia-cublas==13.1.1.3 and nvidia-cublas==13.8.0.4
       because these package versions have conflicting dependencies.
The conflict is caused by:
    The user requested nvidia-cublas==13.1.1.3
    The user requested nvidia-cublas==13.8.0.4
ERROR: ResolutionImpossible
make: *** [Makefile:13: run] Error 1

Seen on run 35437983494 (PR #2144, which changes one line of tests/pytorch/test_transforms_pt.py and nothing under api/).

Why it is not the PR's fault

api/Dockerfile:18 runs make lock at image build time:

RUN pip install --upgrade pip setuptools wheel \
    && make lock \
    && pip install -r /app/requirements.txt

make lock is poetry lock + poetry export, and no lock file is committed, so every build re-resolves against whatever is newest on PyPI. api/pyproject.toml was last touched on 2026-08-21.

The timeline lines up exactly:

date	event
2026-08-21	last change to api/pyproject.toml
2026-09-16	nvidia-cublas==13.8.0.4 published to PyPI
2026-09-18	last successful docker run (05d0615)
2026-09-19	first docker run after that date — fails

Since 09-19 there have been 5 docker runs; the 4 others are still action_required, so the next PR that gets its workflows approved will hit the same wall.

Root cause

torch 2.14.0 asks for two mutually exclusive things on Linux:

• directly: nvidia-cublas==13.8.0.4, nvidia-cuda-nvrtc==13.4.92
• transitively, via cuda-toolkit==13.0.3[cublas,cudart,cufft,cufile,cupti,curand,cusolver,cusparse,nvjitlink,nvrtc,nvtx], which pins nvidia-cublas==13.1.1.3.* and nvidia-cuda-nvrtc==13.0.88.*

poetry lock accepts this because the two sets carry different markers, and poetry export writes both lines out. The markers overlap on the image's own platform, so pip receives two pins for one package.

Reproduced locally with nothing but the two files the Dockerfile copies:

$ cp api/pyproject.toml api/Makefile /tmp/repro/ && cd /tmp/repro
$ poetry lock && poetry export -f requirements.txt --without-hashes -o requirements.txt
$ grep -n nvidia-cublas requirements.txt
28:nvidia-cublas==13.1.1.3 ; ... and platform_system == "Linux" and (platform_machine == "aarch64" or platform_machine == "x86_64")
29:nvidia-cublas==13.8.0.4 ; ... and platform_system == "Linux"

Evaluating both markers for the base image's environment (tiangolo/uvicorn-gunicorn-fastapi:python3.11-slim → Linux, x86_64, CPython 3.11) with packaging.markers:

nvidia-cublas==13.1.1.3   applies here: True
nvidia-cublas==13.8.0.4   applies here: True

Both are active, so the conflict is unavoidable.

Two packages are affected, not one — the CI log stops at the first:

package	pins active on linux/x86_64
nvidia-cublas	13.1.1.3, 13.8.0.4
nvidia-cuda-nvrtc	13.0.88, 13.4.92

A side observation

The export pulls 15 nvidia-* CUDA wheels into the API image (cublas, cudnn-cu13, nccl-cu13, cusparselt-cu13, nvshmem-cu13, …). The API serves CPU inference from a -slim base, so these are several GB of layers that are never used.

Possible directions

I did not send a PR because each option changes how the image is built and you are better placed to pick:

1. Commit a lock file and have the Dockerfile install from it instead of running make lock during the build. This makes the image reproducible and stops PyPI drift from breaking unrelated PRs — but it needs a refresh process.
2. Install CPU-only torch in the API. I tried [[tool.poetry.source]] with https://download.pytorch.org/whl/cpu: it removes the conflict and drops the export from 71 to 51 lines with 0 CUDA packages — but the export then pins torch==2.14.0+cpu, and since poetry export writes no index URL, pip install -r requirements.txt fails with No matching distribution found. So it needs --extra-index-url handling in the Dockerfile to work, which is your call.
3. Post-process the export to drop the losing duplicate. Smallest change, but it papers over a resolver artefact.

Happy to prepare whichever you prefer.

Environment

Reproduced with poetry 2.5.1 / CPython 3.11.15 against api/pyproject.toml at c7f7218.