Python 3.13 support 02 base images - #765
Draft
kmontemayor2-sc wants to merge 5 commits into
Draft
Conversation
…ilures fatal Python 3.12 removed `distutils` and the deprecated `unittest` aliases, and GiGL uses both. The `distutils` imports work today only because `setuptools` is installed transitively and ships a compatibility shim; a library must not depend on that. - Add `gigl.common.utils.parse.str_to_bool` and use it in place of `distutils.util.strtobool` at all 15 call sites. Accepted and rejected spellings match `strtobool`. Every call site already coerced the result with `bool()` or used it as a condition, so the `int` to `bool` return change is not observable. - Rename the 26 `assertEquals` / `assertNotEquals` uses to `assertEqual` / `assertNotEqual`, and select ruff `UP005` so they cannot return. - Make a failed `install_glt.sh` fatal. `main()` returned the child's status but the `__main__` block discarded it, so `requirements/install_py_deps.sh` saw exit 0 under `set -e` and every base image build continued after a failed GLT install. Measured against a stub that exits 7: the old script exits 0, the new one exits 7. A successful install still exits 0. - Run `ty` twice in `make type_check`, at the 3.11 floor and at 3.13. `ty` resolves the standard library against one version per invocation, so the floor pass accepts modules 3.13 removed and only the ceiling pass rejects them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
tensorflow-data-validation 1.21.0 publishes only manylinux_2_39_x86_64 wheels, so no image below glibc 2.39 can take it. The CUDA base was Ubuntu 22.04 (glibc 2.35) and the Dataflow base was Debian bookworm (glibc 2.36). Both move here, on today's lockfile and still on Python 3.11, so that when the dependency stack moves the failure mode is attributable to the stack rather than to the OS or interpreter underneath it. - CUDA base: `nvidia/cuda:12.8.1-cudnn-devel-ubuntu24.04`, digest pinned. Drops `UV_SYSTEM_PYTHON` and the conda interpreter, so the image now builds its own `/gigl_deps/.venv` from `.python-version` exactly as the CPU base does. Verified in the built image: Python 3.11.14, torch 2.8.0+cu128, CUDA 12.8, glibc 2.39. - Dataflow base: adopts Beam's custom-container shape — own base OS, the boot harness copied from the SDK image, and an explicit ENTRYPOINT, which was previously inherited. `RUN_PYTHON_SDK_IN_DEFAULT_ENVIRONMENT=1` is required: without it boot builds a nested venv that cannot see this one, and every worker dies reporting apache-beam missing. - Dataflow images are now built for linux/amd64 only. tensorflow-data-validation ships no aarch64 wheel, so a working arm64 image was never possible; the published manifest's arm64 entry shares layer digests with amd64. - Deletes the `--inexact` branch in the installers. It existed only for the two images that set `UV_SYSTEM_PYTHON`, and neither does now. - Hardens `has_cuda_driver()` in both copies. Callers use it as an `if` condition, which suspends `set -e`, so a missing `whereis` reported "no CUDA" while the build still exited 0. That was survivable while the CUDA base shipped torch preinstalled and `--inexact` kept it; without either, it would silently install CPU torch and a WITH_CUDA=OFF GLT into a GPU image. - Adds `scripts/smoke_test_image.py`, which asserts interpreter, ABI tag, active venv, glibc floor, a caller-declared import set, and optionally CUDA, Beam version and the boot environment variable. The import set is required rather than defaulted: base images carry a metadata-only gigl-core, and the Dataflow image has no GLT by design, so a fixed list cannot describe every image. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Collaborator
Author
|
/e2e_test |
Contributor
GiGL Automation@ 03:46:18UTC : 🔄 @ 05:23:40UTC : ✅ Workflow completed successfully. |
Contributor
GiGL Automation@ 15:50:35UTC : Starting to build base images for CUDA and CPU. |
Contributor
GiGL Automation@ 16:01:48UTC : Built and pushed new images:
Updated |
Collaborator
Author
|
/e2e_test |
Contributor
GiGL Automation@ 16:59:08UTC : 🔄 |
`libcuda.so.1` and `libnvidia-ml.so.1` are the host NVIDIA driver and ship in no image.
Vertex AI and GKE bind-mount them into /usr/local/nvidia/lib{,64} at container start, so
both directories are empty while the image builds. `nvidia/cuda` sets
LD_LIBRARY_PATH=/usr/local/cuda/lib64 and drops them, which broke GPU two ways at once:
- `fbgemm_gpu` names libcuda.so.1 in DT_NEEDED across 19 of its extensions, so
`import torchrec` raised outright. Every trainer reaching `gigl.nn` died, because
`gigl/nn/__init__.py` imports `gigl/nn/models.py` eagerly.
- Everything else imported torch fine, reported no CUDA device, and trained on CPU while
reporting success. Five of nine e2e pipelines passed that way on two T4s each.
Move to `pytorch/pytorch:2.10.0-cuda12.8-cudnn9-devel`, the oldest torch tag on Ubuntu
24.04, which exports the driver paths as the previous 2.8.0 base did. Its own torch and
Python 3.12 go unused: GiGL pins `requires-python = "==3.11.*"`, so install_py_deps.sh
still builds /gigl_deps/.venv from .python-version, and the venv leads PATH. Set both
variables here anyway rather than inherit them, because upstream shuffles them between
releases: 2.11.0 already puts /usr/local/cuda/lib64 first.
Verified in the built image: Ubuntu 24.04, glibc 2.39, nvcc 12.8, Python 3.11.14 from the
venv, torch 2.8.0+cu128. With the driver directory left empty, `import torchrec` fails as
it did in production; with the host driver mounted into it, the same image imports
torchrec and torch 2.8.0+cu128.
`smoke_test_image.py` gains `torchrec` as a checkable import, so the broken chain can be
asserted directly, and `--require-nvidia-driver-path`, which compares LD_LIBRARY_PATH
entries whole. That check needs no GPU, so it can gate an image build on any machine.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
GiGL Automation@ 00:35:38UTC : Starting to build base images for CUDA and CPU. |
Contributor
GiGL Automation@ 00:48:08UTC : Built and pushed new images:
Updated |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Scope of work done
Where is the documentation for this feature?: N/A
Did you add automated tests or write a test plan?
Updated Changelog.md? NO
Ready for code review?: NO