[VL][CI] Add gpu image centos-9-jdk17-cuda13.1-cudf - #12690
Open
marin-ma wants to merge 1 commit into
Open
Conversation
marin-ma
changed the title
Contributor
There was a problem hiding this comment.
Pull request overview
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Adds a new CentOS Stream 9 CUDA 13.1 cuDF GPU Docker image intended to restore missing dependencies (S3/GCS/ABFS) and moves GPU CI usage toward JDK 17.
Changes:
- Adds a new Dockerfile for
centos-9-jdk17-cuda13.1-cudfthat builds Velox/Gluten with cloud filesystem flags enabled. - Extends the GitHub Actions workflow to build and push the new Docker image tag.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
| dev/docker/cudf/Dockerfile.centos-9-jdk17-cuda13.1-cudf | Introduces a new CUDA 13.1 cuDF build image definition intended for JDK 17 + cloud filesystem deps. |
| .github/workflows/docker_image.yml | Adds a CI job to build/push the new Docker image tag. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+18
to
+33
| FROM quay.io/centos/centos:stream9 | ||
| ENV CUDA_ARCHITECTURES=75 | ||
| ENV LD_LIBRARY_PATH=/opt/gluten/ep/build-velox/build/velox_ep/_build/release/_deps/curl-build/lib:$LD_LIBRARY_PATH | ||
| ENV CC=/opt/rh/gcc-toolset-14/root/bin/gcc \ | ||
| CXX=/opt/rh/gcc-toolset-14/root/bin/g++ | ||
|
|
||
| RUN dnf config-manager --add-repo "https://developer.download.nvidia.com/compute/cuda/repos/rhel9/x86_64/cuda-rhel9.repo"; \ | ||
| dnf update; \ | ||
| dnf install -y sudo patch maven perl git gcc-toolset-14 cuda-toolkit-13-1 && \ | ||
| dnf autoremove -y && dnf clean all; \ | ||
| git clone --depth=1 https://github.com/apache/incubator-gluten /opt/gluten && \ | ||
| cd /opt/gluten && \ | ||
| source /opt/rh/gcc-toolset-14/enable && \ | ||
| bash ./dev/buildbundle-veloxbe.sh --run_setup_script=ON --build_arrow=ON --spark_version=3.5 --build_tests=ON --build_benchmarks=ON --enable_s3=ON --enable_gcs=ON --enable_abfs=ON --enable_gpu=ON && \ | ||
| rm -rf /opt/gluten && \ | ||
| rm -rf /root/.cache/ccache |
Comment on lines
+24
to
+27
| RUN dnf config-manager --add-repo "https://developer.download.nvidia.com/compute/cuda/repos/rhel9/x86_64/cuda-rhel9.repo"; \ | ||
| dnf update; \ | ||
| dnf install -y sudo patch maven perl git gcc-toolset-14 cuda-toolkit-13-1 && \ | ||
| dnf autoremove -y && dnf clean all; \ |
| dnf update; \ | ||
| dnf install -y sudo patch maven perl git gcc-toolset-14 cuda-toolkit-13-1 && \ | ||
| dnf autoremove -y && dnf clean all; \ | ||
| git clone --depth=1 https://github.com/apache/incubator-gluten /opt/gluten && \ |
Comment on lines
+153
to
+157
| - name: "node-cleanup" # by default the free runner does not have enough disk space | ||
| run: | | ||
| sudo rm -rf /usr/share/dotnet /usr/local/lib/android /opt/ghc /opt/hostedtoolcache/CodeQL | ||
| sudo docker image prune --all --force | ||
| sudo docker builder prune -a |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The jdk version in the gpu image
apache/gluten:centos-9-jdk8-cudfis now jdk17 after #11835, and some dependencies such as aws-sdk-cpp are missing in the image because it's no longer based on the velox image. Some gpu jobs that rely on theapache/gluten:centos-9-jdk8-cudfimage have failed due to this change.This pr adds a new gpu image
centos-9-jdk17-cuda13.1-cudfwith build flags--enable_s3=ON --enable_gcs=ON --enable_abfs=ONso the essential dependencies will be installed. Keeping the jdk8 version seems unnecessary. We can switch to jdk17 for the gpu build.