Unable to use more than 5 GPU cards

[diandian2023]

What is being built is a kata environment. The host has 8 GPU cards. If 1-5 GPU cards are used to create a pod, nvidia-container-cli will run normally, but problems will occur if 6 GPUs are used. After locating, the main reason is that the code calls ns_enter, switches to the rootfs of the container, and cannot find the corresponding directory when mounting_procfs. The environment version information is as follows:

nvidia-container-toolkit version

root@ubuntu-dev:/# dpkg -l | grep nvidia-container-toolkit
ii nvidia-container-toolkit 1.14.3-1 amd64 NVIDIA Container toolkit
ii nvidia-container-toolkit-base 1.14.3-1 amd64 NVIDIA Container Toolkit Base

nvidia-container-cli，The running log information is as follows：

`
-- WARNING, the following logs are for debugging purposes only --

I0111 10:30:12.940268 149 nvc.c:376] initializing library context (version=1.14.1, build=1eb5a30a6ad0415550a9df632ac8832bf7e2bbba)
I0111 10:30:12.940332 149 nvc.c:350] using root /
I0111 10:30:12.940334 149 nvc.c:351] using ldcache /etc/ld.so.cache
I0111 10:30:12.940336 149 nvc.c:352] using unprivileged user 65534:65534
I0111 10:30:12.940347 149 nvc.c:393] attempting to load dxcore to see if we are running under Windows Subsystem for Linux (WSL)
I0111 10:30:12.940484 149 nvc.c:395] dxcore initialization failed, continuing assuming a non-WSL environment
I0111 10:30:12.943220 179 nvc.c:278] loading kernel module nvidia
I0111 10:30:12.943338 179 nvc.c:282] running mknod for /dev/nvidiactl
I0111 10:30:12.943362 179 nvc.c:286] running mknod for /dev/nvidia0
I0111 10:30:12.943374 179 nvc.c:286] running mknod for /dev/nvidia1
I0111 10:30:12.943382 179 nvc.c:286] running mknod for /dev/nvidia2
I0111 10:30:12.943390 179 nvc.c:286] running mknod for /dev/nvidia3
I0111 10:30:12.943399 179 nvc.c:286] running mknod for /dev/nvidia4
I0111 10:30:12.943407 179 nvc.c:286] running mknod for /dev/nvidia5
I0111 10:30:12.943415 179 nvc.c:290] running mknod for all nvcaps in /dev/nvidia-caps
I0111 10:30:12.947452 179 nvc.c:218] running mknod for /dev/nvidia-caps/nvidia-cap1 from /proc/driver/nvidia/capabilities/mig/config
I0111 10:30:12.947504 179 nvc.c:218] running mknod for /dev/nvidia-caps/nvidia-cap2 from /proc/driver/nvidia/capabilities/mig/monitor
I0111 10:30:12.949693 179 nvc.c:296] loading kernel module nvidia_uvm
I0111 10:30:12.949702 179 nvc.c:300] running mknod for /dev/nvidia-uvm
I0111 10:30:12.949735 179 nvc.c:305] loading kernel module nvidia_modeset
I0111 10:30:12.955489 179 nvc.c:309] running mknod for /dev/nvidia-modeset
I0111 10:30:12.955701 183 rpc.c:71] starting driver rpc service
I0111 10:30:19.067427 229 rpc.c:71] starting nvcgo rpc service
I0111 10:30:19.072896 149 nvc_container.c:246] configuring container with 'compute utility supervised'
I0111 10:30:19.077084 149 nvc_container.c:88] selecting /run/kata-containers/ae1f8199611632c96d7e2ef8a5d5f51894d377259f062f6336911d02f67474d0/rootfs/usr/local/cuda-12.3/compat/libcuda.so.545.23.08
I0111 10:30:19.077456 149 nvc_container.c:88] selecting /run/kata-containers/ae1f8199611632c96d7e2ef8a5d5f51894d377259f062f6336911d02f67474d0/rootfs/usr/local/cuda-12.3/compat/libcudadebugger.so.545.23.08
I0111 10:30:19.077803 149 nvc_container.c:88] selecting /run/kata-containers/ae1f8199611632c96d7e2ef8a5d5f51894d377259f062f6336911d02f67474d0/rootfs/usr/local/cuda-12.3/compat/libnvidia-nvvm.so.545.23.08
I0111 10:30:19.078148 149 nvc_container.c:88] selecting /run/kata-containers/ae1f8199611632c96d7e2ef8a5d5f51894d377259f062f6336911d02f67474d0/rootfs/usr/local/cuda-12.3/compat/libnvidia-ptxjitcompiler.so.545.23.08
I0111 10:30:19.079973 149 nvc_container.c:268] setting pid to 147
I0111 10:30:19.080003 149 nvc_container.c:269] setting rootfs to /run/kata-containers/ae1f8199611632c96d7e2ef8a5d5f51894d377259f062f6336911d02f67474d0/rootfs
I0111 10:30:19.080011 149 nvc_container.c:270] setting owner to 0:0
I0111 10:30:19.080018 149 nvc_container.c:271] setting bins directory to /usr/bin
I0111 10:30:19.080038 149 nvc_container.c:272] setting libs directory to /usr/lib/x86_64-linux-gnu
I0111 10:30:19.080045 149 nvc_container.c:273] setting libs32 directory to /usr/lib/i386-linux-gnu
I0111 10:30:19.080052 149 nvc_container.c:274] setting cudart directory to /usr/local/cuda
I0111 10:30:19.080058 149 nvc_container.c:275] setting ldconfig to @/sbin/ldconfig.real (host relative)
I0111 10:30:19.080064 149 nvc_container.c:276] setting mount namespace to /proc/147/ns/mnt
I0111 10:30:19.080070 149 nvc_container.c:278] detected cgroupv1
I0111 10:30:19.080077 149 nvc_container.c:279] setting devices cgroup to /sys/fs/cgroup/devices/663e4a00_6864_4e19_8a5f_15d850583969/ae1f8199611632c96d7e2ef8a5d5f51894d377259f062f6336911d02f67474d0
I0111 10:30:19.080134 149 nvc_info.c:798] requesting driver information with ''
I0111 10:30:19.083496 149 nvc_info.c:176] selecting /usr/lib/x86_64-linux-gnu/vdpau/libvdpau_nvidia.so.535.146.02
I0111 10:30:19.083715 149 nvc_info.c:176] selecting /usr/lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so.535.146.02
I0111 10:30:19.083826 149 nvc_info.c:176] selecting /usr/lib/x86_64-linux-gnu/libnvidia-pkcs11.so.535.146.02
I0111 10:30:19.083920 149 nvc_info.c:176] selecting /usr/lib/x86_64-linux-gnu/libnvidia-pkcs11-openssl3.so.535.146.02
I0111 10:30:19.084025 149 nvc_info.c:176] selecting /usr/lib/x86_64-linux-gnu/libnvidia-opticalflow.so.535.146.02
I0111 10:30:19.084159 149 nvc_info.c:176] selecting /usr/lib/x86_64-linux-gnu/libnvidia-opencl.so.535.146.02
I0111 10:30:19.084254 149 nvc_info.c:176] selecting /usr/lib/x86_64-linux-gnu/libnvidia-nvvm.so.535.146.02
I0111 10:30:19.084412 149 nvc_info.c:176] selecting /usr/lib/x86_64-linux-gnu/libnvidia-ml.so.535.146.02
I0111 10:30:19.084542 149 nvc_info.c:176] selecting /usr/lib/x86_64-linux-gnu/libnvidia-fbc.so.535.146.02
I0111 10:30:19.084657 149 nvc_info.c:176] selecting /usr/lib/x86_64-linux-gnu/libnvidia-encode.so.535.146.02
I0111 10:30:19.084783 149 nvc_info.c:176] selecting /usr/lib/x86_64-linux-gnu/libnvidia-cfg.so.535.146.02
I0111 10:30:19.084883 149 nvc_info.c:176] selecting /usr/lib/x86_64-linux-gnu/libnvidia-allocator.so.535.146.02
I0111 10:30:19.084963 149 nvc_info.c:176] selecting /usr/lib/x86_64-linux-gnu/libnvcuvid.so.535.146.02
I0111 10:30:19.085071 149 nvc_info.c:176] selecting /usr/lib/x86_64-linux-gnu/libcudadebugger.so.535.146.02
I0111 10:30:19.085101 149 nvc_info.c:176] selecting /usr/lib/x86_64-linux-gnu/libcuda.so.535.146.02
W0111 10:30:19.085145 149 nvc_info.c:402] missing library libnvidia-nscq.so
W0111 10:30:19.085150 149 nvc_info.c:402] missing library libnvidia-gpucomp.so
W0111 10:30:19.085152 149 nvc_info.c:402] missing library libnvidia-fatbinaryloader.so
W0111 10:30:19.085155 149 nvc_info.c:402] missing library libnvidia-compiler.so
W0111 10:30:19.085158 149 nvc_info.c:402] missing library libnvidia-ngx.so
W0111 10:30:19.085160 149 nvc_info.c:402] missing library libnvidia-eglcore.so
W0111 10:30:19.085163 149 nvc_info.c:402] missing library libnvidia-glcore.so
W0111 10:30:19.085165 149 nvc_info.c:402] missing library libnvidia-tls.so
W0111 10:30:19.085167 149 nvc_info.c:402] missing library libnvidia-glsi.so
W0111 10:30:19.085170 149 nvc_info.c:402] missing library libnvidia-ifr.so
W0111 10:30:19.085172 149 nvc_info.c:402] missing library libnvidia-rtcore.so
W0111 10:30:19.085175 149 nvc_info.c:402] missing library libnvoptix.so
W0111 10:30:19.085177 149 nvc_info.c:402] missing library libGLX_nvidia.so
W0111 10:30:19.085180 149 nvc_info.c:402] missing library libEGL_nvidia.so
W0111 10:30:19.085182 149 nvc_info.c:402] missing library libGLESv2_nvidia.so
W0111 10:30:19.085184 149 nvc_info.c:402] missing library libGLESv1_CM_nvidia.so
W0111 10:30:19.085187 149 nvc_info.c:402] missing library libnvidia-glvkspirv.so
W0111 10:30:19.085189 149 nvc_info.c:402] missing library libnvidia-cbl.so
W0111 10:30:19.085192 149 nvc_info.c:406] missing compat32 library libnvidia-ml.so
W0111 10:30:19.085194 149 nvc_info.c:406] missing compat32 library libnvidia-cfg.so
W0111 10:30:19.085197 149 nvc_info.c:406] missing compat32 library libnvidia-nscq.so
W0111 10:30:19.085199 149 nvc_info.c:406] missing compat32 library libcuda.so
W0111 10:30:19.085202 149 nvc_info.c:406] missing compat32 library libcudadebugger.so
W0111 10:30:19.085204 149 nvc_info.c:406] missing compat32 library libnvidia-opencl.so
W0111 10:30:19.085206 149 nvc_info.c:406] missing compat32 library libnvidia-gpucomp.so
W0111 10:30:19.085209 149 nvc_info.c:406] missing compat32 library libnvidia-ptxjitcompiler.so
W0111 10:30:19.085211 149 nvc_info.c:406] missing compat32 library libnvidia-fatbinaryloader.so
W0111 10:30:19.085214 149 nvc_info.c:406] missing compat32 library libnvidia-allocator.so
W0111 10:30:19.085216 149 nvc_info.c:406] missing compat32 library libnvidia-compiler.so
W0111 10:30:19.085219 149 nvc_info.c:406] missing compat32 library libnvidia-pkcs11.so
W0111 10:30:19.085221 149 nvc_info.c:406] missing compat32 library libnvidia-pkcs11-openssl3.so
W0111 10:30:19.085230 149 nvc_info.c:406] missing compat32 library libnvidia-nvvm.so
W0111 10:30:19.085233 149 nvc_info.c:406] missing compat32 library libnvidia-ngx.so
W0111 10:30:19.085235 149 nvc_info.c:406] missing compat32 library libvdpau_nvidia.so
W0111 10:30:19.085238 149 nvc_info.c:406] missing compat32 library libnvidia-encode.so
W0111 10:30:19.085241 149 nvc_info.c:406] missing compat32 library libnvidia-opticalflow.so
W0111 10:30:19.085243 149 nvc_info.c:406] missing compat32 library libnvcuvid.so
W0111 10:30:19.085246 149 nvc_info.c:406] missing compat32 library libnvidia-eglcore.so
W0111 10:30:19.085249 149 nvc_info.c:406] missing compat32 library libnvidia-glcore.so
W0111 10:30:19.085251 149 nvc_info.c:406] missing compat32 library libnvidia-tls.so
W0111 10:30:19.085254 149 nvc_info.c:406] missing compat32 library libnvidia-glsi.so
W0111 10:30:19.085256 149 nvc_info.c:406] missing compat32 library libnvidia-fbc.so
W0111 10:30:19.085259 149 nvc_info.c:406] missing compat32 library libnvidia-ifr.so
W0111 10:30:19.085262 149 nvc_info.c:406] missing compat32 library libnvidia-rtcore.so
W0111 10:30:19.085264 149 nvc_info.c:406] missing compat32 library libnvoptix.so
W0111 10:30:19.085267 149 nvc_info.c:406] missing compat32 library libGLX_nvidia.so
W0111 10:30:19.085270 149 nvc_info.c:406] missing compat32 library libEGL_nvidia.so
W0111 10:30:19.085272 149 nvc_info.c:406] missing compat32 library libGLESv2_nvidia.so
W0111 10:30:19.085275 149 nvc_info.c:406] missing compat32 library libGLESv1_CM_nvidia.so
W0111 10:30:19.085277 149 nvc_info.c:406] missing compat32 library libnvidia-glvkspirv.so
W0111 10:30:19.085280 149 nvc_info.c:406] missing compat32 library libnvidia-cbl.so
I0111 10:30:19.085495 149 nvc_info.c:302] selecting /usr/bin/nvidia-smi
I0111 10:30:19.085511 149 nvc_info.c:302] selecting /usr/bin/nvidia-debugdump
I0111 10:30:19.085525 149 nvc_info.c:302] selecting /usr/bin/nvidia-persistenced
I0111 10:30:19.085549 149 nvc_info.c:302] selecting /usr/bin/nvidia-cuda-mps-control
I0111 10:30:19.085563 149 nvc_info.c:302] selecting /usr/bin/nvidia-cuda-mps-server
W0111 10:30:19.085591 149 nvc_info.c:428] missing binary nv-fabricmanager
I0111 10:30:19.085667 149 nvc_info.c:488] listing firmware path /lib/firmware/nvidia/535.146.02/gsp_ga10x.bin
I0111 10:30:19.085671 149 nvc_info.c:488] listing firmware path /lib/firmware/nvidia/535.146.02/gsp_tu10x.bin
I0111 10:30:19.085688 149 nvc_info.c:561] listing device /dev/nvidiactl
I0111 10:30:19.085691 149 nvc_info.c:561] listing device /dev/nvidia-uvm
I0111 10:30:19.085693 149 nvc_info.c:561] listing device /dev/nvidia-uvm-tools
I0111 10:30:19.085696 149 nvc_info.c:561] listing device /dev/nvidia-modeset
W0111 10:30:19.085712 149 nvc_info.c:352] missing ipc path /var/run/nvidia-persistenced/socket
W0111 10:30:19.085724 149 nvc_info.c:352] missing ipc path /var/run/nvidia-fabricmanager/socket
W0111 10:30:19.085735 149 nvc_info.c:352] missing ipc path /tmp/nvidia-mps
I0111 10:30:19.085739 149 nvc_info.c:854] requesting device information with ''
I0111 10:30:19.093013 149 nvc_info.c:745] listing device /dev/nvidia0 (GPU-15dd6db0-ca52-31f5-3daf-2019882683b0 at 00000000:02:00.0)
I0111 10:30:19.101035 149 nvc_info.c:745] listing device /dev/nvidia1 (GPU-2f4bb339-fc05-e25d-512c-05c7eefd99e1 at 00000000:04:00.0)
I0111 10:30:19.109848 149 nvc_info.c:745] listing device /dev/nvidia2 (GPU-23e7b85e-0792-0721-9d22-ac2a0e9bac2b at 00000000:06:00.0)
I0111 10:30:19.118514 149 nvc_info.c:745] listing device /dev/nvidia3 (GPU-d78a3ff1-47b6-80be-95a5-ffe77853335f at 00000000:08:00.0)
I0111 10:30:19.126998 149 nvc_info.c:745] listing device /dev/nvidia4 (GPU-3dbafd65-2a6e-1445-9569-a70fa746902d at 00000000:0a:00.0)
I0111 10:30:19.135642 149 nvc_info.c:745] listing device /dev/nvidia5 (GPU-d9bc47e7-fd6d-f1dc-ff31-d9675bb73087 at 00000000:0c:00.0)
`

nvidia-container-cli，code positioning analysis is as follows：【src/nvc_mount.c 】

`
int
nvc_driver_mount(struct nvc_context *ctx, const struct nvc_container *cnt, const struct nvc_driver_info *info)
{
const char **mnt, **ptr, **tmp;
size_t nmnt;
int rv = -1;

    if (validate_context(ctx) < 0)
            return (-1);
    if (validate_args(ctx, cnt != NULL && info != NULL) < 0)
            return (-1);

    if (ns_enter(&ctx->err, cnt->mnt_ns, CLONE_NEWNS) < 0)
            return (-1);

    nmnt = 2 + info->nbins + info->nlibs + cnt->nlibs + info->nlibs32 + info->nipcs + info->ndevs + info->nfirmwares;
    mnt = ptr = (const char **)array_new(&ctx->err, nmnt);
    if (mnt == NULL)
            goto fail;

    /* Procfs mount */
    if (ctx->dxcore.initialized)
            log_warn("skipping procfs mount on WSL");
    else if ((*ptr++ = mount_procfs(&ctx->err, ctx->cfg.root, cnt)) == NULL)
            goto fail;

`
After locating, it was found that the problem occurred on ns_enter func. Under normal circumstances during the test, even if the ns_enter interface was called, the file system path inside the virtual machine could be viewed, and then it could be mounted normally. [The mounting path is: /run/kata -Containers/AE1F1999611632C96D7EF8A5D5F51894D377259F06336911d02F67474D0/ROOTFS/Prive/Driver/Nvidia], for example, the abnormal scene used 6 GPU cards and called ns_enter, will directly enter the ROOTFS of the container as the root directory, so mount [/run/kata- containers/ae1f8199611632c96d7e2ef8a5d5f51894d377259f062f6336911d02f67474d0/rootfs/proc/driver/nvidia], the path cannot be found. I don’t understand the reason here. I don’t know how to solve this problem.

[diandian2023]

The nvidia-container-cli process never exited

root@localhost:/# ps -ef | grep con
root 82 2 0 10:30 ? 00:00:00 [ipv6_addrconf]
root 103 2 0 10:30 ? 00:00:00 [ext4-rsv-conver]
nobody 183 123 0 10:30 ? 00:00:05 /usr/bin/nvidia-container-cli --load-kmods --debug=/run/nvidia-container-toolkit.log configure --ldconfig=@/sbin/ldconfig.real --device=all --compute --utility --require=cuda>=12.3 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 brand=tesla,driver>=535,driver<536 brand=unknown,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=geforce,driver>=535,driver<536 brand=geforcertx,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=titan,driver>=535,driver<536 brand=titanrtx,driver>=535,driver<536 --pid=147 /run/kata-containers/ae1f8199611632c96d7e2ef8a5d5f51894d377259f062f6336911d02f67474d0/rootfs

[diandian2023]

The problem has been solved. It was caused by a timeout bug in kata-agent.

[github-actions]

This issue is stale because it has been open 90 days with no activity. This issue will be closed in 30 days unless new comments are made or the stale label is removed. To skip these checks, apply the "lifecycle/frozen" label.