Skip to content

Support for starting sandboxes with cuda-checkpoint - #13987

Open
luiscape wants to merge 1 commit into
google:masterfrom
luiscape:luis/start-job-for-cuda-checkpoint
Open

Support for starting sandboxes with cuda-checkpoint#13987
luiscape wants to merge 1 commit into
google:masterfrom
luiscape:luis/start-job-for-cuda-checkpoint

Conversation

@luiscape

@luiscape luiscape commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

#13400 introduces sequential checkpoints but notes that the option is the most useful when processes are started with cuda-checkpoint --launch-job. That is required--when using NVIDIA driver 610+--in order to support coherent checkpoint/restore of CUDA programs that use IPC (see docs).

I think that this is the most useful checkpointing feature of driver 610+. IPC-enabled workloads are arguably the most popular when using multiple GPUs. These include popular inference engines like SGLang and vLLM. I am adding this to provide first-class support in gvisor.

I'm adding two new CLI flags:

  • nvproxy-cuda-checkpoint-path: indicates the path for cuda-checkpoint when calling runsc create
  • nvproxy-cuda-checkpoint-jobs: prepend all container commands with cuda-checkpoint --launch-job.

These can be combined with --cuda-checkpoint-sequential, providing full support for IPC-enabled workloads.

@github-actions
github-actions Bot requested review from fvoznika and trantoji August 4, 2026 17:56
@ayushr2

ayushr2 commented Aug 4, 2026

Copy link
Copy Markdown
Collaborator

Thanks for the investigation Luis!!!

Maybe we should simplify this:

  • Call the flag --cuda-checkpoint-path, it complements the runsc checkpoint flag already.
  • No need for nvproxy-cuda-checkpoint-jobs IMO. If --cuda-checkpoint-path is specified and nvproxy is enabled AND driver version is 610+, we should just prepend the launch job flag. If IPC is not being used, I think this flag is harmless.
  • This should be a flag on runsc create instead. Since it refers to a path in the container filesystem. gVisor supports multi-container. Each container in the sandbox has a different filesystem. So some sidecar container's filesystem may not have cuda-checkpoint at all and it may be doing non-GPU work.

google#13400 introduces sequential
checkpoints but notes that the option is the most useful when processes
are started with `cuda-checkpoint --launch-job`. That is required--when
using NVIDIA driver 610+--in order to support coherent
checkpoint/restore of CUDA programs that use IPC (see
[docs](https://github.com/NVIDIA/cuda-checkpoint#610-features)).

I think that this is the most useful checkpointing feature of driver
610+. IPC-enabled workloads are arguably the most popular when using
multiple GPUs. These include popular inference engines like SGLang and
vLLM. I am adding this to provide first-class support in gvisor.

I'm adding two new CLI flags:

- `nvproxy-cuda-checkpoint-path`: indicates the path for
  `cuda-checkpoint` when calling `runsc create`
- `nvproxy-cuda-checkpoint-jobs`: prepend all container commands with
  `cuda-checkpoint --launch-job`.

These can be combined with `--cuda-checkpoint-sequential`, providing
full support for IPC-enabled workloads.
@luiscape
luiscape force-pushed the luis/start-job-for-cuda-checkpoint branch from b723e77 to 9c3de61 Compare August 4, 2026 19:20
@EtiennePerot
EtiennePerot requested review from parth-opensrc and removed request for fvoznika August 4, 2026 21:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants