[hodaaaaaaaaaa]

Problem

At start-up, vLLM records a CUDA graph for every batch size it might see (172 sizes for gemma-4-E2B) and records them twice. That takes up to a minute.

Change

The sampler records graphs only for powers of two up to the maximum batch size (9 sizes at 256). A batch runs on the next recorded size up.

Results

From the latest fleet test on GKE (eight GSM8K GRPO jobs launched together: gemma-4-E2B on H100, Qwen3-0.6B on L4, LoRA and full fine-tuning), with all three sampler start-up changes (#358, #359, #360):

• Recording graphs takes 24 s instead of 52 s for the gemma-4-E2B LoRA sampler, and 7 s instead of 19 s for Qwen3-0.6B.
• Sampling time per step shows no consistent change.
• Together, samplers are ready in 36–85 s instead of 91–243 s, and the whole workload takes 678 s instead of 737 s.

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[hodaaaaaaaaaa]

i didn't add an env var for the capture sizes, I wanted to avoid growing the list of knobs when nothing needs it yet. VLLM_ENFORCE_EAGER=1 still turns graphs off completely. If a real need comes up, adding the variable back is a small change.