[hodaaaaaaaaaa]

Problem

Before a sampler can serve, vLLM compiles the model, which takes about a minute (gemma-4-E2B ~80 s, Qwen3-0.6B ~55 s). vLLM can save the result and reuse it, but it saves it inside the container, so it is lost when the pod ends and every new sampler compiles again.

Change

Sampler pods on Kubernetes keep vLLM's cache (VLLM_CACHE_ROOT) on the shared volume, at $OPEN_RL_TMP_DIR/vllm-cache. The first sampler for a given model, GPU type and engine setup compiles and saves the result; later ones load it.

Results

From the latest fleet test on GKE (eight GSM8K GRPO jobs launched together: gemma-4-E2B on H100, Qwen3-0.6B on L4, LoRA and full fine-tuning), with all three sampler start-up changes (#358, #359, #360):

• With the cache filled, compiling takes 2–6 s instead of 42–83 s.
• Samplers are ready in 36–85 s instead of 91–243 s, every job reaches its first step 57–153 s sooner, and the whole workload takes 678 s instead of 737 s.
• With an empty cache, the first sampler of each model, GPU type and setup compiles more slowly (gemma-4-E2B LoRA: 107 s instead of 83 s) because it also writes the cache, and the workload takes 733 s.

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[droot]

/cc @ShubyM also had this one of his sampler's PR.