[ShubyM]

Stacked on #337; review the last two commits.

Sampling now goes through pools, and the in-process queue sampler (server.vllm_sampler) is deleted. SAMPLING_BACKEND is torch or vllm, and vllm means pools. There is no flag to keep both paths alive; the TPU branch will catch up with main.

What a pool is

• vllm serve replicas, one Workload each, plus one CPU pod running the dispatcher, Envoy and the llm-d EPP.
• A replica is Ready only once it holds the pool's weight version, so the router never sends to stale weights.
• Each LoRA save is frozen into sampler_lora/<sha256(ref)>, and vLLM loads it by name.
• Gone with the queue sampler: the Redis sampling list, the sampler_ready handshake and the shutdown sentinel.

The second commit holds the review fixes

• Sampler status. A sampler reads ready only once its pod is Ready and its pool's dispatcher has a pod.
• Clear errors and retries. The server refuses to start without REDIS_URL, accepting a sample retries NFS lookups for up to 2 s, and a fresh install no longer hits the RBAC subject bug.
• One vLLM compile cache per process. Two pods could otherwise corrupt the shared one.
• One tmp dir. The probe, the dispatcher and the worker pods all fall back to /tmp/open-rl, locally and on the cluster.
• Local mode. Its dispatcher restarts along with its vLLM, and the LoRA snapshot folder exists before a LoRA pool starts.
• vLLM's 0.90 memory fraction is back. Pending prompts are held compactly in the dispatcher.

Known limitations. Every pool sampler has its own GPU, so an FFT job takes two until #342 lands. Under sparse-delta sync, an FFT replica that restarts or falls behind can't catch up yet; that needs periodic full checkpoints in a follow-up.

Tested with unit tests (303). LoRA and FFT runs on GKE used this pools code before the flag was removed; the removal touches only routing and the deleted queue sampler.

[droot]

Reviewed the last commit. This is a big switch and the mechanism holds together: readiness gated on the weight version, EPP only picks ready pods, dispatcher double checks the version on the answer. 22 runs on spot is good evidence. Ran the unit suite locally against a redis, 147 pass. It also closes three of my questions on #336 (the @hex suffix, freeze_adapter wiring, pool names), thanks.

Two things I would want settled before this merges, then smaller items.

1. FFT jobs no longer share a GPU between trainer and sampler

exclusive = not meta.shares_gpu() or role == "sampler" means every FFT job now needs trainer GPU(s) plus sampler GPU(s) full time. The description says this is "not carried over", which reads as permanent. That changes the GPU count for every FFT job and the single-GPU FFT story we have been demoing. The time-slicer and snapshot agent stay in the tree but nothing uses them on the sampler side anymore.

I am fine if this is the direction, but it should be a deliberate call with a short note on what replaces it (sleep/wake on the pool replica? the trainer taking the sampler's GPU between steps?) and what happens to the fft-timeslice deploy and e2e scenarios. Right now it is a one-line aside in a PR that also deletes the old path.

2. Sparse delta still has no way back, and retention now makes it worse

Same point as on #337: under weight_sync_strategy=delta the trainer only writes deltas, and the dispatcher refuses a delta whose parent the replica does not hold. This PR adds drop_old_checkpoints, which deletes everything but the newest 3 (plus pinned). So a replica that restarts, or falls more than 3 saves behind, has no chain to replay even in principle. A spot preemption, which is exactly what you tested on, will hit this.

I think the fix is on the trainer side: write a full checkpoint every K saves or whenever a replica reports a version outside the kept window, and have the dispatcher load full when the parent is missing. Could be a follow-up, but then delta mode should be marked unsafe for pools until it lands.

3. Sampler status is now "ready" while vLLM is still loading

openrl_status drops the sampler_ready announce, so a pool sampler reports ready as soon as its pod is Running, which is minutes before it can serve. We just added the starting/ready distinction in #328 for exactly this. The readiness probe you added is the right signal; can the Workload status (or a pod read) carry the Ready condition so the model view uses it?

Smaller

• ensure_dispatcher can block up to 180s in to_thread while an old pool's pod terminates, and create_sampling_session waits on it. Fine for now, but worth a log line so a slow session create is explainable.
• VLLM_CACHE_ROOT and TRITON_CACHE_DIR on the shared NFS volume with several replicas compiling at once: does vLLM lock the cache dir, or can two replicas race on the same artifact? I hit this question on my cold-start branch too, so happy to compare notes.
• Local mode without REDIS_URL: the dispatcher dies on os.environ["REDIS_URL"] and accept_pool_sample fails on store.redis. A clear "pools need REDIS_URL" error at startup would save someone an hour.
• The NetworkPolicy blocks everything but the pool pod on 8000. That also blocks any metrics scrape of vLLM. Not a problem today, just noting it.
• Do we have step-time numbers from the 22 runs against the queue sampler on the same model? The Envoy + EPP hop and the per-answer /weight_info call are on the critical path for every request, so a before/after on fft-gsm8k-rl would be good to have in the PR.

[ShubyM]

Closing in favor of new design