Repository navigation
Conversation
ShubyM
force-pushed
the
feat/sampling-pool-fairness
branch
2 times, most recently
from
October 26, 2026 23:27
a08741a to
5a0224e
Compare
ShubyM
marked this pull request as ready for review
October 27, 2026 05:58
Each model's sampling requests go to a pool stream, open_rl:sampling:<pool>, instead of a Redis list that a sampler emptied on read. asample adds a request and returns. The pool's dispatcher saves one result per request, so a sampler that stops loses nothing. A pool that one job owns runs server.vllm_sampler, vLLM in its own process, when it has one replica, for LoRA and FFT alike. It takes time-slicer turns only when it shares its trainer GPU. A pool with two or more replicas gets vllm serve Workloads behind a dispatcher pod with Envoy and the llm-d EPP. Local pools always have one replica. A pool that shared LoRA jobs use is always routed, even at one replica. Each job's ensure then only adds replicas, so the pool grows to the largest sampler_replicas any of its jobs asks for, and a second job never meets an in-process sampler it would have to replace. A sampler on its trainer's GPU is its pool's only replica, so create_model refuses sampler_replicas above 1 for an FFT job that shares its trainer's GPU, and tells the user to set openrl.exclusive=true. A routed pool's dispatcher Deployment and NetworkPolicy carry an openrl.io/owner label. release_owner deletes them by that label as well as through the pool's Workloads, so they do not outlive Workloads that went first. The worker launcher's Role gains list on deployments and networkpolicies for that lookup. The queue sampler and its vLLM patch are deleted, along with the store's sampler queue reads. The in-process sampler keeps sizing vLLM's batch for its GPU with sampler_batch_limits. sample_params and sample_result build the vLLM params and the saved result for both samplers. The in-process sampler raises RevisionUnloadable for a delta that does not apply on its weights, so those requests fail at once there too. It also fails a request vLLM refuses as a client error, such as a prompt past max_model_len or a token out of vocabulary, and a request that fails with EngineGenerateError, instead of sending it again until the deadline. vLLM loads a LoRA adapter inside the engine step, and a missing adapter raises LoRAAdapterNotFoundError there, which kills EngineCore. So the in-process sampler checks for the snapshot's adapter_config.json before it sends a record. On a miss it lists the snapshot directory once, since an NFS client keeps a failed lookup until it does, and if the file is still missing the record stays pending and is sent again. When the engine dies, every request in flight fails with EngineDeadError, and the sampler saves a failed result for each, as main's sampler did. A request that kills the engine would otherwise be served again by each restarted engine until the deadline. A request sent after the engine died never reached it, so it stays pending for the restarted worker. A sampler that shares its GPU with the trainer runs its batches through TimeSlicedDispatcher in vllm_sampler, one time-slicer turn per batch with main's holding_gpu. It loads weights through the same Dispatcher.ready as the routed dispatcher. The shared Dispatcher knows nothing about turns. A sampling session opened on a bare base-model name keeps main's rule, FFT when the deployment enables FFT and LoRA otherwise. Its runtime was the base model's name, which is also the name of the base model's shared LoRA runtime, so an FFT sampler for it took the LoRA pool's owner and read the LoRA pool's stream. TrainingModelMetadata.runtime now names such a runtime base-<model>. Cluster samplers keep Triton's kernel cache at <OPEN_RL_TMP_DIR>/vllm-cache/triton on the shared volume, beside vLLM's compile cache. Local samplers keep Triton's default, which already persists on one machine. The scheduler worker manager refuses to start without OPEN_RL_TMP_DIR, since its default of /tmp/open-rl is not on the shared volume. Each record names the job that sent it as "job", which job_of reads, so later changes can tell a pool's jobs apart.
A shared LoRA pool serves every job on its base model, and the dispatcher sent its records oldest first. One job's big batch then held every slot until it finished, and other jobs waited behind it. Dispatcher.to_send takes one record from each job, read from the record's "job" field, in turn, oldest first within a job, starting with the job served longest ago. A record accepted before this change has no job and groups under its model.
ShubyM
force-pushed
the
feat/sampling-pool-fairness
branch
from
October 27, 2026 19:06
5a0224e to
2aaae41
Compare
droot
approved these changes
Oct 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Sends each pool's requests round-robin by job, stacked on #378 (only the last commit is new). A shared LoRA pool serves every job on its base model, and the dispatcher sent its records oldest first, so one job's big batch held every slot until it finished and other jobs waited behind it. This addresses Hoda's FIFO comment on the design.
The dispatcher takes one record from each job in turn, using the job #378 writes on every record, oldest first within a job and starting with the job served longest ago. The change is only the dispatcher's send order, so it reverts on its own.