Repository navigation
feat: sampling pool dispatcher - #354
Merged
Merged
Conversation
ShubyM
marked this pull request as ready for review
October 21, 2026 21:24
ShubyM
force-pushed
the
feat/sampling-pool-dispatcher
branch
from
October 21, 2026 21:30
84e5819 to
fe9f5d1
Compare
droot
approved these changes
Oct 22, 2026
Adds sampling_dispatcher with Dispatcher and HttpSampler. The dispatcher
is the pool's only reader. It reads the stream incrementally, holds each
pending record until its result is saved, keeps up to `concurrency` sends
in flight, and fails a record no replica answers before the deadline.
HttpSampler sends through the pool's router, finds its replicas by pod
label, and drops an answer from a replica that no longer reports the
record's revision. In this commit replicas serve only
the weights they start with, so a record for any other revision fails at
its deadline.
SamplingStore gains pending(after=...) and wait_for_new for the
incremental reads, and returns prompts as array('I') since the
dispatcher holds every pending one in memory.
ShubyM
force-pushed
the
feat/sampling-pool-dispatcher
branch
from
October 23, 2026 18:10
fe9f5d1 to
89e5102
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR introduces the dispatcher for sampling pools, building on #352. Each pool keeps its pending requests in a Redis stream and the dispatcher is that stream's one reader. It reads without removing entries, hands each request to a sampler, and saves the first result while deleting the entry in the same transaction, so a sampler that stops never loses a request.
This adds the loop and
HttpSampler, which sends through the pool's router to itsvllm servereplicas and drops any answer that came from the wrong weights. Weight loading, time slicing, and the switch that puts it to use come in the next PRs.