fix(rollout): reject shared prefill with continuous batching - #8583
Open
modelpath-dev wants to merge 2 commits into
Open
modelpath-dev wants to merge 2 commits into
modelpath-dev wants to merge 2 commits into
Conversation
Add a failing unit test for DeepSpeed deepspeedai#8458 before the validator change. Co-authored-by: Cursor <cursoragent@cursor.com>
Raise ValueError when HybridEngineRolloutConfig.use_shared_prefill is set together with SamplingConfig.continuous_batch_size, instead of selecting the continuous path and silently ignoring shared prefill. Document the incompatibility in the inference-engine notes. Fixes deepspeedai#8458. Co-authored-by: Cursor <cursoragent@cursor.com>
modelpath-dev
requested review from
loadams,
tjruwase and
tohtana
as code owners
September 18, 2026 07:51
delock
self-requested a review
September 20, 2026 07:21
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
HybridEngineRollout.generateselects the continuous path wheneverSamplingConfig.continuous_batch_sizeis set, before any shared-prefill logic runs. WithHybridEngineRolloutConfig.use_shared_prefill=True, that flag was therefore ignored with no error. This PR rejects the combination explicitly, matching the approach already taken for CUDA graph capture on the continuous path.Fixes #8458.
Scope
HybridEngineRollout._validate_continuous_inputs: raiseValueErrorwhenuse_shared_prefillis enabled.docs/code-docs/source/inference-engine.rst: document that shared prefill and continuous batching cannot be combined.Out of scope: implementing shared-prefill semantics inside continuous batching.
Tradeoffs
Rejection over a compatible shared-prefill continuous mode. Continuous batching already requires
n_samples_per_prompt=1, so shared prefill has no clear benefit there today.Blast Radius
Only the experimental continuous path changes, and only when shared prefill is also requested. Default
generate()and shared-prefill-only configs are unchanged.Verification
Before the fix,
_validate_continuous_inputsaccepteduse_shared_prefill=Truewithcontinuous_batch_size=1(silent ignore). After the fix it raisesValueError: continuous batching does not support shared prompt prefill.