Repository navigation
Conversation
3fba9b5 to
7640fb3
Compare
Rebase the static-graph reuse onto the segment-KI decode path from deepspeedai#8602. - Keep the DeepSpeedStaticCache, decode inputs and captured graph across generate() calls with the same decode signature (batch, cache length, device, dtype, parameter layout, training mode, autocast, step-kernel use). AutoTP ranks agree on recapture through an all-reduce. - Without segKI, in-place weight updates are replayed without recapture. With segKI, segki_refresh() bumps a generation in the signature and refresh_weights()/sync_weights() release the graphs, because the injected kernels cache tensors derived from the weights. - Size the decode KV cache from the prefill's local KV tensors again, so AutoTP-sharded and narrowed-head models get the right shape. - Apply the repetition penalty on the graph prefill and Python decode paths again; the b=1 step kernel is used only without a penalty. - Give eager attention an additive mask built inside the graph from the boolean decode mask. Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
The static graph path captured only the model forward for b>1 and for sampling or a repetition penalty. After every replay the host chose the next token, wrote the position and attention mask, and checked for EOS. Capture the token selection with the forward for every batch size: the graph applies the repetition penalty over the device token buffer, picks the token (argmax, or HF warpers plus multinomial), records EOS, and advances the position, mask and output buffer for the next replay. The b=1 greedy decode_step_graph kernel from segment KI is still used where it applies. The host only replays and checks for EOS every 16 steps, then trims the output to the step where the last row produced EOS, as generate() does. Sampling, repetition penalty and EOS ids join the graph signature, and the generator state is restored after warmup so the graph draws the same random numbers as generate(). Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
4×L20 validation (
|
CI-matched L20 rerun updateReran with PyTorch 2.10.0+cu128 and current Transformers The whole-step graph, early-EOS, sampling, and in-graph repetition-penalty tests pass. As with #9077, current |
Summary
On the static graph path, #8602 added a
decode_step_graphkernel that picks the token inside the graph, but only for b=1 greedy decoding without a repetition penalty. Every other case (b>1, sampling, repetition penalty, non-bf16 logits) captures only the forward: after every replay the host picks the next token, writes the position and mask, and checks for EOS.This PR captures the whole decode step for every batch size:
decode_step_graphmultinomial) + EOS flag + position / mask / output-buffer update, in PyTorch opsfinishedflagmax_new_tokensgenerate()does_apply_repetition_penaltybit for bit.generate().inference-engine.rstupdated.Tests
test_static_graph_stops_once_every_row_has_produced_eos: every row hits EOS on its third token; graph output equals the eager rollout, and the graph is replayed fewer thanmax_new_tokens - 1times.test_static_graph_capture_receives_a_prebuilt_4d_masknow replays the captured step, since the step itself advances the mask.model.generate()cover the in-graph selection.4× NVIDIA L20, torch 2.14.1+cu130, Transformers 5.17.0, same files as #9077: 221 passed, 3 skipped, 3 failed (the same 3 environment failures as on master).
Qwen3.5-0.8B, segKI + graph: tokens identical to master at b=1 and b=4.
Performance
Qwen3-0.6B, no segKI, L20, prompt 128, 256 new tokens, EOS disabled; decode tok/s:
Throughput is unchanged where decode is GPU-bound. The gain is that rollouts ending in EOS stop without a device sync on every token, for every batch size and sampling setup.
This PR was prepared with AI assistance (GitHub Copilot); I reviewed and tested the changes.