Skip to content

Fix static rollout graph decoding regressions - #9127

Open
jinyouzhi wants to merge 4 commits into
deepspeedai:masterfrom
jinyouzhi:fix-9126
Open

jinyouzhi wants to merge 4 commits into
deepspeedai:masterfrom
jinyouzhi:fix-9126

Conversation

@jinyouzhi

@jinyouzhi jinyouzhi commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Fixes #9126.

  • Size the static rollout KV cache from the actual local prefill tensors instead of guessed global/TP head counts.
  • Pass an additive 4-D mask to eager/SDPA attention while retaining a separate boolean mask for the captured step kernel.
  • Apply repetition penalties on the static graph path, reset capture-time mask mutations before replay, and restrict the full-step kernel to the single-row, unpenalized greedy fast path.

Root cause

The segment-KI rollout change eagerly initialized DeepSpeedStaticCache from global configuration head counts. That allocation can disagree with AutoTP's local KV heads, causing either a copy shape failure or silent wrong-token generation. The same change also passed a boolean 4-D mask directly to attention implementations that expect additive masks, while graph/Python token selection bypassed the repetition penalty. Together these account for the nightly static-graph failures in tests/unit/v1/test_hybrid_engine_rollout_v1.py.

The batched graph step did not need the full-step argmax kernel because Python already selects each row's token. The kernel is now explicitly limited to batch size one; the b=1 EOS check remains amortized over 16 replays because at most 15 trailing steps are padded away, avoiding a synchronization on every replay. Batched decoding continues checking after every step because its dead work is multiplied by the number of active rows.

Duplicate-work check

#9077 is primarily a static graph-reuse performance PR, not the dedicated #9126 fix, but its rebase onto #8602 independently absorbed the same local-KV, additive-mask, and repetition-penalty corrections while preserving that performance work. #9127 remains the isolated hotfix for the nightly regression. If #9127 is merged first, #9077 should rebase onto it and drop the duplicated correctness hunks; its remaining scope is graph reuse and lifecycle. #9079 is stacked on #9077 and separately handles whole-step graph capture, including replacement of the Python token-history loop.

Validation

Hardware: 1x Intel Arc Pro B70 (16 devices visible), Level Zero XPU; driver version is not exposed by the environment. Software: torch 2.14.0.dev20260804+xpu, Transformers 5.17.0.

python -m pytest -q -p no:cacheprovider -o addopts= \
  tests/unit/utils/test_static_cache.py \
  'tests/unit/v1/test_hybrid_engine_rollout_v1.py::test_static_graph_attention_mask_uses_additive_values' \
  'tests/unit/v1/test_hybrid_engine_rollout_v1.py::test_static_graph_uses_prefill_kv_shape' \
  'tests/unit/v1/test_hybrid_engine_rollout_v1.py::test_static_graph_full_step_kernel_preconditions' \
  'tests/unit/v1/test_hybrid_engine_rollout_v1.py::test_static_graph_matches_native_llama_generation[eager]'
# 26 passed in the combined focused set

pre-commit run --files \
  deepspeed/runtime/rollout/hybrid_engine_rollout.py \
  tests/unit/v1/test_hybrid_engine_rollout_v1.py
# all hooks passed

The broad two-file run produced 34 passes and 17 skips before the added direct regressions; its remaining failures were unsupported Level Zero SDPA/distributed operations (UR_RESULT_ERROR_UNSUPPORTED_FEATURE), not assertion regressions in the changed path.

The new direct regressions assert that visible positions are additive zero and masked positions are finfo.min, and that the full-step kernel is not mounted for repetition penalties or batched decode. Existing end-to-end tests compare generated tokens against Hugging Face and cover the penalty result on CUDA CI.

Compatibility coverage

  • Validated: static graph replay on Intel XPU; AutoTP-like local KV head counts 4/2/1; eager Llama attention; greedy decoding; additive mask values; full-step-kernel preconditions; cache replay/position/compaction contracts.
  • Not validated locally: NVIDIA CUDA, SDPA on CUDA, ROCm, sampled/repetition-penalty end-to-end rollout on a CUDA device, Hybrid/linear-attention models, and multi-rank AutoTP. These require hardware/backend coverage outside this environment.
  • ZeRO stage/offload are not affected. No public API, config, checkpoint, convergence, throughput, or peak-memory behavior is changed.

The per-step torch.cat used by repetition penalty remains O(n²), matching the existing continuous rollout implementation. Optimizing only this static path is small (roughly 30–60 minutes plus a focused test), but unifying it with variable-length continuous rollout would take roughly 2–4 hours and require broader compaction/padding/CUDA validation. It is intentionally deferred from this correctness fix and can be handled as a small follow-up performance PR, primarily benefiting repetition_penalty != 1; the default penalty path is unaffected.

AI assistance was used for investigation, implementation, and test preparation; the submitting human reviewed the final diff and is responsible for the DCO sign-off.

Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
@jinyouzhi
jinyouzhi marked this pull request as ready for review November 20, 2026 06:01
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-20T06:08:42.369043Z c5717c9 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@delock
delock self-requested a review November 20, 2026 06:28
@delock
delock enabled auto-merge November 20, 2026 10:04
auto-merge was automatically disabled November 20, 2026 11:18

Head branch was pushed to by a user without write access

@jinyouzhi

Copy link
Copy Markdown
Contributor Author

CI failure fixed in c2a4c5a PTAL Thanks @delock

@jinyouzhi

Copy link
Copy Markdown
Contributor Author

4×L20 validation on the latest SHA (c2a4c5a)

Hardware/software: 4× NVIDIA L20, driver 580.173.02; PyTorch 2.15.0+cu132; CUDA toolkit 13.0 (minor-version compatibility warning accepted); Transformers 5.22.1. Test checkout was clean and matched the PR head.

CUDA_VISIBLE_DEVICES=0,1,2,3 DS_ACCELERATOR=cuda OMP_NUM_THREADS=1 HF_HUB_OFFLINE=1 \
  python -m pytest -q -p no:cacheprovider -o addopts= \
  tests/unit/v1/test_hybrid_engine_rollout_v1.py \
  tests/unit/runtime/rollout/test_hybrid_engine_rollout.py \
  tests/unit/utils/test_static_cache.py \
  -k 'not TestAutoTPContinuousRollout and not TestGlobalWorkRemainingSubgroups and not TestZero3SubgroupGeneration and not TestZero3SynchronizedEarlyStop'
# 209 passed, 2 skipped, 5 deselected

The #9127 static-mask, local-KV-shape, sampling, repetition-penalty, and native Llama eager/SDPA paths all passed.

Native segKI file:

python -m pytest -q -p no:cacheprovider -o addopts= tests/unit/module_inject/test_segment_ki_kernels.py
# 21 passed, 4 skipped, 1 failed

The one segKI failure is the existing Transformers 5.22.1 compatibility issue Qwen3RMSNorm no longer exposes .eps; the other native-kernel tests pass. This is also called out in the existing #9077/#9057 environment results.

The five deselected DistributedTest cases could not be completed on this shared host: individual world-size 2/4 runs reached the 600-second framework watchdog and teardown failed with CUDA_ERROR_DEVICE_UNAVAILABLE. No assertion from #9127 failed before the infrastructure teardown. The non-distributed rollout/cache matrix completed successfully.

@jinyouzhi

Copy link
Copy Markdown
Contributor Author

CI-matched L20 rerun update

Reran the non-distributed rollout/cache matrix in an isolated Python 3.12 environment matching the workflow defaults more closely: PyTorch 2.10.0+cu128, CUDA 12.8 runtime, and Transformers main at 4bb61661e15e65c135839acfb8ed0781605aabe2 (5.24.0.dev0). GPU: 4× NVIDIA L20, driver 580.173.02.

python -m pytest -q -p no:cacheprovider -o addopts= \
  tests/unit/v1/test_hybrid_engine_rollout_v1.py \
  tests/unit/runtime/rollout/test_hybrid_engine_rollout.py \
  tests/unit/utils/test_static_cache.py \
  -k 'not TestAutoTPContinuousRollout and not TestGlobalWorkRemainingSubgroups and not TestZero3SubgroupGeneration and not TestZero3SynchronizedEarlyStop'
# 208 passed, 3 skipped, 5 deselected

This supersedes the older environment-specific note about Transformers 5.22.1. With current transformers/main, the segKI GLU-cache integration test reaches another upstream API compatibility issue: Qwen3Attention.forward now expects cache, while the segKI wrapper supplies the old past_key_values position plus cache from the deprecation wrapper, producing multiple values for argument 'cache'. #9127 does not modify segment_ki.py, and all tests covering its rollout changes pass. The local host lacks a CUDA 12.8 nvcc (only 11.5/13.0), so the complete native segKI CUDA matrix could not be rerun under the exact CI toolkit pair.

Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

nightly regression: modal-torch-latest full suite failed

2 participants