Repository navigation
Conversation
Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Head branch was pushed to by a user without write access
4×L20 validation on the latest SHA (
|
CI-matched L20 rerun updateReran the non-distributed rollout/cache matrix in an isolated Python 3.12 environment matching the workflow defaults more closely: PyTorch 2.10.0+cu128, CUDA 12.8 runtime, and Transformers python -m pytest -q -p no:cacheprovider -o addopts= \
tests/unit/v1/test_hybrid_engine_rollout_v1.py \
tests/unit/runtime/rollout/test_hybrid_engine_rollout.py \
tests/unit/utils/test_static_cache.py \
-k 'not TestAutoTPContinuousRollout and not TestGlobalWorkRemainingSubgroups and not TestZero3SubgroupGeneration and not TestZero3SynchronizedEarlyStop'
# 208 passed, 3 skipped, 5 deselectedThis supersedes the older environment-specific note about Transformers 5.22.1. With current |
Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
Summary
Fixes #9126.
Root cause
The segment-KI rollout change eagerly initialized
DeepSpeedStaticCachefrom global configuration head counts. That allocation can disagree with AutoTP's local KV heads, causing either a copy shape failure or silent wrong-token generation. The same change also passed a boolean 4-D mask directly to attention implementations that expect additive masks, while graph/Python token selection bypassed the repetition penalty. Together these account for the nightly static-graph failures intests/unit/v1/test_hybrid_engine_rollout_v1.py.The batched graph step did not need the full-step argmax kernel because Python already selects each row's token. The kernel is now explicitly limited to batch size one; the b=1 EOS check remains amortized over 16 replays because at most 15 trailing steps are padded away, avoiding a synchronization on every replay. Batched decoding continues checking after every step because its dead work is multiplied by the number of active rows.
Duplicate-work check
#9077 is primarily a static graph-reuse performance PR, not the dedicated #9126 fix, but its rebase onto #8602 independently absorbed the same local-KV, additive-mask, and repetition-penalty corrections while preserving that performance work. #9127 remains the isolated hotfix for the nightly regression. If #9127 is merged first, #9077 should rebase onto it and drop the duplicated correctness hunks; its remaining scope is graph reuse and lifecycle. #9079 is stacked on #9077 and separately handles whole-step graph capture, including replacement of the Python token-history loop.
Validation
Hardware: 1x Intel Arc Pro B70 (16 devices visible), Level Zero XPU; driver version is not exposed by the environment. Software: torch
2.14.0.dev20260804+xpu, Transformers5.17.0.The broad two-file run produced 34 passes and 17 skips before the added direct regressions; its remaining failures were unsupported Level Zero SDPA/distributed operations (
UR_RESULT_ERROR_UNSUPPORTED_FEATURE), not assertion regressions in the changed path.The new direct regressions assert that visible positions are additive zero and masked positions are
finfo.min, and that the full-step kernel is not mounted for repetition penalties or batched decode. Existing end-to-end tests compare generated tokens against Hugging Face and cover the penalty result on CUDA CI.Compatibility coverage
The per-step
torch.catused by repetition penalty remains O(n²), matching the existing continuous rollout implementation. Optimizing only this static path is small (roughly 30–60 minutes plus a focused test), but unifying it with variable-length continuous rollout would take roughly 2–4 hours and require broader compaction/padding/CUDA validation. It is intentionally deferred from this correctness fix and can be handled as a small follow-up performance PR, primarily benefitingrepetition_penalty != 1; the default penalty path is unaffected.AI assistance was used for investigation, implementation, and test preparation; the submitting human reviewed the final diff and is responsible for the DCO sign-off.