Skip to content

Fix HybridEngineRollout KV caches under autocast when keys and values differ in dtype - #9124

Open
ArchanaChetan07 wants to merge 1 commit into
deepspeedai:masterfrom
ArchanaChetan07:fix/9102-rollout-autocast-cache-dtype
Open

ArchanaChetan07 wants to merge 1 commit into
deepspeedai:masterfrom
ArchanaChetan07:fix/9102-rollout-autocast-cache-dtype

Conversation

@ArchanaChetan07

@ArchanaChetan07 ArchanaChetan07 commented Nov 19, 2026 •

Copy link
Copy Markdown

Fixes #9102.

Root cause

The issue attributes the failure to caches being created with the parameter dtype. Tracing it on master, the cause is slightly different: under torch.autocast(dtype=bf16) an fp32 Llama model produces keys and values in different dtypes:

  • k_proj/v_proj run under autocast and return bf16.
  • apply_rotary_pos_emb multiplies the bf16 keys by the fp32 cos/sin (the rotary module runs with autocast disabled), so the keys are promoted back to fp32. The values stay bf16.

Caches that take their dtype from the first keys (fp32) then reject the bf16 values:

  • DeepSpeedStaticLayer.update (continuous-batching decode, with or without graph capture): Index put requires the source and destination dtypes match, got Float ... BFloat16.
  • HF StaticCache.update, used for the prefill of the static graph path (_generate_graph) and of the continuous paths (_continuous_prefill_cache, for models that set _supports_cache_class): index_copy_(): self and source expected to have the same dtype ... Float ... BFloat16. HF StaticCache cannot hold fp32 keys and bf16 values at once, whatever dtype it is allocated in.

Allocating the caches in the autocast dtype would not fix this, because the keys really are fp32.

Change (+27 / -3 in deepspeed/)

  • DeepSpeedStaticLayer.update: cast the incoming key/value states to the cache dtype before writing. .to() returns the same tensor when the dtypes already match, so graph capture and the non-autocast paths are unchanged.
  • New HybridEngineRollout._create_prefill_cache, used by both prefill sites (_generate_graph, _continuous_prefill_cache):
    • Models whose layers are all full attention (no sliding_window, every layer_types entry full_attention) prefill into a lazily sized DeepSpeedStaticCache, which is sized from the prefill keys so AutoTP-local KV heads still work. With no write position set, it writes the prompt at [0, prompt_len) and gives a single prefill forward the same mask sizes and offsets as a fresh HF StaticCache.
    • All other models keep HF StaticCache, exactly as on master. DeepSpeedStaticCache models every layer as full attention, so a sliding-window model prefilled into it would silently attend outside the window. Hybrid models also need the HF cache because their GDN slots are bound to its layers (feat(rollout): segment-based kernel injection for hybrid-engine greedy decode #8602).

Model families by path (default configs, transformers 5.12 and 5.23):

  • Fixed by this PR: Llama, Qwen2/Qwen3, Mixtral, Phi-3, GPT-NeoX.
  • Unchanged (still fail under fp32-param + bf16 autocast on the cache-prefill paths): Mistral and Gemma 2/3, which default to a sliding window, and Qwen3-Next/Qwen3.5, which have linear-attention layers.

Behaviour without autocast

To check this doesn't change behaviour anywhere else, I ran the same 30 rollout scenarios on master and on this branch and compared the generated tokens (or the raised error):

  • Models: Llama, sliding-window Mistral, sliding-window Qwen2.
  • Attention: eager and sdpa.
  • Paths: static graph, continuous graph, continuous eager, and the continuous paths with _supports_cache_class.

All 30 are identical. The sliding-window runs use prompts longer than the window, and they still raise the same error they raise on master.

Tradeoff

The cast can also narrow. When the decode cache is allocated eagerly in the parameter dtype and that dtype is narrower than the incoming states (e.g. fp16 parameters under bf16 autocast), keys and values are now rounded to that dtype where master raised an error. That only happens under autocast and matches the model's own parameter precision; I can restrict the cast to widening and raise otherwise if you prefer.

Relation to other PRs

Tests

Every new test fails on master. Reverting either half of the fix makes some of them fail:

Test master cast reverted prefill routing reverted this PR
test_rollout_under_autocast_matches_hf_generate: 6 cases = {static graph, continuous graph, continuous eager} × {default, _supports_cache_class=True}. Greedy rollout of an fp32 tiny Llama under bf16 autocast, token-for-token vs HF generate() under the same autocast 6 fail 6 fail 4 fail 6 pass
test_static_layer_accepts_values_in_a_different_dtype_than_keys: fp32 keys + bf16 values through both write paths (prefill and per-row decode positions) 2 fail 2 fail pass 2 pass
test_prefill_cache_keeps_hf_static_cache_unless_full_attention_only: routing for full-attention, sliding-window and linear-attention configs 5 fail pass 5 fail 5 pass

Suites:

Setup Command Result
1x NVIDIA T1000 8GB (sm_75), driver 596.51, torch 2.6.0+cu124, transformers 5.12.1, Windows 11, Python 3.11 pytest unit/v1/test_hybrid_engine_rollout_v1.py unit/utils/test_static_cache.py unit/runtime/rollout 240 passed, 6 failed, 10 skipped
CPU (WSL2 Ubuntu), torch 2.15.0+cpu, transformers 5.23.0, Python 3.12, pytest 8.3.5, DS_ACCELERATOR=cpu pytest --forked -n 4 unit/runtime/rollout unit/utils/test_static_cache.py unit/v1/test_hybrid_engine_rollout_v1.py 209 passed, 47 skipped

The 6 GPU failures fail on master on the same machine with byte-identical assertion output:

  • test_static_graph_uses_prefill_kv_shape[1,2,4]
  • test_static_graph_matches_native_llama_generation[eager]
  • test_decode_repetition_penalty_matches_hf_generate[0.0-True-None] and [0.7-True-None]

They look specific to this setup (sm_75 with bf16 emulation, torch 2.6, transformers 5.12).

pre-commit run --files <changed files> passes.

Not validated: bf16-native GPUs (sm_80+), fp16 autocast, hybrid (GDN) models, ZeRO-3 / AutoTP rollout under autocast, and multi-GPU. Without a GPU, upstream CI only runs the continuous-eager cases, the static-cache tests and the routing test; the graph-capture cases need a CUDA run.

AI assistance

AI assistance (Claude Code) was used to reproduce the bug, trace the root cause, draft the change and tests, and run the commands above. I reviewed every changed line and ran the tests myself.

🤖 Generated with Claude Code

@gpu-test-labeler gpu-test-labeler Bot added the skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. label Nov 19, 2026
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-19T04:01:32.007090Z 9031d60 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@jinyouzhi

jinyouzhi commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Thanks for the fix and the clear reproduction test! Reviewed against master @ b5e0ba67f.

The fix looks correct. .to(self.dtype) after lazy init is the right root-cause fix for #9102: under torch.autocast(bf16) with an fp32 model, RoPE promotes keys back to fp32 while values stay bf16, so the lazily-initialized layer (dtype taken from the first keys) rejects the value writes. Since .to() is a no-op when dtypes already match, graph capture/replay and non-autocast paths are unaffected. Swapping the _generate_graph prefill cache to a lazy DeepSpeedStaticCache also matches the existing _generate_continuous pattern — the "write position ignored → writes land at [0, kv_length)" semantics are fine for the single full-prompt prefill used there, and the comment explaining why HF StaticCache is not used is helpful.

Blocker: needs a rebase. The PR currently conflicts with master (mergeable: CONFLICTING); the drift comes from a single commit, 97aa3c10d (#8602), which touched both source files (+521/−99). I verified that master still has the bug (lazy_initialization still derives self.dtype from key_states and update() has no cast), so this PR is still needed — please rebase onto master and re-run CI.

Non-blocking suggestions:

  1. Continuous-batch staging cache. _generate_continuous / _generate_continuous_graph still stage prefill through _create_static_cache(StaticCache, ..., dtype=model_dtype) in _continuous_prefill_cache. If HF StaticCache.update is dtype-strict, that is the same fp32-cache-gets-bf16-values hazard. The new (False, 2) parametrization should exercise it — worth confirming it actually reaches that branch (models with _supports_cache_class set).
  2. Static-cache-level unit test. A cheap addition to tests/unit/utils/test_static_cache.py: lazy-init a DeepSpeedStaticCache, then update() with fp32 keys + bf16 values, assert no error and values promoted to self.dtype. Existing tests always pass explicit dtypes + set_write_position, so this path is otherwise untested.
  3. GPU coverage. With skip-gpu-test, upstream CI only ran the (False, 2) parametrization on CPU (graph-capture cases skip without CUDA). Consider removing the label or linking a local CUDA run so the autocast + graph-capture path gets CI coverage.
  4. Heads-up on Reuse the static rollout decode graph across calls #9077 — it also touches ~3 lines in _generate_graph, so the rebase may have overlapping hunks.

Thanks!

Under torch.autocast an fp32 Llama-style model produces fp32 keys (the
rotary embedding multiplies the bf16 key projection by fp32 cos/sin) but
bf16 values. DeepSpeedStaticLayer sizes its buffers from the first keys it
sees and then rejected the bf16 values, and HF StaticCache, used for the
static graph and continuous prefills, rejects them the same way.

Cast incoming key/value states to the cache dtype in
DeepSpeedStaticLayer.update, and route both prefills through
_create_prefill_cache: models whose layers are all full attention prefill
into a lazily sized DeepSpeedStaticCache; sliding-window and
linear-attention models keep HF StaticCache, since DeepSpeedStaticCache
models every layer as full attention and hybrid GDN slots bind to the HF
cache.

Fixes deepspeedai#9102

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: ArchanaChetan07 <apatil@sandiego.edu>
@ArchanaChetan07
ArchanaChetan07 force-pushed the fix/9102-rollout-autocast-cache-dtype branch from 9031d60 to 39e0fd5 Compare November 19, 2026 21:31
@gpu-test-labeler gpu-test-labeler Bot added skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. and removed skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. labels Nov 19, 2026
@ArchanaChetan07

Copy link
Copy Markdown
Author

Thanks for the careful review! I rebased onto master (40a9f40) and addressed the points. I've updated the description with the full details.

Rebase. After #8602, hybrid models bind their GDN slots to the HF StaticCache prefill by reference, so an unconditional switch to DeepSpeedStaticCache would break them. While checking that, I also found that the switch would silently break sliding-window models. DeepSpeedStaticCache treats every layer as full attention, so on master Mistral or Qwen2 with a prompt longer than the window raises, but with the switch it produced wrong tokens. Both prefill sites now go through one helper, _create_prefill_cache. It uses a lazily sized DeepSpeedStaticCache only when every layer is full attention (no sliding_window, all layer_types == "full_attention") and otherwise keeps HF StaticCache exactly as on master. Without autocast, master and this branch produce identical tokens (or the identical error) in 30 scenarios: Llama, sliding-window Mistral and Qwen2; eager and sdpa; static graph, continuous graph, continuous eager, and continuous with _supports_cache_class.

1. Continuous-batch staging cache. Good catch: the (False, 2) case never reached it, because Llama has no _supports_cache_class. With it forced on, both continuous paths failed in HF StaticCache.update with the same index_copy_ error. _continuous_prefill_cache now uses the same helper. The autocast test is parametrized over _supports_cache_class in {unset, True}, and with the routing change reverted, 4 of its 6 cases fail.

2. Static-cache unit test. Added test_static_layer_accepts_values_in_a_different_dtype_than_keys, covering both write paths (prefill, and per-row decode positions). There's also a small routing test for full-attention, sliding-window and linear-attention configs.

3. GPU coverage. I can't add labels here. If a maintainer could add the GPU label or trigger the CUDA run, the graph-capture cases would get upstream coverage. Locally, on a T1000 (sm_75, torch 2.6.0+cu124, transformers 5.12.1), the rollout and static-cache suites give 240 passed, 6 failed. The same 6 fail on master on this machine with byte-identical assertion output (test_static_graph_uses_prefill_kv_shape[1,2,4], test_static_graph_matches_native_llama_generation[eager], test_decode_repetition_penalty_matches_hf_generate[*-True-None]), so they aren't caused by this change. CPU with --forked -n 4: 209 passed. pre-commit is clean.

4. #9077. Noted. In hybrid_engine_rollout.py, this PR now changes only the two _create_static_cache(...) call lines plus the new helper, so a rebase either way should be trivial.

One tradeoff I flagged in the description: the cast can also narrow (e.g. fp16 parameters under bf16 autocast, where the decode cache is allocated in fp16). Master raised there; this rounds to the parameter dtype. Happy to restrict it to widening if you'd prefer.

@gpu-test-labeler gpu-test-labeler Bot added skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. and removed skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. labels Nov 19, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] HybridEngineRollout fails under autocast when the autocast dtype differs from the model dtype

2 participants