Skip to content

[fix] preserve indexed Qwen3-MoE expert views for Gemini - #6575

Draft
refinedheart wants to merge 1 commit into
hpcaitech:mainfrom
refinedheart:fix/issue6564-gemini-view-pr
Draft

refinedheart wants to merge 1 commit into
hpcaitech:mainfrom
refinedheart:fix/issue6564-gemini-view-pr

Conversation

@refinedheart

Copy link
Copy Markdown
Collaborator

Summary

Fix the storage lifetime issue in native Transformers v5 Qwen3-MoE expert indexing under Gemini.

When a 3D expert parameter is indexed, the resulting view can retain the source Gemini chunk storage. Gemini may release that storage after forward, while autograd still reads the view during backward. This can raise setStorage ... storage size 0 for parameters such as gate_up_proj.

This change clones indexed results for 3D ColoParameter inputs. The clone keeps the autograd history while decoupling the active expert slice from the released source storage.

The change also adds a CPU regression test that releases the source parameter storage after indexing and verifies backward and parameter gradient shape.

Validation

  • Fixed Torch 2.13.0+cu130 container
  • Transformers 5.17.0
  • pytest 7.4.4
  • CPU regression: 1 passed, 0 failed, exit code 0
  • No model weights downloaded

Scope

The CPU regression covers the storage lifetime sequence. Native Qwen3-MoE plus Gemini GPU validation remains pending, as do the later 8-card ZeRO-3 bootstrap, checkpoint recovery, full-parameter SFT, and benchmark.

The branch is based on the current official main and contains one independent commit with the fix and its regression test.

Fixes #6564

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG]: GeminiPlugin fails in backward on Qwen3-MoE (transformers v5): setStorage out of bounds for storage of size 0

1 participant