Repository navigation
[fix] preserve indexed Qwen3-MoE expert views for Gemini - #6575
Draft
refinedheart wants to merge 1 commit into
Draft
refinedheart wants to merge 1 commit into
refinedheart wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fix the storage lifetime issue in native Transformers v5 Qwen3-MoE expert indexing under Gemini.
When a 3D expert parameter is indexed, the resulting view can retain the source Gemini chunk storage. Gemini may release that storage after forward, while autograd still reads the view during backward. This can raise
setStorage ... storage size 0for parameters such asgate_up_proj.This change clones indexed results for 3D
ColoParameterinputs. The clone keeps the autograd history while decoupling the active expert slice from the released source storage.The change also adds a CPU regression test that releases the source parameter storage after indexing and verifies backward and parameter gradient shape.
Validation
2.13.0+cu130container5.17.07.4.41 passed,0 failed, exit code0Scope
The CPU regression covers the storage lifetime sequence. Native Qwen3-MoE plus Gemini GPU validation remains pending, as do the later 8-card ZeRO-3 bootstrap, checkpoint recovery, full-parameter SFT, and benchmark.
The branch is based on the current official
mainand contains one independent commit with the fix and its regression test.Fixes #6564