Repository navigation
[gemma4]: support 26B-A4B shared MoE L1-L2 validation - #6574
Open
richardoo-707 wants to merge 4 commits into
Open
richardoo-707 wants to merge 4 commits into
richardoo-707 wants to merge 4 commits into
Conversation
richardoo-707
enabled auto-merge
November 19, 2026 15:39
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes
Fixes #6573 (phase-two 2C3). Builds on the merged shared-MoE implementation #6477.
Gemma 4 26B-A4B previously failed the dense-only policy gate and had no L2 registry entry. This adds a DP-only expert adapter using the shared MoE dispatch/grouped GEMM from #6477, preserving the original parameters, HF checkpoint keys, router, GELU and dense/routed branches. Explicit local dispatch prevents a DP WORLD group from becoming EP. Autocast casts and cast-per-expert contribution reduction follow native Gemma semantics.
The new structural tiny entry uses an unmodified eager HF reference and declares tied weights, token-normalized loss and router diagnostics.
NumericsSpec.blank_control_fnpermits an explicit HF self-control for the reference recipe; independent candidate-versus-HF checks retain the existing judge/tolerances. Fault injection confirms that a broken candidate still fails. Both FP16 runs declare initial scale 1024 after the native/reference trial overflowed at 65536.Validation
Manual results at c057819, based on main 14d1c15:
Environment: maintained Torch 2.13.0+cu130 / Transformers 5.17.0 / Python 3.10.12 image, H20 GPU4/5, FAST_TEST=FAST_MODE=0, OMP=MKL=1. The primary run used gpu-h20-5; the supplemental regression used gpu-h20-4 with identical saved image config/rootfs verified across Docker storage backends. Black 24.10, isort, diff and syntax checks passed. These manual checks do not substitute for this PR's CI.
The first dense-regression run stopped after 46 passes on TCPStore EADDRINUSE before model arithmetic. Its log and exit 1 are retained. The failing node passed a targeted rerun, then the full 62-case dense suite and 21-case checkpoint suite passed on the same SHA. No numerical retries or tolerance relaxation. Existing optional-extension/deprecation warnings and a CheckpointIO destructor warning in the negative checkpoint probes remain recorded.
Standard report:
/mnt/beegfs/ColossalAI/shared-results/numerics/2C/transformers_gemma4_26b_a4b_for_causal_lm/20261119-c0578194-tiny-two-gpu.jsonSHA256:
049f1a6d31860b51e745833b837475df83131df1457e0d3501d25c55625eff02.Evidence:
/mnt/beegfs/ricardoo/gemma4-moe-20261119/verified-c0578194.jsonandevidence-final-c0578194.tar.gz; 89 final evidence files verified. Task containers/locks released.Boundaries
This validates a structural tiny text model, not real 26B weights/training: six layers (5 sliding + 1 full), hidden32, dense64/expert32, eight experts/top2, global K=V, no PLE/cross-layer KV sharing. EP/TP/PP, multimodal behavior, long-context, convergence and performance remain outside this change. Dense E4B real-weight training evidence remains with the earlier PR and is not re-claimed here.