Skip to content

[gemma4]: support 26B-A4B shared MoE L1-L2 validation - #6574

Open
richardoo-707 wants to merge 4 commits into
mainfrom
feature/gemma4-moe-l1l2-validation
Open

richardoo-707 wants to merge 4 commits into
mainfrom
feature/gemma4-moe-l1l2-validation

Conversation

@richardoo-707

Copy link
Copy Markdown
Collaborator

What changes

Fixes #6573 (phase-two 2C3). Builds on the merged shared-MoE implementation #6477.

Gemma 4 26B-A4B previously failed the dense-only policy gate and had no L2 registry entry. This adds a DP-only expert adapter using the shared MoE dispatch/grouped GEMM from #6477, preserving the original parameters, HF checkpoint keys, router, GELU and dense/routed branches. Explicit local dispatch prevents a DP WORLD group from becoming EP. Autocast casts and cast-per-expert contribution reduction follow native Gemma semantics.

The new structural tiny entry uses an unmodified eager HF reference and declares tied weights, token-normalized loss and router diagnostics. NumericsSpec.blank_control_fn permits an explicit HF self-control for the reference recipe; independent candidate-versus-HF checks retain the existing judge/tolerances. Fault injection confirms that a broken candidate still fails. Both FP16 runs declare initial scale 1024 after the native/reference trial overflowed at 65536.

Validation

Manual results at c057819, based on main 14d1c15:

Check Result
CPU model/shared-MoE/numerics regressions 125 passed, 41 accelerator-resource skips
Actual CUDA grouped GEMM and recomputation 7 passed, 0 skips
MoE L1, 1/2 GPUs, 6 plugins, normal/lazy+checkpoint 24 passed, 0 skips
Unified L2, default 14 configurations 57 passed, 0 failed/unsupported/not_run, including 28 two-GPU DP checks
Existing dense E2B/E4B training/alignment 62 passed, 0 skips
Checkpoint regressions 21 passed, 0 skips

Environment: maintained Torch 2.13.0+cu130 / Transformers 5.17.0 / Python 3.10.12 image, H20 GPU4/5, FAST_TEST=FAST_MODE=0, OMP=MKL=1. The primary run used gpu-h20-5; the supplemental regression used gpu-h20-4 with identical saved image config/rootfs verified across Docker storage backends. Black 24.10, isort, diff and syntax checks passed. These manual checks do not substitute for this PR's CI.

The first dense-regression run stopped after 46 passes on TCPStore EADDRINUSE before model arithmetic. Its log and exit 1 are retained. The failing node passed a targeted rerun, then the full 62-case dense suite and 21-case checkpoint suite passed on the same SHA. No numerical retries or tolerance relaxation. Existing optional-extension/deprecation warnings and a CheckpointIO destructor warning in the negative checkpoint probes remain recorded.

Standard report:
/mnt/beegfs/ColossalAI/shared-results/numerics/2C/transformers_gemma4_26b_a4b_for_causal_lm/20261119-c0578194-tiny-two-gpu.json

SHA256: 049f1a6d31860b51e745833b837475df83131df1457e0d3501d25c55625eff02.
Evidence: /mnt/beegfs/ricardoo/gemma4-moe-20261119/verified-c0578194.json and evidence-final-c0578194.tar.gz; 89 final evidence files verified. Task containers/locks released.

Boundaries

This validates a structural tiny text model, not real 26B weights/training: six layers (5 sliding + 1 full), hidden32, dense64/expert32, eight experts/top2, global K=V, no PLE/cross-layer KV sharing. EP/TP/PP, multimodal behavior, long-context, convergence and performance remain outside this change. Dense E4B real-weight training evidence remains with the earlier PR and is not re-claimed here.

  • Self-reviewed the change and added relevant tests/documentation.
  • Preserved failures, skips and exact validation SHA rather than claiming zero-skip CPU/all-CI success.

@richardoo-707
richardoo-707 requested a review from a team as a code owner November 19, 2026 15:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[gemma4]: track 26B-A4B shared MoE L1-L2 integration

1 participant