You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[gemma4] Track 2C3 26B-A4B shared MoE L1-L2 support #6572
Phase-two task 2C3 needs Gemma 4 26B-A4B L1-L2 coverage using the shared MoE implementation merged in #6477. The existing Gemma policy from #6520 supports dense E2B/E4B and rejects MoE. A generic expert replacement must preserve Gemma's native router, GELU, parallel dense/routed branches, parameter identities and low-precision combine order; DP must not accidentally use WORLD as an expert-parallel group.
Scope and acceptance
Add replicated shared dispatch/grouped-GEMM experts for TP=1, EP=1, retaining native HF parameter names and router/decoder behavior.
Register a structural tiny 26B-A4B text entry with an independent native HF reference, tied-parameter and valid-token contracts, and router diagnostics.
Validate CUDA grouped GEMM, lazy/checkpointed training, one/two-GPU DP, deterministic resume/HF readback, and existing E2B/E4B compatibility.
Submit the standard 14-configuration numerical JSON to the team shared-results directory. Explicitly disclose precision recipes, skips and infrastructure retries.
Full-size 26B training, convergence/performance, long contexts, multimodal validation and EP/TP/PP expansion are outside this task.
Current evidence
Commit c0578194496dce463fd51b31fefe766a3472a169: CPU 125 passed/41 accelerator-dependent skips; CUDA expert tests 7 passed; tiny MoE L1 24 passed; default unified L2 57 passed (including 28 two-GPU DP checks), zero failed/unsupported/not_run; dense regression 62 passed; checkpoint regression 21 passed. One prior TCPStore EADDRINUSE infrastructure failure was retained and followed by focused and full same-SHA reruns with identical image contents.
This issue tracks 2C3 separately from the earlier 2C2 numerics requirements in #6488.
Activity