Skip to content

[gemma4] Track 2C3 26B-A4B shared MoE L1-L2 support #6572

Open

Description

@richardoo-707

Problem

Phase-two task 2C3 needs Gemma 4 26B-A4B L1-L2 coverage using the shared MoE implementation merged in #6477. The existing Gemma policy from #6520 supports dense E2B/E4B and rejects MoE. A generic expert replacement must preserve Gemma's native router, GELU, parallel dense/routed branches, parameter identities and low-precision combine order; DP must not accidentally use WORLD as an expert-parallel group.

Scope and acceptance

  • Add replicated shared dispatch/grouped-GEMM experts for TP=1, EP=1, retaining native HF parameter names and router/decoder behavior.
  • Register a structural tiny 26B-A4B text entry with an independent native HF reference, tied-parameter and valid-token contracts, and router diagnostics.
  • Validate CUDA grouped GEMM, lazy/checkpointed training, one/two-GPU DP, deterministic resume/HF readback, and existing E2B/E4B compatibility.
  • Submit the standard 14-configuration numerical JSON to the team shared-results directory. Explicitly disclose precision recipes, skips and infrastructure retries.

Full-size 26B training, convergence/performance, long contexts, multimodal validation and EP/TP/PP expansion are outside this task.

Current evidence

Commit c0578194496dce463fd51b31fefe766a3472a169: CPU 125 passed/41 accelerator-dependent skips; CUDA expert tests 7 passed; tiny MoE L1 24 passed; default unified L2 57 passed (including 28 two-GPU DP checks), zero failed/unsupported/not_run; dense regression 62 passed; checkpoint regression 21 passed. One prior TCPStore EADDRINUSE infrastructure failure was retained and followed by focused and full same-SHA reruns with identical image contents.

This issue tracks 2C3 separately from the earlier 2C2 numerics requirements in #6488.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions