Skip to content

[optimizer] add Muon with a built-in AdamW branch - #6521

Open
yuxuan-lou wants to merge 3 commits into
mainfrom
feat/muon-optimizer
Open

yuxuan-lou wants to merge 3 commits into
mainfrom
feat/muon-optimizer

Conversation

@yuxuan-lou

@yuxuan-lou yuxuan-lou commented Nov 11, 2026 •

Copy link
Copy Markdown
Collaborator

1. Related issues

No issue yet. This PR carries the first two steps of Muon support:

  1. Single device and data parallelism through TorchDDPPlugin (commit 7c6dde0).
  2. LowLevelZeroPlugin, ZeRO stage 1 and 2 (commit 307baa1, merged into this branch from [optimizer] run Muon under LowLevelZeRO on whole matrices #6570, whose description has the details of that step).

Gemini and FSDP reject Muon at boost. OrScale follows in #6571, stacked on this branch.

2. Background

Muon updates 2D hidden-layer weights with momentum orthogonalized by a Newton–Schulz iteration; embeddings, the output head, norms and biases still need AdamW. torch.optim.Muon exists only from torch 2.9, while requirements.txt allows torch>=2.5, and it handles 2D parameters only, so a training script would need two optimizers, where a Booster takes one.

LowLevelZeRO hands its inner optimizer a flat 1/N slice of every parameter, while Newton–Schulz needs whole matrices.

3. Changes

  • colossalai/nn/optimizer/muon.py:
    • Muon runs Muon on parameter groups with use_muon=True and AdamW on groups with use_muon=False, all in one optimizer.
    • The Muon update follows torch.optim.Muon: EMA momentum, Nesterov, KellerJordan's Newton–Schulz coefficients in bf16, and adjust_lr_fn "original" or "match_rms_adamw" (Moonlight's 0.2·√max(m, n)). It is reimplemented with mm/addmm only, so it does not need torch 2.9.
    • The AdamW branch uses torch.optim.AdamW's arithmetic.
    • muon_param_groups(model) builds both groups. Embeddings, the module returned by get_output_embeddings() (tied or not) and every non-2D parameter go to AdamW; param.use_muon = True/False overrides the rule.
    • A Muon group with a non-2D parameter is rejected at construction. A 1D gradient at step() that setup_distributed was not told about raises a clear error instead of being orthogonalized.
    • Step 2, under LowLevelZeRO, as in Moonlight's distributed Muon: each rank updates the momentum of its own slices, the bf16 Newton–Schulz inputs are all-gathered over the data-parallel group in buckets of at most 2^28 elements, and every rank orthogonalizes the whole matrices and keeps the slice of each update it owns. The gathered input is the bf16 cast the unsharded iteration makes anyway, so a sharded step gives the same bits as the unsharded one.
    • Step 2: Muon is a DistributedOptim. LowLevelZeroPlugin and HybridParallelPlugin call setup_distributed on the object itself instead of replacing it through cast_to_distributed, so an LR scheduler built on it before boost keeps driving it. Tensor parallelism raises, and so do shards that do not span the data-parallel group.
  • tests/test_optimizer/test_muon.py, using tests/kit/numerics:
    • Newton–Schulz and 3 optimizer steps (Muon + AdamW branch) against torch.optim.Muon + torch.optim.AdamW, over weight_decay, nesterov and adjust_lr_fn, on CPU and CUDA. The float64 truth is transcribed from torch/optim/_muon.py without its bf16 cast; the weight deltas and momentum buffers are judged with judge.
    • scenarios.independent_impl.check_optimizer against torch.optim.Muon.
    • Negative checks that must fail: wrong lr scaling, Nesterov off, momentum 0.9 and 0.949, weight decay 0.09, lr × 1.05.
    • Parameter grouping (tied and untied head, overrides), the two error cases, and a state_dict round trip.
    • 2-GPU TorchDDPPlugin against one device on the full batch with judge_relative_bound (bf16 bound), plus a run without gradient synchronization that must fail.
  • colossalai/booster/plugin/gemini_plugin.py, torch_fsdp_plugin.py (step 2): boost raises NotImplementedError for Muon and points to LowLevelZeRO and TorchDDP.
  • tests/test_optimizer/test_dist_muon.py (step 2):
    • Hand-made ZeRO shards over gloo, 2 and 3 ranks, against whole matrices, bit for bit. Runs without GPUs.
    • LowLevelZeroPlugin with 2 ranks against 1 rank bit for bit (stage 1 and 2, fp32 and bf16, CPU offload); 4 ranks within judge_relative_bound; resume from a checkpoint in fp32 bit for bit.
    • Negative checks that must fail: a subclass that orthogonalizes each rank's slice on its own.
  • .github/scripts/ci/run_cpu_unit_tests.sh: runs test_muon.py and test_dist_muon.py in the CPU job.

Not included: Gemini / FSDP support, tensor parallelism, an owner-per-matrix scheme under ZeRO, OrScale and the Polar Express coefficients (#6571), convolution weights (Muon groups take 2D parameters only), docs.

4. User and compatibility impact

  • Python API: adds colossalai.nn.optimizer.Muon and muon_param_groups. Muon is a DistributedOptim with setup_distributed. GeminiPlugin and TorchFSDPPlugin raise NotImplementedError at boost when given Muon; nothing else existing changes.
  • Checkpoint format: no change. Muon's optimizer state is momentum_buffer, and the AdamW branch keeps step, exp_avg and exp_avg_sq; hyperparameters are plain data (checked with json.dumps). state_dict() / load_state_dict() are tested, and under LowLevelZeRO a resume from a sharded optimizer checkpoint in fp32, bit for bit.
  • Default behavior, model accuracy, performance or memory of existing code: none.
  • Performance and memory under LowLevelZeRO: one bf16 all-gather of the Muon parameters per step, and every rank orthogonalizes all matrices, N times the work of an owner-per-matrix scheme. No throughput numbers in this PR.
  • Python / PyTorch / CUDA versions, C++/CUDA kernels, Docker images, existing training scripts: none. Muon itself runs on torch>=2.5; the steps run on torch 2.5.1 in the CPU job.
  • CI time: test_muon.py takes about 6 s on CPU and 16 s with 2 GPUs. test_dist_muon.py adds about 30 s in the CPU job; the two files together take about 3.5 minutes with 4 GPUs.

5. Test environment

  • OS: Ubuntu 22.04.1 LTS host, CI image torch2.13-cu130-tf5.17.0
  • Python: 3.10.12
  • GPU: 2× NVIDIA H20 for step 1, 4× for step 2
  • NVIDIA driver: 595.71.05 (nvidia-smi CUDA Version 13.2)
  • PyTorch: 2.13.0+cu130, torch.version.cuda 13.0
  • CUDA toolkit / nvcc: 13.0 (V13.0.88)
  • Transformers: 5.17.0
  • flash-attn: not installed
  • ColossalAI, step 1: 7c6dde0 on main 69c9ff5. test_muon.py was also run on a merge with main da8f5c2.
  • ColossalAI, step 2: 307baa1 merged with main d3916b5 (no conflicts). The current head f5c44d9 has the same tree as 307baa1.
  • Also the CPU job's environment, rebuilt with the install commands of fork_pr_checks.yml: torch 2.5.1+cpu, requirements/requirements.txt, Python 3.10.12

6. Test commands and results

Step 1, on 7c6dde0:

CUDA_VISIBLE_DEVICES= python -m pytest -q tests/test_optimizer/test_muon.py
  • 24 passed, 0 failed, 1 skipped (the DDP test needs 2 GPUs), exit code 0. Same on the merge with main da8f5c2.
CUDA_VISIBLE_DEVICES=6,7 python -m pytest -q tests/test_optimizer/test_muon.py
  • 36 passed, 0 failed, 0 skipped, exit code 0, in 3 consecutive runs, and once more on the merge with main da8f5c2.
bash .github/scripts/ci/run_cpu_unit_tests.sh
  • torch 2.13: 186 passed, 0 failed, 20 skipped, exit code 0.
  • torch 2.5.1 (CPU job environment): 161 passed, 0 failed, 45 skipped, exit code 0. In test_muon.py, 6 passed and 19 skipped: 18 compare against torch.optim.Muon (torch>=2.9), 1 needs 2 GPUs.

Step 2, on 307baa1 merged with main d3916b5:

CUDA_VISIBLE_DEVICES= python -m pytest -q tests/test_optimizer/test_muon.py tests/test_optimizer/test_dist_muon.py
  • torch 2.13: 28 passed, 0 failed, 10 skipped (they need GPUs), exit code 0.
  • torch 2.5.1: 10 passed, 0 failed, 28 skipped (18 compare against torch.optim.Muon, 10 need GPUs), exit code 0.
CUDA_VISIBLE_DEVICES=<4 GPUs> python -m pytest -q tests/test_optimizer/test_muon.py tests/test_optimizer/test_dist_muon.py
  • 49 passed, 0 failed, 0 skipped, exit code 0.
  • The full run_cpu_unit_tests.sh was not rerun on this branch after step 2. It was run on the branch of [optimizer] add OrScale and OrScale-LM #6571, which contains both commits: 354 passed on torch 2.13 and 326 passed on torch 2.5.1, 0 failed.

Details:

  • On CPU in fp32 the updates are bit for bit those of torch.optim.Muon + torch.optim.AdamW.
  • DDP against one device: largest relative error 1.28e-7 (bound 8.29%). Without gradient synchronization, all 6 tensors fail at 4.9–12.3× the bound.
  • FAST_TEST is set to 1 by the CPU script; test_muon.py does not read it. No xfail or --ignore; the only skips are the torch-version and 2-GPU conditions above.
  • Reruns: one GPU run failed in test_muon_ddp after the development container lost GPU access ("Failed to initialize NVML: Unknown Error" following a host cgroup reload, the problem [ci] Preserve GPU access and isolate container scratch files #6512 addresses in CI). No traceback was kept. After recreating the container with explicit --device rules, every run passed.
  • Logs and JUnit XML of the CPU script are kept on the development server, not attached.

7. Known issues

  • In the CPU job (torch 2.5.1) every comparison against torch.optim.Muon skips. The GPU batch that runs tests/test_optimizer picks them up; it has not been run for this PR yet.
  • check_optimizer judges the weights after each step, and judge scales tol by each tensor's RMS. With unit-variance weights and lr = 1e-3 the updates are far below that scale, so the default tol lets lr × 1.05 and momentum 0.9 pass. The test passes tol=1e-7; the numerics README could mention this. Not introduced by this PR.
  • The LowLevelZeRO tests need 2 GPUs, one of them 4. They run in the same GPU batch, not in the PR gate.
  • Only fp32 resume is checked bit for bit. In bf16 no optimizer resumes exactly under LowLevelZeRO, as the numerics README describes; not introduced here.
  • The other distributed optimizers are replaced by cast_to_distributed, which leaves an LR scheduler built before boost on the old object. That is [optimizer] keep the LR scheduler on the optimizer that steps after cast_to_distributed #6524 and is not touched here; Muon avoids it by being set up in place.

8. Risk and rollback

Only code that builds Muon is affected, plus one isinstance check at boost in the Gemini and FSDP plugins. Rollback: revert the two commits.

9. Docs and release

No README, docs, changelog or version.txt changes. The plugin compatibility table of distributed_optimizers.md does not list Muon yet.

Checklist

  • I have created an issue for this PR for traceability (none yet)
  • The title follows the standard format
  • I have added relevant tags
  • I have installed pre-commit. The hooks' formatters at their pinned versions (autoflake 2.3.1, isort 5.13.2, black 24.10.0) were run on the changed files and left them unchanged.
  • I have performed a self-review of my code
  • I have added thorough tests
  • I have added docstrings for the public classes and functions

🤖 Generated with Claude Code

Add colossalai.nn.optimizer.Muon, which runs Muon on 2D hidden-layer weights
and AdamW on every other parameter in one optimizer, so a Booster still gets a
single optimizer. The Muon update follows torch.optim.Muon (EMA momentum,
Nesterov, KellerJordan's Newton-Schulz coefficients in bf16, "original" or
"match_rms_adamw" lr scaling); it is reimplemented because requirements still
allow torch < 2.9. The AdamW branch uses torch.optim.AdamW's arithmetic.

- muon.py: Muon, zeropower_via_newtonschulz, and muon_param_groups, which
  sends embeddings, the output head, norms and biases to AdamW
  (param.use_muon overrides). A 1D gradient, as LowLevelZeRO hands over its
  master slices, raises instead of being orthogonalized; ZeRO / Gemini /
  FSDP support is a follow-up.
- test_muon.py: 3 steps against torch.optim.Muon + torch.optim.AdamW with a
  float64 truth transcribed from torch, and check_optimizer against
  torch.optim.Muon; wrong hyperparameters (lr scaling, nesterov, momentum
  0.949, weight decay 0.09, lr x 1.05) must fail. Parameter grouping, error
  cases and the state_dict round trip. 2-GPU TorchDDP against one device
  under judge_relative_bound, with an unsynchronized run that must fail.
- run_cpu_unit_tests.sh: run test_muon.py in the CPU job. Its torch 2.5.1
  has no torch.optim.Muon, so the comparisons skip there; the
  tests/test_optimizer GPU batch picks them up.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@yuxuan-lou
yuxuan-lou requested a review from a team as a code owner November 11, 2026 07:42
LowLevelZeRO (stage 1 and 2) hands the inner optimizer a flat 1/N slice of
every parameter, while Newton-Schulz needs whole matrices, so Muon raised
there. As in Moonlight's distributed Muon, each rank now updates the momentum
of its own slices, the bf16 Newton-Schulz inputs are all-gathered over the
data-parallel group, and every rank orthogonalizes the whole matrices and
keeps the slice of each update it owns. The gathered input is the bf16 cast
the unsharded iteration makes anyway, so a sharded step gives the same bits
as the unsharded one. The cost is one bf16 all-gather of the Muon parameters
per step (in buckets of at most 512 MiB) and every rank orthogonalizing all
of them.

- muon.py: Muon is a DistributedOptim. LowLevelZeroPlugin and
  HybridParallelPlugin call setup_distributed on the object itself; it is
  not replaced through cast_to_distributed, so an LR scheduler built on it
  before boost keeps driving it. Tensor parallelism and shards that do not
  span the data-parallel group (expert parameters) raise.
- gemini_plugin.py, torch_fsdp_plugin.py: Muon raises NotImplementedError at
  boost. Gemini's chunks split matrices at arbitrary boundaries and FSDP
  hands over flattened parameters in one group.
- test_dist_muon.py: hand-made shards over gloo for 2 and 3 ranks against
  whole matrices, bit for bit (runs in the CPU job); LowLevelZeRO with 2
  ranks against 1 rank bit for bit (stage 1 and 2, fp32 and bf16, CPU
  offload), 4 ranks within judge_relative_bound, resume from a checkpoint
  bit for bit in fp32. A subclass that orthogonalizes each slice on its own
  must fail in both settings.
- run_cpu_unit_tests.sh: run test_dist_muon.py in the CPU job.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
[optimizer] run Muon under LowLevelZeRO on whole matrices
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant