Repository navigation
[optimizer] add Muon with a built-in AdamW branch - #6521
Open
yuxuan-lou wants to merge 3 commits into
Open
yuxuan-lou wants to merge 3 commits into
yuxuan-lou wants to merge 3 commits into
Conversation
Add colossalai.nn.optimizer.Muon, which runs Muon on 2D hidden-layer weights and AdamW on every other parameter in one optimizer, so a Booster still gets a single optimizer. The Muon update follows torch.optim.Muon (EMA momentum, Nesterov, KellerJordan's Newton-Schulz coefficients in bf16, "original" or "match_rms_adamw" lr scaling); it is reimplemented because requirements still allow torch < 2.9. The AdamW branch uses torch.optim.AdamW's arithmetic. - muon.py: Muon, zeropower_via_newtonschulz, and muon_param_groups, which sends embeddings, the output head, norms and biases to AdamW (param.use_muon overrides). A 1D gradient, as LowLevelZeRO hands over its master slices, raises instead of being orthogonalized; ZeRO / Gemini / FSDP support is a follow-up. - test_muon.py: 3 steps against torch.optim.Muon + torch.optim.AdamW with a float64 truth transcribed from torch, and check_optimizer against torch.optim.Muon; wrong hyperparameters (lr scaling, nesterov, momentum 0.949, weight decay 0.09, lr x 1.05) must fail. Parameter grouping, error cases and the state_dict round trip. 2-GPU TorchDDP against one device under judge_relative_bound, with an unsynchronized run that must fail. - run_cpu_unit_tests.sh: run test_muon.py in the CPU job. Its torch 2.5.1 has no torch.optim.Muon, so the comparisons skip there; the tests/test_optimizer GPU batch picks them up. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
LowLevelZeRO (stage 1 and 2) hands the inner optimizer a flat 1/N slice of every parameter, while Newton-Schulz needs whole matrices, so Muon raised there. As in Moonlight's distributed Muon, each rank now updates the momentum of its own slices, the bf16 Newton-Schulz inputs are all-gathered over the data-parallel group, and every rank orthogonalizes the whole matrices and keeps the slice of each update it owns. The gathered input is the bf16 cast the unsharded iteration makes anyway, so a sharded step gives the same bits as the unsharded one. The cost is one bf16 all-gather of the Muon parameters per step (in buckets of at most 512 MiB) and every rank orthogonalizing all of them. - muon.py: Muon is a DistributedOptim. LowLevelZeroPlugin and HybridParallelPlugin call setup_distributed on the object itself; it is not replaced through cast_to_distributed, so an LR scheduler built on it before boost keeps driving it. Tensor parallelism and shards that do not span the data-parallel group (expert parameters) raise. - gemini_plugin.py, torch_fsdp_plugin.py: Muon raises NotImplementedError at boost. Gemini's chunks split matrices at arbitrary boundaries and FSDP hands over flattened parameters in one group. - test_dist_muon.py: hand-made shards over gloo for 2 and 3 ranks against whole matrices, bit for bit (runs in the CPU job); LowLevelZeRO with 2 ranks against 1 rank bit for bit (stage 1 and 2, fp32 and bf16, CPU offload), 4 ranks within judge_relative_bound, resume from a checkpoint bit for bit in fp32. A subclass that orthogonalizes each slice on its own must fail in both settings. - run_cpu_unit_tests.sh: run test_dist_muon.py in the CPU job. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
This was referenced Nov 18, 2026
[optimizer] run Muon under LowLevelZeRO on whole matrices
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
1. Related issues
No issue yet. This PR carries the first two steps of Muon support:
TorchDDPPlugin(commit 7c6dde0).LowLevelZeroPlugin, ZeRO stage 1 and 2 (commit 307baa1, merged into this branch from [optimizer] run Muon under LowLevelZeRO on whole matrices #6570, whose description has the details of that step).Gemini and FSDP reject Muon at
boost. OrScale follows in #6571, stacked on this branch.2. Background
Muon updates 2D hidden-layer weights with momentum orthogonalized by a Newton–Schulz iteration; embeddings, the output head, norms and biases still need AdamW.
torch.optim.Muonexists only from torch 2.9, whilerequirements.txtallows torch>=2.5, and it handles 2D parameters only, so a training script would need two optimizers, where a Booster takes one.LowLevelZeRO hands its inner optimizer a flat 1/N slice of every parameter, while Newton–Schulz needs whole matrices.
3. Changes
colossalai/nn/optimizer/muon.py:Muonruns Muon on parameter groups withuse_muon=Trueand AdamW on groups withuse_muon=False, all in one optimizer.torch.optim.Muon: EMA momentum, Nesterov, KellerJordan's Newton–Schulz coefficients in bf16, andadjust_lr_fn"original"or"match_rms_adamw"(Moonlight's 0.2·√max(m, n)). It is reimplemented with mm/addmm only, so it does not need torch 2.9.torch.optim.AdamW's arithmetic.muon_param_groups(model)builds both groups. Embeddings, the module returned byget_output_embeddings()(tied or not) and every non-2D parameter go to AdamW;param.use_muon = True/Falseoverrides the rule.step()thatsetup_distributedwas not told about raises a clear error instead of being orthogonalized.Muonis aDistributedOptim.LowLevelZeroPluginandHybridParallelPlugincallsetup_distributedon the object itself instead of replacing it throughcast_to_distributed, so an LR scheduler built on it beforeboostkeeps driving it. Tensor parallelism raises, and so do shards that do not span the data-parallel group.tests/test_optimizer/test_muon.py, usingtests/kit/numerics:torch.optim.Muon+torch.optim.AdamW, overweight_decay,nesterovandadjust_lr_fn, on CPU and CUDA. The float64 truth is transcribed fromtorch/optim/_muon.pywithout its bf16 cast; the weight deltas and momentum buffers are judged withjudge.scenarios.independent_impl.check_optimizeragainsttorch.optim.Muon.state_dictround trip.TorchDDPPluginagainst one device on the full batch withjudge_relative_bound(bf16 bound), plus a run without gradient synchronization that must fail.colossalai/booster/plugin/gemini_plugin.py,torch_fsdp_plugin.py(step 2):boostraisesNotImplementedErrorfor Muon and points to LowLevelZeRO and TorchDDP.tests/test_optimizer/test_dist_muon.py(step 2):LowLevelZeroPluginwith 2 ranks against 1 rank bit for bit (stage 1 and 2, fp32 and bf16, CPU offload); 4 ranks withinjudge_relative_bound; resume from a checkpoint in fp32 bit for bit..github/scripts/ci/run_cpu_unit_tests.sh: runstest_muon.pyandtest_dist_muon.pyin the CPU job.Not included: Gemini / FSDP support, tensor parallelism, an owner-per-matrix scheme under ZeRO, OrScale and the Polar Express coefficients (#6571), convolution weights (Muon groups take 2D parameters only), docs.
4. User and compatibility impact
colossalai.nn.optimizer.Muonandmuon_param_groups.Muonis aDistributedOptimwithsetup_distributed.GeminiPluginandTorchFSDPPluginraiseNotImplementedErroratboostwhen given Muon; nothing else existing changes.momentum_buffer, and the AdamW branch keepsstep,exp_avgandexp_avg_sq; hyperparameters are plain data (checked withjson.dumps).state_dict()/load_state_dict()are tested, and under LowLevelZeRO a resume from a sharded optimizer checkpoint in fp32, bit for bit.test_muon.pytakes about 6 s on CPU and 16 s with 2 GPUs.test_dist_muon.pyadds about 30 s in the CPU job; the two files together take about 3.5 minutes with 4 GPUs.5. Test environment
torch2.13-cu130-tf5.17.0nvidia-smiCUDA Version 13.2)torch.version.cuda13.0test_muon.pywas also run on a merge with main da8f5c2.fork_pr_checks.yml: torch 2.5.1+cpu,requirements/requirements.txt, Python 3.10.126. Test commands and results
Step 1, on 7c6dde0:
test_muon.py, 6 passed and 19 skipped: 18 compare againsttorch.optim.Muon(torch>=2.9), 1 needs 2 GPUs.Step 2, on 307baa1 merged with main d3916b5:
torch.optim.Muon, 10 need GPUs), exit code 0.run_cpu_unit_tests.shwas not rerun on this branch after step 2. It was run on the branch of [optimizer] add OrScale and OrScale-LM #6571, which contains both commits: 354 passed on torch 2.13 and 326 passed on torch 2.5.1, 0 failed.Details:
torch.optim.Muon+torch.optim.AdamW.FAST_TESTis set to 1 by the CPU script;test_muon.pydoes not read it. Noxfailor--ignore; the only skips are the torch-version and 2-GPU conditions above.test_muon_ddpafter the development container lost GPU access ("Failed to initialize NVML: Unknown Error" following a host cgroup reload, the problem [ci] Preserve GPU access and isolate container scratch files #6512 addresses in CI). No traceback was kept. After recreating the container with explicit--devicerules, every run passed.7. Known issues
torch.optim.Muonskips. The GPU batch that runstests/test_optimizerpicks them up; it has not been run for this PR yet.check_optimizerjudges the weights after each step, andjudgescalestolby each tensor's RMS. With unit-variance weights and lr = 1e-3 the updates are far below that scale, so the defaulttollets lr × 1.05 and momentum 0.9 pass. The test passestol=1e-7; the numerics README could mention this. Not introduced by this PR.cast_to_distributed, which leaves an LR scheduler built beforebooston the old object. That is [optimizer] keep the LR scheduler on the optimizer that steps after cast_to_distributed #6524 and is not touched here; Muon avoids it by being set up in place.8. Risk and rollback
Only code that builds
Muonis affected, plus oneisinstancecheck atboostin the Gemini and FSDP plugins. Rollback: revert the two commits.9. Docs and release
No README, docs, changelog or
version.txtchanges. The plugin compatibility table ofdistributed_optimizers.mddoes not list Muon yet.Checklist
🤖 Generated with Claude Code