Repository navigation
[optimizer] run Muon under LowLevelZeRO on whole matrices - #6570
Merged
Merged
Conversation
LowLevelZeRO (stage 1 and 2) hands the inner optimizer a flat 1/N slice of every parameter, while Newton-Schulz needs whole matrices, so Muon raised there. As in Moonlight's distributed Muon, each rank now updates the momentum of its own slices, the bf16 Newton-Schulz inputs are all-gathered over the data-parallel group, and every rank orthogonalizes the whole matrices and keeps the slice of each update it owns. The gathered input is the bf16 cast the unsharded iteration makes anyway, so a sharded step gives the same bits as the unsharded one. The cost is one bf16 all-gather of the Muon parameters per step (in buckets of at most 512 MiB) and every rank orthogonalizing all of them. - muon.py: Muon is a DistributedOptim. LowLevelZeroPlugin and HybridParallelPlugin call setup_distributed on the object itself; it is not replaced through cast_to_distributed, so an LR scheduler built on it before boost keeps driving it. Tensor parallelism and shards that do not span the data-parallel group (expert parameters) raise. - gemini_plugin.py, torch_fsdp_plugin.py: Muon raises NotImplementedError at boost. Gemini's chunks split matrices at arbitrary boundaries and FSDP hands over flattened parameters in one group. - test_dist_muon.py: hand-made shards over gloo for 2 and 3 ranks against whole matrices, bit for bit (runs in the CPU job); LowLevelZeRO with 2 ranks against 1 rank bit for bit (stage 1 and 2, fp32 and bf16, CPU offload), 4 ranks within judge_relative_bound, resume from a checkpoint bit for bit in fp32. A subclass that orthogonalizes each slice on its own must fail in both settings. - run_cpu_unit_tests.sh: run test_dist_muon.py in the CPU job. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
4 of 7 tasks
4 of 7 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
1. Related issues
No issue yet. This is the second step of Muon support: Muon under
LowLevelZeroPlugin(ZeRO stage 1 and 2).It is stacked on #6521: the base branch is
feat/muon-optimizer, so the diff shows this step only. OrScale follows in #6571, stacked on this one.The PR gates (
ci_layer_1_pr.yml,ci_layer_2_gpu_pr.yml,fork_pr_checks.yml) run for pull requests tomainonly, so they do not run here until this PR is retargeted tomainafter #6521 merges. Section 6 has the local runs on a merge withmain.2. Background
LowLevelZeRO hands its inner optimizer a flat 1/N slice of every parameter (padded to a multiple of N), while Muon's Newton–Schulz iteration needs whole matrices. With #6521 alone, Muon raises under LowLevelZeRO.
3. Changes
colossalai/nn/optimizer/muon.py:Muonis now aDistributedOptim.LowLevelZeroPluginandHybridParallelPlugincallsetup_distributedon the object itself; it is not replaced throughcast_to_distributed, so an LR scheduler built on it beforeboostkeeps driving it.colossalai/booster/plugin/gemini_plugin.py,torch_fsdp_plugin.py:boostraisesNotImplementedErrorfor Muon and points to LowLevelZeRO and TorchDDP. Gemini's chunks split matrices at arbitrary boundaries; FSDP hands the optimizer flattened parameters in one group. Neither could run Muon before.tests/test_optimizer/test_dist_muon.py, usingtests/kit/numerics:judge_equal). The sizes are odd, so every matrix is padded. Runs without GPUs.LowLevelZeroPluginwith 2 ranks against 1 rank: losses, weights, float32 master weights, momentum and AdamW moments bit for bit, for stage 1 and 2 in fp32 and bf16, and stage 2 in bf16 with CPU offload.judge_relative_boundon the updates (averaging 4 equal gradients rounds).setup_distributedrejects tensor parallelism and foreign shards; Gemini and FSDP reject Muon; LowLevelZeRO sets Muon up in place..github/scripts/ci/run_cpu_unit_tests.sh: runstest_dist_muon.pyin the CPU job (the gloo tests; the rest skip without GPUs).Not included: Gemini and FSDP support, tensor parallelism, an owner-per-matrix scheme (each matrix orthogonalized by one rank instead of all), OrScale, docs.
4. User and compatibility impact
Muonsubclassescolossalai.interface.optimizer.DistributedOptim(still atorch.optim.Optimizer) and gainssetup_distributed. UnderLowLevelZeroPluginit now trains instead of raising. UnderGeminiPluginandTorchFSDPPluginit fails atboostwith a message naming the plugins that work.test_muon.pyandtest_dist_muon.pytogether take about 3.5 minutes with 4 GPUs.5. Test environment
torch2.13-cu130-tf5.17.0nvidia-smiCUDA Version 13.2)torch.version.cuda13.06. Test commands and results
torch.optim.Muon, 10 need GPUs), exit code 0.Details:
run_cpu_unit_tests.shwas not run on this commit alone. It was run on the branch of [optimizer] add OrScale and OrScale-LM #6571, which contains this commit: 354 passed on torch 2.13 and 326 passed on torch 2.5.1, 0 failed.xfailor--ignore; the only skips are the torch-version and GPU-count conditions above.7. Known issues
main(section 1).tests/test_optimizer, not in the PR gate.cast_to_distributed, which leaves an LR scheduler built beforebooston the old object. That is [optimizer] keep the LR scheduler on the optimizer that steps after cast_to_distributed #6524 and is not touched here; Muon avoids it by being set up in place.8. Risk and rollback
Only code that builds
Muonis affected, plus oneisinstancecheck atboostin the Gemini and FSDP plugins. Rollback: revert this commit.9. Docs and release
No README, docs, changelog or
version.txtchanges. The plugin compatibility table ofdistributed_optimizers.mddoes not list Muon yet.Checklist
🤖 Generated with Claude Code