Skip to content

[optimizer] run Muon under LowLevelZeRO on whole matrices - #6570

Merged
yuxuan-lou merged 1 commit into
feat/muon-optimizerfrom
feat/muon-zero
Nov 18, 2026
Merged

yuxuan-lou merged 1 commit into
feat/muon-optimizerfrom
feat/muon-zero

Conversation

@yuxuan-lou

@yuxuan-lou yuxuan-lou commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

1. Related issues

No issue yet. This is the second step of Muon support: Muon under LowLevelZeroPlugin (ZeRO stage 1 and 2).

It is stacked on #6521: the base branch is feat/muon-optimizer, so the diff shows this step only. OrScale follows in #6571, stacked on this one.

The PR gates (ci_layer_1_pr.yml, ci_layer_2_gpu_pr.yml, fork_pr_checks.yml) run for pull requests to main only, so they do not run here until this PR is retargeted to main after #6521 merges. Section 6 has the local runs on a merge with main.

2. Background

LowLevelZeRO hands its inner optimizer a flat 1/N slice of every parameter (padded to a multiple of N), while Muon's Newton–Schulz iteration needs whole matrices. With #6521 alone, Muon raises under LowLevelZeRO.

3. Changes

  • colossalai/nn/optimizer/muon.py:
    • As in Moonlight's distributed Muon: each rank updates the momentum of its own slices, the bf16 Newton–Schulz inputs are all-gathered over the data-parallel group, and every rank orthogonalizes the whole matrices and keeps the slice of each update it owns.
    • The gathered input is the bf16 cast the unsharded iteration makes anyway, so a sharded step gives the same bits as the unsharded one.
    • The gather runs in buckets of at most 2^28 elements (512 MiB in bf16), which bounds the extra memory of a step. With CPU offload the inputs are moved to the accelerator first.
    • Muon is now a DistributedOptim. LowLevelZeroPlugin and HybridParallelPlugin call setup_distributed on the object itself; it is not replaced through cast_to_distributed, so an LR scheduler built on it before boost keeps driving it.
    • Tensor parallelism raises, and so do shards that do not span the data-parallel group (expert parameters that ZeRO shards over their own group).
    • The AdamW branch is elementwise and runs on the shards as they are.
  • colossalai/booster/plugin/gemini_plugin.py, torch_fsdp_plugin.py: boost raises NotImplementedError for Muon and points to LowLevelZeRO and TorchDDP. Gemini's chunks split matrices at arbitrary boundaries; FSDP hands the optimizer flattened parameters in one group. Neither could run Muon before.
  • tests/test_optimizer/test_dist_muon.py, using tests/kit/numerics:
    • Hand-made ZeRO shards over gloo, 2 and 3 ranks, against whole matrices: updates and momentum bit for bit (judge_equal). The sizes are odd, so every matrix is padded. Runs without GPUs.
    • LowLevelZeroPlugin with 2 ranks against 1 rank: losses, weights, float32 master weights, momentum and AdamW moments bit for bit, for stage 1 and 2 in fp32 and bf16, and stage 2 in bf16 with CPU offload.
    • 4 ranks against 1 rank with judge_relative_bound on the updates (averaging 4 equal gradients rounds).
    • Negative checks that must fail, on the hand-made shards and under the plugin: a subclass that orthogonalizes each rank's slice on its own.
    • Resume from a checkpoint in fp32 (sharded optimizer checkpoint), bit for bit.
    • setup_distributed rejects tensor parallelism and foreign shards; Gemini and FSDP reject Muon; LowLevelZeRO sets Muon up in place.
  • .github/scripts/ci/run_cpu_unit_tests.sh: runs test_dist_muon.py in the CPU job (the gloo tests; the rest skip without GPUs).

Not included: Gemini and FSDP support, tensor parallelism, an owner-per-matrix scheme (each matrix orthogonalized by one rank instead of all), OrScale, docs.

4. User and compatibility impact

  • Python API: Muon subclasses colossalai.interface.optimizer.DistributedOptim (still a torch.optim.Optimizer) and gains setup_distributed. Under LowLevelZeroPlugin it now trains instead of raising. Under GeminiPlugin and TorchFSDPPlugin it fails at boost with a message naming the plugins that work.
  • Checkpoint format: no change. Under ZeRO Muon's state is the momentum of each slice plus the AdamW branch's state, which LowLevelZeRO's checkpoint IO gathers and splits like any other optimizer's tensors.
  • Performance and memory: one bf16 all-gather of the Muon parameters per step, and every rank orthogonalizes all matrices, N times the work of an owner-per-matrix scheme. No throughput numbers in this PR.
  • Default behavior, model accuracy, Python / PyTorch / CUDA versions, C++/CUDA kernels, Docker images, existing training scripts: none.
  • CI time: about 30 s more in the CPU job (4 tests that spawn 2 or 3 processes); test_muon.py and test_dist_muon.py together take about 3.5 minutes with 4 GPUs.

5. Test environment

  • OS: Ubuntu 22.04.1 LTS host, CI image torch2.13-cu130-tf5.17.0
  • Python: 3.10.12
  • GPU: 4× NVIDIA H20
  • NVIDIA driver: 595.71.05 (nvidia-smi CUDA Version 13.2)
  • PyTorch: 2.13.0+cu130, torch.version.cuda 13.0
  • CUDA toolkit / nvcc: 13.0
  • Transformers: 5.17.0
  • flash-attn: not installed
  • ColossalAI: this branch, 307baa1, merged with main d3916b5 (no conflicts)
  • Also the CPU job's environment: torch 2.5.1+cpu, Python 3.10.12

6. Test commands and results

CUDA_VISIBLE_DEVICES= python -m pytest -q tests/test_optimizer/test_muon.py tests/test_optimizer/test_dist_muon.py
  • torch 2.13: 28 passed, 0 failed, 10 skipped (they need GPUs), exit code 0.
  • torch 2.5.1: 10 passed, 0 failed, 28 skipped (18 compare against torch.optim.Muon, 10 need GPUs), exit code 0.
CUDA_VISIBLE_DEVICES=<4 GPUs> python -m pytest -q tests/test_optimizer/test_muon.py tests/test_optimizer/test_dist_muon.py
  • 49 passed, 0 failed, 0 skipped, exit code 0.

Details:

  • The full run_cpu_unit_tests.sh was not run on this commit alone. It was run on the branch of [optimizer] add OrScale and OrScale-LM #6571, which contains this commit: 354 passed on torch 2.13 and 326 passed on torch 2.5.1, 0 failed.
  • No xfail or --ignore; the only skips are the torch-version and GPU-count conditions above.
  • Logs are kept on the development server, not attached.

7. Known issues

  • The CI gates do not run on this PR while its base is not main (section 1).
  • The LowLevelZeRO tests need 2 GPUs, one of them 4. They run in the GPU batch that covers tests/test_optimizer, not in the PR gate.
  • Only fp32 resume is checked bit for bit. In bf16 no optimizer resumes exactly under LowLevelZeRO, as the numerics README describes; not introduced here.
  • The other distributed optimizers are replaced by cast_to_distributed, which leaves an LR scheduler built before boost on the old object. That is [optimizer] keep the LR scheduler on the optimizer that steps after cast_to_distributed #6524 and is not touched here; Muon avoids it by being set up in place.

8. Risk and rollback

Only code that builds Muon is affected, plus one isinstance check at boost in the Gemini and FSDP plugins. Rollback: revert this commit.

9. Docs and release

No README, docs, changelog or version.txt changes. The plugin compatibility table of distributed_optimizers.md does not list Muon yet.

Checklist

  • I have created an issue for this PR for traceability (none yet)
  • The title follows the standard format
  • I have added relevant tags
  • I have installed pre-commit. The hooks' formatters at their pinned versions (autoflake 2.3.1, isort 5.13.2, black 24.10.0) were run on the changed files and left them unchanged.
  • I have performed a self-review of my code
  • I have added thorough tests
  • I have added docstrings for the public classes and functions

🤖 Generated with Claude Code

LowLevelZeRO (stage 1 and 2) hands the inner optimizer a flat 1/N slice of
every parameter, while Newton-Schulz needs whole matrices, so Muon raised
there. As in Moonlight's distributed Muon, each rank now updates the momentum
of its own slices, the bf16 Newton-Schulz inputs are all-gathered over the
data-parallel group, and every rank orthogonalizes the whole matrices and
keeps the slice of each update it owns. The gathered input is the bf16 cast
the unsharded iteration makes anyway, so a sharded step gives the same bits
as the unsharded one. The cost is one bf16 all-gather of the Muon parameters
per step (in buckets of at most 512 MiB) and every rank orthogonalizing all
of them.

- muon.py: Muon is a DistributedOptim. LowLevelZeroPlugin and
  HybridParallelPlugin call setup_distributed on the object itself; it is
  not replaced through cast_to_distributed, so an LR scheduler built on it
  before boost keeps driving it. Tensor parallelism and shards that do not
  span the data-parallel group (expert parameters) raise.
- gemini_plugin.py, torch_fsdp_plugin.py: Muon raises NotImplementedError at
  boost. Gemini's chunks split matrices at arbitrary boundaries and FSDP
  hands over flattened parameters in one group.
- test_dist_muon.py: hand-made shards over gloo for 2 and 3 ranks against
  whole matrices, bit for bit (runs in the CPU job); LowLevelZeRO with 2
  ranks against 1 rank bit for bit (stage 1 and 2, fp32 and bf16, CPU
  offload), 4 ranks within judge_relative_bound, resume from a checkpoint
  bit for bit in fp32. A subclass that orthogonalizes each slice on its own
  must fail in both settings.
- run_cpu_unit_tests.sh: run test_dist_muon.py in the CPU job.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@yuxuan-lou
yuxuan-lou requested a review from a team as a code owner November 18, 2026 10:56
@yuxuan-lou
yuxuan-lou merged commit f5c44d9 into feat/muon-optimizer Nov 18, 2026
1 check passed
@yuxuan-lou
yuxuan-lou deleted the feat/muon-zero branch November 18, 2026 11:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant