Skip to content

[optimizer] add OrScale and OrScale-LM - #6571

Open
yuxuan-lou wants to merge 1 commit into
feat/muon-optimizerfrom
feat/orscale
Open

yuxuan-lou wants to merge 1 commit into
feat/muon-optimizerfrom
feat/orscale

Conversation

@yuxuan-lou

@yuxuan-lou yuxuan-lou commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

1. Related issues

No issue yet. This is the third step of the Muon line: OrScale and OrScale-LM.

It is stacked on #6521: the base branch is feat/muon-optimizer, so the diff shows this step only. The step it builds on, Muon under LowLevelZeRO (#6570), has been merged into that branch.

The PR gates (ci_layer_1_pr.yml, ci_layer_2_gpu_pr.yml, fork_pr_checks.yml) run for pull requests to main only, so they do not run here until this PR is retargeted to main after #6521 merges. Section 6 has the local runs on a merge with main.

2. Background

OrScale (arXiv:2605.07815, official code NUS-HPC-AI-Lab/OrScale, MIT) scales Muon's orthogonalized update by a layer-wise trust ratio and couples the weight decay into the scaled step. For a matrix W with Nesterov momentum M, Q = PolarExpress(M) and a shape scale s:

r = clip(||W||_F / (c * ||wd * W + s * Q||_F + eps), r_min, r_max)
W <- W - lr * r * (wd * W + s * Q)
  • OrScale: s = 1, c = 1.
  • OrScale-LM: s = 0.2·√max(m, n) (Moonlight's), and c is set per layer at its first step so that r = 1 there.

The official implementation uses two optimizers (OrScale for the matrices, AdamW for the rest) and its own ZeRO-1 wrapper. A Booster takes one optimizer, and LowLevelZeRO shards every parameter into flat slices.

3. Changes

  • colossalai/nn/optimizer/orscale.py:
    • OrScale subclasses Muon, so it has the same built-in AdamW branch, takes muon_param_groups(model), and runs under TorchDDPPlugin and LowLevelZeroPlugin. variant is "orscale" or "orscale_lm".
    • The arithmetic follows the official code at commit a45fc2b statement by statement: accumulated float32 momentum (not Muon's moving average), bf16 Polar Express orthogonalization (polar_express), float32 norms.
    • OrScale-LM's constant c_denom is calibrated at each layer's first step and kept in the optimizer state as a Python float. A 0-dim tensor would not do: LowLevelZeRO's checkpoints treat every tensor in the state as sharded.
    • Under LowLevelZeRO the matrices are orthogonalized whole, as Muon does. For the trust ratio each rank sums the squares over its slice and one all-reduce per bucket (two numbers per matrix) gives the two norms.
  • colossalai/nn/optimizer/muon.py: the orthogonalization and the application of the update become methods a subclass overrides. No change in behavior.
  • colossalai/booster/plugin/gemini_plugin.py, torch_fsdp_plugin.py: the error at boost names the optimizer class, so it reads correctly for OrScale too.
  • tests/test_optimizer/orscale_reference.py: an excerpt of the official code, cut down to the two variants of the paper, with its MIT license notice. It is the reference of the tests.
  • tests/test_optimizer/test_orscale.py, using tests/kit/numerics:
    • polar_express against the official orthogonalization, with the official iteration in float64 as the truth; Muon's Newton–Schulz must fail the same check. Independently of the official code, the result must be within 10% of U Vᵀ from an SVD.
    • 3 steps of both variants against the official optimizer with check_optimizer, on CPU and CUDA: with and without weight decay, calibrated and given c_denom. The weights are scaled so that the trust ratio is clipped from below, unclipped, and clipped from above, plus one zero-initialized layer.
    • One optimizer (OrScale + AdamW branch) against the official pair of optimizers.
    • Negative checks that must fail: lr × 1.01, momentum 0.94, weight decay 0.09, r_min 0.6, r_max 1.4, eps 1e-3, 4 iterations, the other variant, a c_denom off by 2%, weight decay decoupled from the trust ratio, and Muon's moving-average momentum.
    • OrScale-LM's first step equals Moonlight's for every layer, whatever the scale of its weights.
    • Parameter grouping, the state_dict round trip (the calibrated constants included), invalid arguments, and 2-GPU TorchDDPPlugin against one device.
  • tests/test_optimizer/test_dist_muon.py:
    • OrScale on hand-made ZeRO shards over gloo, 2 and 3 ranks, against whole matrices: momentum bit for bit, weights and c_denom within the float32 relative bound.
    • LowLevelZeroPlugin with 2 ranks against 1 after one step, within the float32 bound on the optimizer state (stage 1 fp32, stage 2 bf16, stage 2 bf16 with CPU offload); 4 ranks over 3 steps within the bf16 bound.
    • Negative check that must fail in both settings: a trust ratio taken from the norms of the local slice.
    • Resume in fp32 with sharded and unsharded optimizer checkpoints, bit for bit, which covers the float in the state.
  • .github/scripts/ci/run_cpu_unit_tests.sh: runs test_orscale.py in the CPU job.

Not included: the ablation variants of the official code, its flattening of convolution weights (matrix groups take 2D parameters only), torch.compile of the orthogonalization, Gemini / FSDP / tensor parallelism, convergence experiments, docs.

4. User and compatibility impact

  • Python API: adds colossalai.nn.optimizer.OrScale. Nothing existing changes behavior; the Gemini / FSDP error message for Muon now starts with the class name.
  • Defaults: lr=0.02, weight_decay=0, momentum=0.95, trust ratio clipped to [0.5, 1.5], as in the official code. The default variant is "orscale", while the official class defaults to "orscale_lm". The AdamW groups share lr unless a group sets its own; the official default is a tenth of it.
  • Checkpoint format: no change. OrScale's state is momentum_buffer (float32, accumulated, so it has the official scale and not that of Muon's buffer) and, for OrScale-LM, a float c_denom per matrix.
  • Performance and memory: on top of Muon's all-gather under ZeRO, one all-reduce of two floats per matrix per bucket. On whole matrices the trust ratio is read with .item() once per matrix and step, as in the official code. No throughput numbers in this PR.
  • Model accuracy of existing code, Python / PyTorch / CUDA versions, C++/CUDA kernels, Docker images, existing training scripts: none. OrScale runs on torch>=2.5; all of test_orscale.py except the DDP test runs on torch 2.5.1 in the CPU job.
  • CI time: about 40 s more in the CPU job; about 4.5 minutes more with 4 GPUs for tests/test_optimizer.

5. Test environment

  • OS: Ubuntu 22.04.1 LTS host, CI image torch2.13-cu130-tf5.17.0
  • Python: 3.10.12
  • GPU: 4× NVIDIA H20
  • NVIDIA driver: 595.71.05 (nvidia-smi CUDA Version 13.2)
  • PyTorch: 2.13.0+cu130, torch.version.cuda 13.0
  • CUDA toolkit / nvcc: 13.0
  • Transformers: 5.17.0
  • flash-attn: not installed
  • ColossalAI: this branch, 569cbdf, merged with main d3916b5 (no conflicts)
  • Also the CPU job's environment: torch 2.5.1+cpu, Python 3.10.12

6. Test commands and results

CUDA_VISIBLE_DEVICES= python -m pytest -q tests/test_optimizer/test_orscale.py tests/test_optimizer/test_muon.py tests/test_optimizer/test_dist_muon.py
  • torch 2.13: 71 passed, 0 failed, 23 skipped (they need GPUs), exit code 0.
  • torch 2.5.1: 53 passed, 0 failed, 41 skipped (18 compare against torch.optim.Muon, 23 need GPUs), exit code 0.
CUDA_VISIBLE_DEVICES=<4 GPUs> python -m pytest -q tests/test_optimizer/test_orscale.py tests/test_optimizer/test_muon.py tests/test_optimizer/test_dist_muon.py
  • 115 passed, 0 failed, 0 skipped, exit code 0.
bash .github/scripts/ci/run_cpu_unit_tests.sh
  • torch 2.13: 354 passed, 0 failed, 620 skipped, exit code 0.
  • torch 2.5.1: 326 passed, 0 failed, 648 skipped, exit code 0.
  • Both were run before the last edits, which touched two GPU-only tests and docstrings; the three optimizer files were rerun afterwards (the first command).

Details:

  • Against the full official code (the three source files of a45fc2b loaded as they are, ORSCALE_DISABLE_TORCH_COMPILE=1): the orthogonalization for 7 shapes up to 3072×768, and 5 steps of the optimizer (weights, momentum, c_denom) for both variants, weight decay 0 and 0.1, calibrated and given c_denom, both clipping ranges, parameters in float32, float64 and bfloat16. The official code, the excerpt and this implementation give the same bits everywhere, on CPU and CUDA. This check is a one-off script, not part of the test suite.
  • The official code compiles its orthogonalization with torch.compile when CUDA is available. Compiled, it differs from this implementation by 1.3–3.2% (relative Frobenius error of Q), which is the size of the bf16 error itself: against the float64 iteration the compiled official function is off by 0.9–2.1% and this one by 1.1–2.6%.
  • With Muon's moving-average momentum instead of the accumulated one, each step's update differs by 2.8–5.1%, again through the bf16 rounding of the Polar Express input. That is why the momentum follows the official form.
  • Shards against whole matrices over gloo: weights within 3.5e-8 and c_denom within 9.9e-8 (bound 1.26e-6). With the trust ratio taken from the local slice the weights are 34–1011× outside the bound.
  • LowLevelZeRO, 2 ranks against 1 after one step: master weights within 3.1e-9, c_denom within 7.1e-8. With the local trust ratio the two matrices are 31× and 4.6× outside the bound. 4 ranks over 3 steps: updates within 5.6e-7 (bf16 bound 8.29e-2).
  • No xfail or --ignore; the only skips are the torch-version and GPU-count conditions above.
  • Logs are kept on the development server, not attached.

7. Known issues

  • The CI gates do not run on this PR while its base is not main (section 1).
  • Under LowLevelZeRO a sharded step is not bit for bit the unsharded one, unlike Muon's: the two norms are summed slice by slice, which rounds differently. The trust ratio differs by float32 rounding.
  • The float32 bound is applied to the weights, not to their change: the difference of two float32 weights does not resolve one step to 1e-6 of itself. Over several steps under the plugin, the bf16 orthogonalization can turn that rounding into bf16 rounding, so only the one-step comparison uses the float32 bound and the 3-step one uses the bf16 bound.
  • In the LowLevelZeRO negative check the smaller matrix is only 4.6× outside the bound after one warm-up step, so the test requires the larger one to fail and allows the smaller one to.
  • The LowLevelZeRO and DDP tests need 2 GPUs, the 4-rank test 4. They run in the GPU batch that covers tests/test_optimizer, not in the PR gate.

8. Risk and rollback

Only code that builds OrScale is affected. The change to muon.py moves code into overridable methods; test_muon.py and the Muon cases of test_dist_muon.py pass unchanged. Rollback: revert this commit.

9. Docs and release

No README, docs, changelog or version.txt changes. The plugin compatibility table of distributed_optimizers.md does not list Muon or OrScale yet.

Checklist

  • I have created an issue for this PR for traceability (none yet)
  • The title follows the standard format
  • I have added relevant tags
  • I have installed pre-commit. The hooks' formatters at their pinned versions (autoflake 2.3.1, isort 5.13.2, black 24.10.0) were run on the changed files and left them unchanged.
  • I have performed a self-review of my code
  • I have added thorough tests
  • I have added docstrings for the public classes and functions

🤖 Generated with Claude Code

Add colossalai.nn.optimizer.OrScale: Muon's orthogonalized update scaled by
a layer-wise trust ratio, with the weight decay coupled into the scaled step
(arXiv:2605.07815). It subclasses Muon, so it has the same built-in AdamW
branch for non-matrix parameters and runs under TorchDDP and LowLevelZeRO.
The arithmetic follows the official implementation
(NUS-HPC-AI-Lab/OrScale @ a45fc2b) statement by statement: accumulated
float32 momentum, bf16 Polar Express orthogonalization, float32 norms. Run
eagerly, the official code and this one give the same bits on CPU and CUDA.
The official code compiles its orthogonalization with torch.compile on CUDA,
which alone moves the result by 1-3%, as much as bf16 moves it from a
float64 iteration.

- orscale.py: OrScale with variant "orscale" or "orscale_lm", and
  polar_express. OrScale-LM's per-layer constant is calibrated at the first
  step and kept in the state as a float, which LowLevelZeRO's checkpoints
  carry. Under ZeRO each rank sums the squares of its slice and one
  all-reduce per bucket gives the two norms of the trust ratio, so a sharded
  step equals the unsharded one up to float32 rounding, not bit for bit.
  The ablation variants and the flattening of convolution weights of the
  official code are left out.
- muon.py: the orthogonalization and the application of the update are
  methods a subclass overrides; no change in behavior.
- gemini_plugin.py, torch_fsdp_plugin.py: the error names the optimizer
  class.
- orscale_reference.py: excerpt of the official code (MIT) as the reference
  of the tests.
- test_orscale.py: Polar Express and 3 steps of both variants against the
  reference under check_optimizer, with clipped and unclipped trust ratios
  and a zero-initialized layer; one optimizer against the official pair
  (OrScale + AdamW); wrong hyperparameters, decoupled weight decay and
  Muon's averaged momentum must fail. Grouping, state_dict round trip, error
  cases, 2-GPU TorchDDP.
- test_dist_muon.py: OrScale on hand-made shards for 2 and 3 ranks and
  under LowLevelZeRO (2 ranks against 1 after one step within the float32
  bound, 4 ranks within the bf16 bound, resume bit for bit with sharded and
  unsharded optimizer checkpoints). A trust ratio taken from the local slice
  must fail.
- run_cpu_unit_tests.sh: run test_orscale.py in the CPU job.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@yuxuan-lou
yuxuan-lou requested a review from a team as a code owner November 18, 2026 10:57
Base automatically changed from feat/muon-zero to feat/muon-optimizer November 18, 2026 11:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant