Skip to content

Add subchannel FP8/INT8 quantized training for nn.Linear and AutoEP experts - #9131

Open
PKUWZP wants to merge 3 commits into
masterfrom
subchannel-fp8-int8-training
Open

PKUWZP wants to merge 3 commits into
masterfrom
subchannel-fp8-int8-training

Conversation

@PKUWZP

@PKUWZP PKUWZP commented Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator

Adds an opt-in quantization.subchannel config section that runs the three GEMMs of nn.Linear layers and AutoEP grouped experts (forward, dx, dw) in FP8 or INT8, each configured independently. With one scale per operand row (block_size: 0) the FP8 path trains Qwen2.5-32B 1.21x faster per step than BF16; with block scales it is a numerics feature and slower than BF16.

Proposal: #9132 (not yet accepted). Related: RFC #8777 and its draft PR #8784 (low-precision training via TorchAO HiFloat8), roadmap #8672 ("FP8 GEMM for MoE").

What it does

"quantization": {
  "subchannel": {
    "enabled": true,
    "forward": {"format": "e4m3", "block_size": 128, "hadamard_size": 0, "stochastic_rounding": false},
    "dx": {"format": "e4m3", "block_size": 128, "hadamard_size": 0, "stochastic_rounding": true},
    "dw": {"format": "e4m3", "block_size": 128, "hadamard_size": 0, "stochastic_rounding": true}
  }
}
  • Formats: e4m3, e5m2, int8, or none per GEMM.
  • Subchannel scaling: both operands are quantized along the GEMM's reduction dimension in blocks of 32/64/128/256 elements with one FP32 scale each. In dw the reduction runs over tokens and blocks restart at every expert.
  • Row scaling (block_size: 0, FP8 only): one scale per operand row, per expert in dw. Scales are applied once per output element, so the reduction is a plain FP8 tensor-core loop.
  • Stochastic rounding: unbiased, default on for dx and dw.
  • Fused butterfly Hadamard (hadamard_size): rotates each chunk of a block in registers before quantizing; the two operands' rotations cancel in the product.
  • Parameters, gradients, optimizer state and module outputs keep their usual precision. Only GEMM operands are quantized, on the fly, so there is no memory saving.
  • Default excludes keep lm_head and MoE routers in full precision.

Implementation

  • deepspeed/ops/triton_ops/subchannel_quant.py: Triton quantization and GEMM kernels (block-scaled and row-scaled, rows-grouped and reduction-grouped).
  • deepspeed/runtime/quantization/: config models, a PyTorch reference implementation (the test oracle, and the fallback on devices without the kernels), one autograd function shared by dense and grouped layers, and module conversion.
  • Core-file wiring is small: engine.py (+14, one call at init), runtime/config.py (+6), ep_experts.py (+31, a grouped_mm_fn hook), abstract_accelerator.py (+6, a concrete method supported_subchannel_quantization_formats() returning () by default), cuda_accelerator.py (+14).
  • About 1,130 added lines under deepspeed/ and accelerator/, most of it the standalone kernel module.

Results

Hardware: 4x NVIDIA H200 (143 GiB), driver 595.112.07, torch 2.8.0+cu129, Triton 3.4.0, transformers 5.23.0. The full-training runs below use two seeds; every other table is a single run with a single seed.

Full training to convergence, two seeds

A 1.82B-parameter Llama-architecture model (8 layers, hidden 4096, MLP 11008, GPT-2 tokenizer) trained from random initialization on all of WikiText-103 (118M tokens) for 4 epochs: 28,788 steps, sequence length 2048, batch 4 per GPU on 2 GPUs, ZeRO-2, AdamW (peak lr 2e-4, 1000 warmup steps, cosine decay to zero, weight decay 0.1), gradient clipping 1.0, no gradient checkpointing. Both FP8 recipes use nearest rounding forward and stochastic rounding for dx and dw. Within a seed, every recipe starts from the same weights and sees the same data order. Two runs shared the machine at a time, 2 GPUs each.

Recipe Seed Final val loss Test loss Test ppl Step p50 Peak memory
bf16 0 2.9597 2.9761 19.61 323 ms 29.9 GiB
e4m3 row-scaled 0 2.9557 2.9747 19.58 307 ms 29.9 GiB
e4m3 block 128 0 2.9518 2.9738 19.57 386 ms 29.9 GiB
bf16 1 2.9650 2.9860 19.81 326 ms 29.9 GiB
e4m3 row-scaled 1 2.9631 2.9828 19.74 305 ms 29.9 GiB
e4m3 block 128 1 2.9545 2.9792 19.67 386 ms 29.9 GiB
  • The two BF16 seeds differ by 0.010 in test loss; each FP8 run is within 0.007 of BF16 at the same seed. FP8 is marginally lower in all four comparisons, which with two seeds shows it is not worse, not that it is better.
  • The validation curves track each other for the whole run, with no divergence or instability. Validation loss was still falling at the end, so best and final validation loss coincide.
  • Row-scaled FP8 is about 5% faster than BF16 on this model, block-scaled about 19% slower. Peak memory is identical.
  • Not covered at this scale: AutoEP/MoE experts, INT8, the Hadamard rotation, a second architecture or dataset.

Dense models: Qwen2.5-14B and Qwen2.5-32B

100 steps of continued pretraining on WikiText-103, 8192 tokens per GPU per step (batch 4 x sequence 2048), gradient checkpointing, BF16 master weights and optimizer state, AdamW lr 1e-5. 14B under ZeRO-2, 32B under ZeRO-3.

Model Recipe Val loss @0 Val loss @100 Step p50 vs BF16 Peak memory
Qwen2.5-14B bf16 1.7576 1.6783 2145 ms 1.00x 66.2 GiB
e4m3, row-scaled 1.7622 1.6839 1965 ms 1.09x 66.2 GiB
e4m3, block 128 1.7611 1.6816 2896 ms 0.74x 66.2 GiB
Qwen2.5-32B bf16 1.7340 1.6561 4594 ms 1.00x 82.2 GiB
e4m3, row-scaled 1.7436 1.6604 3801 ms 1.21x 82.2 GiB
e4m3, block 128 1.7356 1.6584 6175 ms 0.74x 82.2 GiB

MoE models: Qwen1.5-MoE-A2.7B and Qwen3-30B-A3B

300 steps, AutoEP with autoep_size=4, ZeRO-2, 2048 tokens per GPU per step.

Model Recipe Val loss @300 Step p50 Peak memory
Qwen1.5-MoE-A2.7B bf16 1.8900 341 ms 64.5 GiB
e4m3, block 128 1.8923 500 ms 64.5 GiB
e4m3, row-scaled 1.8941 518 ms 64.5 GiB
int8, block 128 1.8903 522 ms 64.6 GiB
int8, block 128, Hadamard 128 1.8902 502 ms 64.6 GiB
Qwen3-30B-A3B bf16 1.9084 622 ms 91.8 GiB
e4m3, block 128 1.9118 789 ms 91.8 GiB
e4m3, row-scaled 1.9116 947 ms 91.8 GiB
int8, block 128 1.9090 841 ms 91.8 GiB
int8, block 128, Hadamard 128 1.9084 927 ms 91.8 GiB

The INT8 rows were measured before the stochastic-rounding kernel was made cheaper and were not rerun.

Single layer step (forward + dx + dw), one H200

Shape (tokens x in -> out) bf16 e4m3 block 128 e4m3 row-scaled Row vs BF16
32768 x 8192 -> 8192 21.2 ms 29.6 ms 15.3 ms 1.39x
16384 x 8192 -> 8192 9.7 ms 14.8 ms 7.3 ms 1.34x
8192 x 4096 -> 14336 3.9 ms 6.6 ms 3.4 ms 1.16x
8192 x 4096 -> 4096 1.09 ms 2.01 ms 1.39 ms 0.78x
8192 x 2048 -> 2048 0.30 ms 1.04 ms 1.28 ms 0.23x
16 experts, 32768 x 2048, inter 768 2.4 ms 4.6 ms 4.8 ms 0.51x

What the results show, and do not

  • Row-scaled FP8 is faster than BF16 only on wide layers with many tokens. It is slower on both MoE models and on layers 4096 wide or narrower, where the per-call quantization passes and kernel-launch overhead outweigh the faster GEMM.
  • Block-scaled GEMMs are slower than BF16 everywhere: scales have to be applied between tensor-core dot products.
  • In full training from scratch, both FP8 modes reach the same validation and test loss as BF16, within seed-to-seed variation.
  • In the short continued-pretraining runs, validation loss stays within 0.006 of BF16. Row scaling costs about twice the loss gap of block scaling on the dense models.
  • Stochastic rounding and the Hadamard rotation made no measurable difference to loss in these short runs. On synthetic data with outlier channels the rotation cuts INT8 GEMM error about 3.5x; it does not help FP8.
  • The pretrained-model runs are short continued pretraining at one learning rate: a consistency check only. The evidence for on-par final quality is the one full-training comparison above (one dense architecture, one dataset, two seeds); MoE experts, INT8 and the Hadamard rotation have not been trained to convergence.
  • The Qwen2.5 dense runs use gradient checkpointing, which runs each forward twice and so weights the forward GEMM more heavily than a run without it.

Tests

cd tests
pytest --forked unit/v1/runtime/test_subchannel_quantization.py                            # 49 passed (2 of the 4 H200s)
CUDA_VISIBLE_DEVICES="" pytest --forked unit/v1/runtime/test_subchannel_quantization.py    # 28 passed, 21 skipped
pytest unit/v1/moe/test_autoep_unit.py unit/v1/moe/test_autoep_expert_activation.py \
       unit/runtime/test_ds_config_model.py unit/runtime/test_ds_config_dict.py \
       -k "not TestFusedGateUp and not TestDeferredExpertWeightGrad"                        # 355 passed, 7 failed
  • The new tests compare forward, dx and dw against FP64 matmuls on both the kernels and the PyTorch reference, check stochastic rounding is unbiased, check module conversion, and run distributed training (world_size=2): dense under ZeRO 1/2/3 and AutoEP experts under ZeRO-2.
  • The 7 failures are all in test_ds_config_dict.py and are CUDAMismatchException from the FusedAdam JIT build on this machine (system CUDA 13.2 vs torch built for 12.9). They are environmental; I did not confirm them on an unmodified checkout.
  • TestFusedGateUp and TestDeferredExpertWeightGrad were excluded because they hang on this machine on unmodified master as well: torch._grouped_mm in its 2D x 2D form does not return on torch 2.8 when an expert has no tokens. That is a separate pre-existing problem and is not addressed here.
  • ci/tests_fetcher.py --base origin/master selects the full tests/unit/v1 suite for this diff.

Compatibility

Axis Validated Not validated
ZeRO stage 1, 2, 3 for nn.Linear (unit tests; 2 and 3 on real models); 2 for AutoEP 0; ZeRO-3 with AutoEP
Precision bf16 fp32, fp16, torch_autocast
Offload none CPU and NVMe offload
Parallelism data parallel, AutoEP AutoTP and PipelineModule are rejected at init; sequence parallelism untested
Other gradient checkpointing, gradient clipping gradient accumulation, torch.compile, checkpoint save/load round-trip
Accelerators CUDA sm90 (H200) every other device. Non-CUDA accelerators and CUDA below sm89 use the PyTorch reference, which simulates the quantization in FP32; this was run on CPU only

AutoEP's gate_up_impl="fused" and overlap_weight_grad are rejected for quantized experts. No existing config key, checkpoint format or public API changes; the feature is off by default.

Process gaps

AI assistance

This change was written with AI assistance (Claude Code), including the kernels, tests, benchmarks and this description. I reviewed the PR line-by-line and approved the changes.

PKUWZP and others added 2 commits November 20, 2026 00:43
…xperts

A new opt-in `quantization.subchannel` config section quantizes the three
GEMMs of a linear layer (forward, dx, dw) independently: FP8 (e4m3/e5m2) or
INT8 operands with one FP32 scale per block of the reduction dimension,
optional stochastic rounding, and an optional fused butterfly Hadamard
rotation inside each block. `block_size: 0` gives each operand row a single
scale, which lets the FP8 GEMM apply scales once per output element and run
faster than BF16 on wide layers.

The GEMMs run on Triton kernels on CUDA devices of compute capability 8.9+
(8.0+ for INT8) and on a PyTorch reference implementation elsewhere.
Parameters, gradients and optimizer state keep their usual precision.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: Zhipeng Wang <7927074+PKUWZP@users.noreply.github.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: Zhipeng Wang <7927074+PKUWZP@users.noreply.github.com>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-20T18:17:49.294053Z db8daae Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: db8daaef30

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +930 to +931
def _configure_quantization(self, model):
recipe = self._config.quantization_config.subchannel

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Obtain proposal acceptance before proceeding with the feature

The commit message explicitly states that proposal #9132 is not yet accepted and was opened after implementation. This introduces a new user-facing quantized-training feature, but the repository requires maintainer acceptance of the proposal before implementation begins. Obtain that acceptance before proceeding with implementation review; keeping the change in draft does not satisfy the prerequisite.

AGENTS.md reference: AGENTS.md:L21-L23

Useful? React with 👍 / 👎.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request roadmap Roadmap direction for DeepSpeed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant