Repository navigation
Conversation
…xperts A new opt-in `quantization.subchannel` config section quantizes the three GEMMs of a linear layer (forward, dx, dw) independently: FP8 (e4m3/e5m2) or INT8 operands with one FP32 scale per block of the reduction dimension, optional stochastic rounding, and an optional fused butterfly Hadamard rotation inside each block. `block_size: 0` gives each operand row a single scale, which lets the FP8 GEMM apply scales once per output element and run faster than BF16 on wide layers. The GEMMs run on Triton kernels on CUDA devices of compute capability 8.9+ (8.0+ for INT8) and on a PyTorch reference implementation elsewhere. Parameters, gradients and optimizer state keep their usual precision. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Signed-off-by: Zhipeng Wang <7927074+PKUWZP@users.noreply.github.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Signed-off-by: Zhipeng Wang <7927074+PKUWZP@users.noreply.github.com>
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: db8daaef30
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| def _configure_quantization(self, model): | ||
| recipe = self._config.quantization_config.subchannel |
There was a problem hiding this comment.
Obtain proposal acceptance before proceeding with the feature
The commit message explicitly states that proposal #9132 is not yet accepted and was opened after implementation. This introduces a new user-facing quantized-training feature, but the repository requires maintainer acceptance of the proposal before implementation begins. Obtain that acceptance before proceeding with implementation review; keeping the change in draft does not satisfy the prerequisite.
AGENTS.md reference: AGENTS.md:L21-L23
Useful? React with 👍 / 👎.
Adds an opt-in
quantization.subchannelconfig section that runs the three GEMMs ofnn.Linearlayers and AutoEP grouped experts (forward,dx,dw) in FP8 or INT8, each configured independently. With one scale per operand row (block_size: 0) the FP8 path trains Qwen2.5-32B 1.21x faster per step than BF16; with block scales it is a numerics feature and slower than BF16.Proposal: #9132 (not yet accepted). Related: RFC #8777 and its draft PR #8784 (low-precision training via TorchAO HiFloat8), roadmap #8672 ("FP8 GEMM for MoE").
What it does
e4m3,e5m2,int8, ornoneper GEMM.dwthe reduction runs over tokens and blocks restart at every expert.block_size: 0, FP8 only): one scale per operand row, per expert indw. Scales are applied once per output element, so the reduction is a plain FP8 tensor-core loop.dxanddw.hadamard_size): rotates each chunk of a block in registers before quantizing; the two operands' rotations cancel in the product.lm_headand MoE routers in full precision.Implementation
deepspeed/ops/triton_ops/subchannel_quant.py: Triton quantization and GEMM kernels (block-scaled and row-scaled, rows-grouped and reduction-grouped).deepspeed/runtime/quantization/: config models, a PyTorch reference implementation (the test oracle, and the fallback on devices without the kernels), one autograd function shared by dense and grouped layers, and module conversion.engine.py(+14, one call at init),runtime/config.py(+6),ep_experts.py(+31, agrouped_mm_fnhook),abstract_accelerator.py(+6, a concrete methodsupported_subchannel_quantization_formats()returning()by default),cuda_accelerator.py(+14).deepspeed/andaccelerator/, most of it the standalone kernel module.Results
Hardware: 4x NVIDIA H200 (143 GiB), driver 595.112.07, torch 2.8.0+cu129, Triton 3.4.0, transformers 5.23.0. The full-training runs below use two seeds; every other table is a single run with a single seed.
Full training to convergence, two seeds
A 1.82B-parameter Llama-architecture model (8 layers, hidden 4096, MLP 11008, GPT-2 tokenizer) trained from random initialization on all of WikiText-103 (118M tokens) for 4 epochs: 28,788 steps, sequence length 2048, batch 4 per GPU on 2 GPUs, ZeRO-2, AdamW (peak lr 2e-4, 1000 warmup steps, cosine decay to zero, weight decay 0.1), gradient clipping 1.0, no gradient checkpointing. Both FP8 recipes use nearest rounding forward and stochastic rounding for dx and dw. Within a seed, every recipe starts from the same weights and sees the same data order. Two runs shared the machine at a time, 2 GPUs each.
Dense models: Qwen2.5-14B and Qwen2.5-32B
100 steps of continued pretraining on WikiText-103, 8192 tokens per GPU per step (batch 4 x sequence 2048), gradient checkpointing, BF16 master weights and optimizer state, AdamW lr 1e-5. 14B under ZeRO-2, 32B under ZeRO-3.
MoE models: Qwen1.5-MoE-A2.7B and Qwen3-30B-A3B
300 steps, AutoEP with
autoep_size=4, ZeRO-2, 2048 tokens per GPU per step.The INT8 rows were measured before the stochastic-rounding kernel was made cheaper and were not rerun.
Single layer step (forward + dx + dw), one H200
What the results show, and do not
Tests
dxanddwagainst FP64 matmuls on both the kernels and the PyTorch reference, check stochastic rounding is unbiased, check module conversion, and run distributed training (world_size=2): dense under ZeRO 1/2/3 and AutoEP experts under ZeRO-2.test_ds_config_dict.pyand areCUDAMismatchExceptionfrom the FusedAdam JIT build on this machine (system CUDA 13.2 vs torch built for 12.9). They are environmental; I did not confirm them on an unmodified checkout.TestFusedGateUpandTestDeferredExpertWeightGradwere excluded because they hang on this machine on unmodifiedmasteras well:torch._grouped_mmin its 2D x 2D form does not return on torch 2.8 when an expert has no tokens. That is a separate pre-existing problem and is not addressed here.ci/tests_fetcher.py --base origin/masterselects the fulltests/unit/v1suite for this diff.Compatibility
nn.Linear(unit tests; 2 and 3 on real models); 2 for AutoEPtorch_autocasttorch.compile, checkpoint save/load round-tripAutoEP's
gate_up_impl="fused"andoverlap_weight_gradare rejected for quantized experts. No existing config key, checkpoint format or public API changes; the feature is off by default.Process gaps
CONTRIBUTING.mdasks for an accepted proposal before a feature PR; [RFC] Subchannel FP8/INT8 quantized training for nn.Linear and AutoEP experts (Triton, CUDA) #9132 was opened after this PR. This PR stays a draft until that discussion concludes.docs/_pages/config-json.mdand a tutorial,docs/_tutorials/subchannel-quantized-training.md. The docs site was not built locally.quantizationconfig section.AI assistance
This change was written with AI assistance (Claude Code), including the kernels, tests, benchmarks and this description. I reviewed the PR line-by-line and approved the changes.