Skip to content

Add a subchannel FP8/INT8 quantized training example - #1024

Draft
PKUWZP wants to merge 2 commits into
masterfrom
subchannel-quantization-example
Draft

PKUWZP wants to merge 2 commits into
masterfrom
subchannel-quantization-example

Conversation

@PKUWZP

@PKUWZP PKUWZP commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Adds training/subchannel_quantization/, the companion example for deepspeedai/DeepSpeed#9131 (proposal: deepspeedai/DeepSpeed#9132). It should not merge before that PR does.

What it adds

  • train.py: continues training a pretrained causal LM on WikiText-103 with DeepSpeed, either as a dense model (data parallel) or as an MoE model with AutoEP. --recipe selects the quantization.subchannel settings: bf16 (baseline), e4m3_row, e4m3, int8, int8_h128. Every recipe sees the same data in the same order, and the script reports validation loss, median step time and peak memory.
  • from_scratch.py: trains a Llama-architecture model (default 1.82B parameters) from random initialization on all of WikiText-103 with a warmup + cosine schedule, under bf16, e4m3_row or e4m3, and reports final validation and test loss. This is the script behind the full-training results in Add subchannel FP8/INT8 quantized training for nn.Linear and AutoEP experts DeepSpeed#9131.
  • README.md: prerequisites, the recipes, launch commands for both scripts, and reference results.
  • requirements.txt: installs DeepSpeed from the tip of Add subchannel FP8/INT8 quantized training for nn.Linear and AutoEP experts DeepSpeed#9131.

Validation

Hardware: 4x NVIDIA H200, driver 595.112.07, torch 2.8.0+cu129, Triton 3.4.0, transformers 5.23.0.

  • The script as committed was run for 6 steps with e4m3_row on Qwen1.5-MoE-A2.7B (AutoEP) and on Qwen2.5-14B (--dense --grad_checkpointing --bf16_states); both completed and wrote their result files.
  • The reference results in the README were produced with an earlier version of the same script. The differences are cosmetic for those runs: model paths were hard-coded, gradient checkpointing was implied by --dense rather than a separate flag, and memory statistics came from torch.cuda instead of the accelerator interface.
  • The full-training results in the README (three recipes, two seeds, 28,788 steps each) were produced with from_scratch.py before it was moved here. The committed version differs in a license header, a --data_dir option replacing a fixed cache path, creating the output directory, and an argument check. It was rerun after those changes for 100 steps with bf16 and e4m3_row on 2 GPUs with a small model (--layers 2 --hidden 1024 --intermediate 2048); both completed. The full runs were not repeated, and --prep was not rerun with the new --data_dir option.
  • The repository's pre-commit yapf hook (v0.32.0) does not install on this machine's Python, so formatting was checked with DeepSpeed's yapf hook instead; flake8 was run with this repository's arguments and passes.
  • Models were loaded from local copies. Loading by Hugging Face id, as the README commands do, was not run.
  • requirements.txt was not installed into a clean environment.

AI assistance

This example was written with AI assistance (Claude Code) and needs human review before it leaves draft.

Companion to deepspeedai/DeepSpeed#9131: continues training a causal LM (dense or AutoEP MoE) on WikiText-103 under BF16 and several quantization recipes, and reports validation loss, step time and peak memory.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: Zhipeng Wang <7927074+PKUWZP@users.noreply.github.com>
…example

from_scratch.py trains a Llama-architecture model from random initialization
on all of WikiText-103 with a warmup + cosine schedule and reports final
validation and test loss, so FP8 recipes can be compared with BF16 at
convergence. The README gains the launch commands and two-seed reference
results.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: Zhipeng Wang <7927074+PKUWZP@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant