Repository navigation
Conversation
Companion to deepspeedai/DeepSpeed#9131: continues training a causal LM (dense or AutoEP MoE) on WikiText-103 under BF16 and several quantization recipes, and reports validation loss, step time and peak memory. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Signed-off-by: Zhipeng Wang <7927074+PKUWZP@users.noreply.github.com>
…example from_scratch.py trains a Llama-architecture model from random initialization on all of WikiText-103 with a warmup + cosine schedule and reports final validation and test loss, so FP8 recipes can be compared with BF16 at convergence. The README gains the launch commands and two-seed reference results. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Signed-off-by: Zhipeng Wang <7927074+PKUWZP@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
training/subchannel_quantization/, the companion example for deepspeedai/DeepSpeed#9131 (proposal: deepspeedai/DeepSpeed#9132). It should not merge before that PR does.What it adds
train.py: continues training a pretrained causal LM on WikiText-103 with DeepSpeed, either as a dense model (data parallel) or as an MoE model with AutoEP.--recipeselects thequantization.subchannelsettings:bf16(baseline),e4m3_row,e4m3,int8,int8_h128. Every recipe sees the same data in the same order, and the script reports validation loss, median step time and peak memory.from_scratch.py: trains a Llama-architecture model (default 1.82B parameters) from random initialization on all of WikiText-103 with a warmup + cosine schedule, underbf16,e4m3_rowore4m3, and reports final validation and test loss. This is the script behind the full-training results in Add subchannel FP8/INT8 quantized training for nn.Linear and AutoEP experts DeepSpeed#9131.README.md: prerequisites, the recipes, launch commands for both scripts, and reference results.requirements.txt: installs DeepSpeed from the tip of Add subchannel FP8/INT8 quantized training for nn.Linear and AutoEP experts DeepSpeed#9131.Validation
Hardware: 4x NVIDIA H200, driver 595.112.07, torch 2.8.0+cu129, Triton 3.4.0, transformers 5.23.0.
e4m3_rowon Qwen1.5-MoE-A2.7B (AutoEP) and on Qwen2.5-14B (--dense --grad_checkpointing --bf16_states); both completed and wrote their result files.--denserather than a separate flag, and memory statistics came fromtorch.cudainstead of the accelerator interface.from_scratch.pybefore it was moved here. The committed version differs in a license header, a--data_diroption replacing a fixed cache path, creating the output directory, and an argument check. It was rerun after those changes for 100 steps withbf16ande4m3_rowon 2 GPUs with a small model (--layers 2 --hidden 1024 --intermediate 2048); both completed. The full runs were not repeated, and--prepwas not rerun with the new--data_diroption.requirements.txtwas not installed into a clean environment.AI assistance
This example was written with AI assistance (Claude Code) and needs human review before it leaves draft.