feat(deepseek-v41): add H100-80GB recipes, Tulu3 data preparation, and coverage notes - #3962
Open
linmuchuiyang wants to merge 1 commit into
Open
linmuchuiyang wants to merge 1 commit into
linmuchuiyang wants to merge 1 commit into
Conversation
Contributor
|
/ok to test 0520f9c |
…d coverage notes Add four DeepSeek-V4.1-Flash recipes validated on 8-GPU H100-80GB nodes (100 updates each): HellaSwag full SFT on 16 nodes (GBS 128), HellaSwag LoRA on 8 nodes, Tulu3 4k CP8 on 32 nodes, and Tulu3 packed 32k CP8 on 64 nodes. Each header lists the differences from the GB200 recipe and the reason: the extra ep_shard FSDP axis over the experts, the GBS floor at dp=128, defer_fsdp_grad_sync: false for gradient accumulation on 80 GB parts, and moe.reshard_after_forward: false as the interim work-around for issue NVIDIA-NeMo#3958 until PR NVIDIA-NeMo#3961 lands. Add prepare_tulu3_data.py, which builds the DS41_DATA_ROOT Arrow layouts the Tulu3 recipes expect (4096-token padded rows and prepacked 32k THD packs) from tulu3_chat_template.jinja. Extend the model-coverage page with an H100 section: recipe table with measured memory and losses, the data-preparation entry point, the NCCL_PXN_DISABLE=1 requirement on multi-node InfiniBand (issue NVIDIA-NeMo#3959), the memory behaviour of defer_fsdp_grad_sync, node-local JIT caches, and the checkpoint size of the packed recipe. Closes NVIDIA-NeMo#3960 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: linmuchuiyang <6169320+linmuchuiyang@users.noreply.github.com>
linmuchuiyang
force-pushed
the
feat/deepseek-v41-h100-recipes
branch
from
September 21, 2026 08:42
0520f9c to
7d8ebc8
Compare
jgerh
reviewed
Sep 21, 2026
jgerh
left a comment
jgerh
left a comment
Contributor
There was a problem hiding this comment.
Completed tech pubs review of docs/model-coverage/llm/deepseek-ai/deepseek-v41-flash.mdx and provided a few comments.
Comment on lines
8
to
9
| uses a 40-layer causal encoder/decoder backbone with 384 routed experts, one | ||
| shared expert, and six selected experts per token. Its CSA2 attention shares |
Contributor
There was a problem hiding this comment.
Suggested change
| uses a 40-layer causal encoder/decoder Mixture-of-Experts (MoE) backbone with | |
| 384 routed experts, one shared expert, and six selected experts per token. Its CSA2 attention shares |
| | Recipe | Topology and Batch | Optimizer Moments and Checkpoints | | ||
| |---|---|---| | ||
| | [HellaSwag EP64](https://github.com/NVIDIA-NeMo/Automodel/blob/main/examples/llm_finetune/deepseek_v41/deepseek_v41_flash_hellaswag_ep64_16nodes.yaml) | 16 nodes × 4 GB200; EP64, owner64, TP1/PP1/CP1; global batch 64, local batch 1 | BF16 moments; checkpoint saving disabled | | ||
| | [HellaSwag EP64, H100](https://github.com/NVIDIA-NeMo/Automodel/blob/main/examples/llm_finetune/deepseek_v41/deepseek_v41_flash_hellaswag_ep64_h100_16nodes.yaml) | 16 nodes × 8 H100-80GB; EP64 with ep_shard2, owner128, TP1/PP1/CP1; global batch 128, local batch 1 | BF16 moments; checkpoint saving disabled; see [H100 80 GB Systems](#h100-80-gb-systems) | |
Contributor
There was a problem hiding this comment.
Suggested change
| | [HellaSwag EP64, H100](https://github.com/NVIDIA-NeMo/Automodel/blob/main/examples/llm_finetune/deepseek_v41/deepseek_v41_flash_hellaswag_ep64_h100_16nodes.yaml) | 16 nodes × 8 H100-80GB; EP64 with ep_shard2, owner128, TP1/PP1/CP1; global batch 128, local batch 1 | BF16 moments; checkpoint saving disabled; see [H100 80 GB Systems](#h100-80-gb-systems) | | |
| | [HellaSwag EP64, H100](https://github.com/NVIDIA-NeMo/Automodel/blob/main/examples/llm_finetune/deepseek_v41/deepseek_v41_flash_hellaswag_ep64_h100_16nodes.yaml) | 16 nodes × 8 H100 80 GB; EP64 with ep_shard2, owner128, TP1/PP1/CP1; global batch 128, local batch 1 | BF16 moments; checkpoint saving disabled; see [H100 80 GB Systems](#h100-80-gb-systems) | |
Comment on lines
66
to
67
| See the [Slurm launcher guide](/job-launchers/slurm-cluster) and | ||
| [installation guide](/get-started/installation) for setup. Compilation caches |
Contributor
There was a problem hiding this comment.
Suggested change
| See the [Slurm Launcher Guide](/job-launchers/slurm-cluster) and | |
| [Installation Guide](/get-started/installation) for setup. Compilation caches |
| `ep_size: 64` on 128 or more GPUs the mesh derives an additional `ep_shard` | ||
| FSDP axis over the routed experts, which halves the per-rank expert state. | ||
| The variants below were validated on 8-GPU H100 nodes with InfiniBand, 100 | ||
| updates each, with the NGC `nemo-automodel:26.08` container plus this |
Contributor
There was a problem hiding this comment.
Suggested change
| updates each, with the NGC `nemo-automodel:26.08` container plus this | |
| updates each, using the NGC `nemo-automodel:26.08` container plus this |
| updates each, with the NGC `nemo-automodel:26.08` container plus this | ||
| repository at `e2c47c5b` (torch 2.13, NCCL 2.30.5). | ||
|
|
||
| | Recipe | Topology and Batch | Overrides versus the GB200 recipe | Peak memory / result | |
Contributor
There was a problem hiding this comment.
Suggested change
| | Recipe | Topology and Batch | Overrides versus the GB200 recipe | Peak memory / result | | |
| | Recipe | Topology and Batch | Overrides Compared to the GB200 Recipe | Peak Memory / Result | |
| | Recipe | Topology and Batch | Overrides versus the GB200 recipe | Peak memory / result | | ||
| |---|---|---|---| | ||
| | [HellaSwag EP64, H100](https://github.com/NVIDIA-NeMo/Automodel/blob/main/examples/llm_finetune/deepseek_v41/deepseek_v41_flash_hellaswag_ep64_h100_16nodes.yaml) | 16 × 8 H100; EP64, ep_shard2, owner128; GBS 128, LBS 1, one microbatch | `global_batch_size: 128` | 58.4 GiB; train 1.8185, validation 1.9800 (GB200: 1.8177 / 2.0173) | | ||
| | [HellaSwag LoRA, H100](https://github.com/NVIDIA-NeMo/Automodel/blob/main/examples/llm_finetune/deepseek_v41/deepseek_v41_flash_hellaswag_lora_h100_8nodes.yaml) | 8 × 8 H100; EP32, ep_shard2, owner64; GBS 64, LBS 1, one microbatch | none | 43.9 GiB; train 2.2655, validation 2.5737 | |
Contributor
There was a problem hiding this comment.
Suggested change
| | [HellaSwag LoRA, H100](https://github.com/NVIDIA-NeMo/Automodel/blob/main/examples/llm_finetune/deepseek_v41/deepseek_v41_flash_hellaswag_lora_h100_8nodes.yaml) | 8 × 8 H100; EP32, ep_shard2, owner64; GBS 64, LBS 1, one microbatch | none | 43.9 GiB; train 2.2655, validation 2.5737 | | |
| | [HellaSwag LoRA, H100](https://github.com/NVIDIA-NeMo/Automodel/blob/main/examples/llm_finetune/deepseek_v41/deepseek_v41_flash_hellaswag_lora_h100_8nodes.yaml) | 8 × 8 H100; EP32, ep_shard2, owner64; GBS 64, LBS 1, one microbatch | None | 43.9 GiB; train 2.2655, validation 2.5737 | |
| Text inputs use unpacked `input_ids [batch, sequence]`, right padding, and | ||
| zero-based positions. The optional `labels [batch, sequence]` API computes | ||
| shifted causal cross-entropy with `-100` ignored. Attention KV and index | ||
| quantize/dequantize boundaries remain part of every forward and use |
Contributor
There was a problem hiding this comment.
Suggested change
| quantize and dequantize boundaries remain part of every forward and use |
|
|
||
| The released vision tower is enabled by the default model configuration and | ||
| disabled by the HellaSwag recipe. `DeepseekV41Processor` supports ordinary | ||
| text/image conversations and save/reload. Image inputs require unpacked |
Contributor
There was a problem hiding this comment.
Suggested change
| text and image conversations and save/reload. Image inputs require unpacked |
| `input_ids [batch, sequence]`, `pixel_values [all_patches, 3, patch_size, patch_size]`, | ||
| `image_grid_hws [images, 2]`, and `vision_token_types [batch, sequence]`. | ||
| The image encoder and projector reuse DeepSeek V4 modules; image delimiters | ||
| and visual router bias are retained. Specialized tool/reasoning formatting |
Contributor
There was a problem hiding this comment.
Suggested change
| and visual router bias are retained. Specialized tool and reasoning formatting |
| This is a measured baseline, not a strict parity pass. | ||
|
|
||
| An earlier PR commit, `ae380ea74b8417a000c1639ebd64991aafeb42b8`, completed | ||
| three resident forward/backward iterations using the now-removed custom FP32 |
Contributor
There was a problem hiding this comment.
Suggested change
| three resident forward and backward iterations using the now-removed custom FP32 |
svcnvidia-nemo-ci
added
the
waiting-on-customer
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Follow-up to #3960 (H100 guidance requested by @HuiyingLi). Adds four DeepSeek-V4.1-Flash recipes validated on 8-GPU H100-80GB nodes, a Tulu3 data-preparation script, and an H100 section on the model-coverage page.
deepseek_v41_flash_hellaswag_ep64_h100_16nodes.yamldeepseek_v41_flash_hellaswag_lora_h100_8nodes.yamldeepseek_v41_flash_tulu3_cp8_h100_32nodes.yamldeepseek_v41_flash_tulu3_packed_cp8_32k_h100_64nodes.yamlEvery recipe body is the GB200 recipe with only the keys listed in its header changed (topology-driven
global_batch_size,defer_fsdp_grad_sync: falsewhere gradient accumulation is required on 80 GB parts, andmoe.reshard_after_forward: falseas the interim work-around for #3958 until #3961 lands).ci:blocks are omitted because the runs need 16–64 eight-GPU nodes.prepare_tulu3_data.pyproduces theDS41_DATA_ROOTArrow layouts the Tulu3 recipes describe in their headers (4096-tokenpadding=max_lengthrows, and prepacked 32k THD packs viapack_dataset(..., cp_size=1, pad_to_multiple_of=2)), using the existingChatDataset,tokenize_dataset_parallelandpack_datasethelpers.The coverage page gains an "H100 80 GB Systems" section: recipe table, the data-prep entry point,
NCCL_PXN_DISABLE=1on multi-node InfiniBand (#3959), the memory behaviour ofdefer_fsdp_grad_sync, node-local JIT caches, and the checkpoint size/restore note for the packed recipe.Environment: NGC
nemo-automodel:26.08+ this repo ate2c47c5b, torch 2.13, NCCL 2.30.5, Slurm/pyxis, 16–64 nodes of 8 × H100-80GB with InfiniBand.Closes #3960
Checklist
ruff check🤖 Generated with Claude Code