Add the NVFP4 PTQ recipe for Qwen/Qwen3.8-2.4T-A95B - #2302
Conversation
This is the recipe used to produce nvidia/Qwen3.8-2.4T-A95B-NVFP4 (https://huggingface.co/nvidia/Qwen3.8-2.4T-A95B-NVFP4). Qwen/Qwen3.8-2.4T-A95B is a `qwen3_5_moe_text` MoE -- 92 layers, 512 routed experts (top-10) plus a shared expert, with hybrid attention: gated-delta (linear-attention) layers interleaved with full-attention layers. It is transformers-native from >= 5.9 and its config ships `base_model_ep_plan`, so no ModelOpt plugin is needed. The recipe applies: routed experts NVFP4 (MSE-searched static weight scales, dynamic input scales) self-attention FP8 (W8A8, all projections) linear-attention FP8 (W8A8, the gated-delta projections) KV cache FP8 (cast mode) everything else BF16 -- including MTP, which is left unquantized Quantizing the gated-delta projections is the part worth calling out. The conv1d and the norms carry no Linear quantizer, so the recurrent state path itself is never quantized -- only the projections around it are. That was validated rather than assumed: the exported checkpoint was evaluated against the BF16 baseline on GPQA, AA-LCR, SciCode, IFBench and Terminal-Bench 2.1, with no meaningful accuracy regression on any of them. One loading detail is documented in the header because it affects what the scales mean: the source checkpoint ships as native block-FP8 (`quant_method=fp8`, `weight_block_size [128, 128]`, dynamic activations), and the loader dequantizes it to BF16 before quantizers are inserted -- so the calibrated scales are against BF16 weights, not against the shipped FP8. Filed under modelopt_recipes/models/ per the split introduced in #2219 (per-model_type recipes vs model-hub checkpoint recipes); this one targets a published checkpoint. Signed-off-by: Shengliang Xu <shengliangx@nvidia.com>
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (1)
Included review availability: Your plan provides up to 12 included reviews per hour; 11 remain after this review. 📝 WalkthroughWalkthroughAdds a Qwen3.8-2.4T-A95B PTQ recipe and documents its mixed-precision layout. Routed experts use NVFP4, attention projections and KV caching use FP8, and remaining components use BF16. ChangesQwen PTQ recipe
Estimated code review effort: 2 (Simple) | ~10 minutes Merge Risk: ⚪ Minimal · up to This PR adds a localized model recipe and related documentation without changing existing behavior; no actionable merge-blocking risk remains beyond normal checks and review. Suggested reviewers: 🚥 Pre-merge checks | ✅ 6✅ Passed checks (6 passed)
Full details: Docstring CoverageExplanation No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0 files. (1 skipped: 1 unsupported.) Full details: Security Anti-PatternsExplanation PASS: The net pull-request diff from the merged main parent contains only one YAML recipe and one Markdown documentation change. It contains no modelopt package or examples Python changes, no new dependencies, and none of the specified security-sensitive patterns. Therefore, no listed security anti-pattern was introduced. ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
4777e9f to
187cc3d
Compare
Add a checkpoint-mirror entry for models/Qwen/Qwen3.8-2.4T-A95B/ptq/nvfp4_experts_mse-fp8_self_attn-fp8_linear_attn-kv_fp8_cast to modelopt_recipes/ptq.md so the new recipe is documented in the PTQ recipe guide, satisfying tests/unit/recipe/test_recipe_docs.py::test_every_model_specific_ptq_dir_is_mentioned. Signed-off-by: Shengliang Xu <shengliangx@nvidia.com>
187cc3d to
6aa0a33
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #2302 +/- ##
==========================================
- Coverage 79.05% 78.58% -0.48%
==========================================
Files 525 525
Lines 61109 61109
==========================================
- Hits 48311 48021 -290
- Misses 12798 13088 +290
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
1 participant
What does this PR do?
Type of change: new feature (model recipe)
Adds the NVFP4 PTQ recipe for Qwen/Qwen3.8-2.4T-A95B — the recipe used to produce
nvidia/Qwen3.8-2.4T-A95B-NVFP4.
Qwen/Qwen3.8-2.4T-A95B is a
qwen3_5_moe_textMoE: 92 layers, 512 routed experts (top-10)plus a shared expert, with hybrid attention — gated-delta (linear-attention) layers
interleaved with full-attention layers. It is transformers-native from >= 5.9 and its config
ships
base_model_ep_plan, so no ModelOpt plugin is required.The recipe applies:
Two things are documented in the file header because they affect how the recipe should be
read:
Linear quantizer, so the recurrent state path itself is never quantized — only the
projections around it (
in_proj_qkv/in_proj_z/in_proj_a/in_proj_b,out_proj).quant_method=fp8,weight_block_size [128,128],dynamic activations). The loader dequantizes it to BF16 before quantizers are inserted, so
the calibrated scales are against BF16 weights, not against the shipped FP8.
Filed under
modelopt_recipes/models/per the split introduced in #2219 (per-model_typerecipes vs model-hub checkpoint recipes); this one targets a published checkpoint, alongside
deepseek-ai/DeepSeek-V4-Pro-0813and the Nemotron-3 entries.Usage
Testing
The exported checkpoint was evaluated against the BF16 baseline on GPQA, AA-LCR, SciCode,
IFBench and Terminal-Bench 2.1, with no meaningful accuracy regression on any of them. The
published
nvidia/Qwen3.8-2.4T-A95B-NVFP4checkpoint is the artifact this recipe produces.No new unit tests: this is a declarative recipe composed entirely of existing units
(
base_disable_all,nvfp4,nvfp4_static,fp8,kv_fp8_cast), all already covered.Before your PR is "Ready for review"
CONTRIBUTING.md: N/AAdditional Information
Model card: https://huggingface.co/nvidia/Qwen3.8-2.4T-A95B-NVFP4
Summary by CodeRabbit