From 161ee5d252f6dbd9be911075a9519c3a8845870a Mon Sep 17 00:00:00 2001 From: Shengliang Xu Date: Wed, 2 Sep 2026 09:59:04 -0700 Subject: [PATCH 1/3] Add the NVFP4 experts-only PTQ recipe for zai-org/GLM-5.3-Flash GLM-5.3-Flash is a `glm5_next` VLM MoE: 45 decoder layers, 288 routed experts, and hybrid attention -- KDA (linear-attention) layers interleaved with NoPE sparse-MLA layers. It needs transformers >= 5.16.1; earlier releases cannot parse the config. Routed experts -> NVFP4 W4A4 (both weight and input quantizers), KV cache -> FP8 cast, everything else BF16: dense MLP, shared experts, router gate, both attention families, the vision tower, embeddings and lm_head. The quant_cfg is deliberately minimal -- four entries. `base_disable_all` sets every quantizer to `enable: false`, and the only entries after it re-enable the routed experts and the KV cache, so nothing else needs listing: - The `*block_sparse_moe*` globs from the general experts-only recipe are dropped. They target Mixtral-style naming and match nothing here (0 keys in the exported index against 146,016 `.experts.` keys). - The shared `default_disabled_quantizers` unit is not imported. Verified against all 38,321 modules of an exported checkpoint: no module matched by `*.experts.*` is also matched by any of its patterns, so it is a no-op for this model. (`*.experts.*` needs a literal `.experts.`, so `mlp.shared_experts.*` -- 129 modules -- does not match and the shared experts stay BF16.) Both simplifications are behaviour-preserving; the resulting quantized module set is identical to the general recipe's on this architecture. Two model facts are documented in the header because they are easy to get wrong: - `layerwise.enable=false` is required rather than incidental. This is a VLM, so the decoder layers nest under `model.language_model.layers` and layerwise_calibrate cannot locate them. - The MTP head is inlined as decoder layer 45 and is NOT quantized. The config declares `num_hidden_layers: 45`, so transformers builds layers 0-44 and never instantiates layer 45; no quantizer attaches to it and it stays BF16. ptq.md gains a matching entry under "Checkpoint mirrors", required by tests/unit/recipe/test_recipe_docs.py::test_every_model_specific_ptq_dir_is_mentioned, which asserts every `models///ptq/` folder appears in the doc. The entry states plainly that this one does not mix precisions across component types the way its neighbours do -- it is the general scheme pinned to a planned checkpoint, capturing the two model facts above. All five checks in that test file pass. Signed-off-by: Shengliang Xu --- .../ptq/nvfp4_experts_only-kv_fp8_cast.yaml | 72 +++++++++++++++++++ modelopt_recipes/ptq.md | 17 +++++ 2 files changed, 89 insertions(+) create mode 100644 modelopt_recipes/models/zai-org/GLM-5.3-Flash/ptq/nvfp4_experts_only-kv_fp8_cast.yaml diff --git a/modelopt_recipes/models/zai-org/GLM-5.3-Flash/ptq/nvfp4_experts_only-kv_fp8_cast.yaml b/modelopt_recipes/models/zai-org/GLM-5.3-Flash/ptq/nvfp4_experts_only-kv_fp8_cast.yaml new file mode 100644 index 00000000000..bfb3acc5f4f --- /dev/null +++ b/modelopt_recipes/models/zai-org/GLM-5.3-Flash/ptq/nvfp4_experts_only-kv_fp8_cast.yaml @@ -0,0 +1,72 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Recipe for zai-org/GLM-5.3-Flash: NVFP4 on the routed experts only, FP8 KV cache. +# +# GLM-5.3-Flash is a `glm5_next` VLM MoE -- 45 decoder layers, 288 routed experts, and +# HYBRID attention: KDA (linear-attention) layers interleaved with NoPE sparse-MLA +# layers. It needs transformers >= 5.16.1 (earlier releases cannot parse the config). +# +# Quantized: the routed experts only, W4A4 -- both the weight and the input quantizers +# are NVFP4 -- plus an FP8 KV cache in cast mode. Everything else stays BF16: the dense +# MLP, shared experts, the router gate, both attention families, the vision tower, +# embeddings and lm_head. +# +# The quant_cfg is deliberately minimal. `base_disable_all` sets every quantizer to +# `enable: false`, and the only entries after it re-enable the routed experts and the KV +# cache -- so nothing else needs listing. In particular the shared `default_disabled_quantizers` +# unit is not imported: every pattern it turns off (router gates, `lm_head`, the linear-attn +# `conv1d` and `in_proj_a`/`in_proj_b`, ...) is already off and is not matched by +# `*.experts.*`. Note `*.experts.*` requires a literal `.experts.`, so `mlp.shared_experts.*` +# does not match it and the shared experts stay BF16. +# +# Two model-specific notes: +# +# * `layerwise.enable=false` is required, not incidental. This is a VLM, so the decoder +# layers nest under `model.language_model.layers` and layerwise_calibrate cannot find +# them. +# +# * The MTP head is inlined as decoder layer 45 and is NOT quantized. The config declares +# `num_hidden_layers: 45`, so transformers builds layers 0-44 and never instantiates +# layer 45; no quantizer attaches to it and it stays BF16. + +imports: + base_disable_all: configs/ptq/units/base_disable_all + nvfp4: configs/numerics/nvfp4 + kv_fp8_cast: configs/ptq/units/kv_fp8_cast + +metadata: + recipe_type: ptq + description: >- + GLM-5.3-Flash: NVFP4 (W4A4) on the routed experts only, plus an FP8 KV cache in cast + mode using constant amax; max calibration. The dense MLP, shared experts, router gate, + KDA and MLA attention, vision tower, embeddings, lm_head and the inlined MTP layer all + stay BF16. +quantize: + algorithm: + method: max + # Max calibration is fast and does not typically need checkpointing. + # layerwise.enable=false required for VLMs where the decoder layers are nested under + # `model.language_model.layers` (layerwise_calibrate can't find them otherwise). + layerwise: {enable: false} + quant_cfg: + - $import: base_disable_all + - quantizer_name: '*.experts.*weight_quantizer' + cfg: + $import: nvfp4 + - quantizer_name: '*.experts.*input_quantizer' + cfg: + $import: nvfp4 + - $import: kv_fp8_cast diff --git a/modelopt_recipes/ptq.md b/modelopt_recipes/ptq.md index 5e6b332de1c..898b775391e 100644 --- a/modelopt_recipes/ptq.md +++ b/modelopt_recipes/ptq.md @@ -406,6 +406,23 @@ checkpoint's** quant config verbatim: Q4_K/Q5_0 linears → NVFP4 W4A4 (attention q/k/v/o kept uniform so export can fuse them), the Q6_K MLP `down_proj` layers → FP8 W8A8, embeddings → NVFP4 W4A16, `lm_head` → FP8 W8A16, and the F32 tensors (conv1d, norms) → BF16. +- **`models/zai-org/GLM-5.3-Flash/ptq/nvfp4_experts_only-kv_fp8_cast`** is the + planned NVFP4 config for `zai-org/GLM-5.3-Flash`, a `glm5_next` VLM MoE with + **hybrid attention** — KDA (linear-attention) layers interleaved with NoPE + sparse-MLA layers. Routed experts → NVFP4 W4A4; KV cache → FP8 cast; + everything else BF16, including the dense MLP, shared experts, router gate, + both attention families, the vision tower and `lm_head`. Unlike the other + entries here it does **not** mix precisions across component types — it is the + general `nvfp4_experts_only-kv_fp8_cast` scheme pinned to this checkpoint, and + it captures two model facts instead: `layerwise.enable=false` is *required* + because the decoder layers nest under `model.language_model.layers` where + `layerwise_calibrate` cannot find them, and the MTP head — inlined as decoder + layer 45, which `num_hidden_layers: 45` never instantiates — is never + quantized and stays BF16. The `quant_cfg` is deliberately minimal: after + `base_disable_all` only the expert and KV entries remain, so the shared + `default_disabled_quantizers` unit is not imported (every pattern it disables + is already off and none is matched by `*.experts.*`, which needs a literal + `.experts.` and so skips `mlp.shared_experts.*`). *Why special:* unlike any general recipe, these **mix FP8 and NVFP4 across different component types — or individual layers** — and hardcode the precise From d47c4cf94c7a9d1121216d69bcb8317f20979576 Mon Sep 17 00:00:00 2001 From: Shengliang Xu Date: Wed, 2 Sep 2026 10:39:05 -0700 Subject: [PATCH 2/3] Shorten the GLM-5.3-Flash entry in ptq.md The entry ran to 17 lines against 4-9 for its neighbours in the same section, and most of the extra length was detail that belongs in the recipe header rather than the index. - Drop "planned" -- it is the NVFP4 config for this checkpoint, no qualifier needed. - Drop the `default_disabled_quantizers` explanation (glob matching, the `mlp.shared_experts.*` note, the `base_disable_all` walkthrough). The recipe header already covers why the quant_cfg is minimal; the index does not need it. - Collapse the full BF16 component list to "everything else", keeping only the MTP block at decoder layer 45, which is the non-obvious part. - Replace the laboured "unlike the other entries here it does not mix precisions" contrast with "carries the experts-only scope rather than a mixed-precision map". What a reader scanning the index needs is kept: the model and its hybrid attention, what is quantized, that MTP stays BF16, and the `layerwise.enable=false` requirement with its reason. tests/unit/recipe/test_recipe_docs.py still passes -- `zai-org/GLM-5.3-Flash` remains present, as test_every_model_specific_ptq_dir_is_mentioned requires. Signed-off-by: Shengliang Xu --- modelopt_recipes/ptq.md | 23 +++++++---------------- 1 file changed, 7 insertions(+), 16 deletions(-) diff --git a/modelopt_recipes/ptq.md b/modelopt_recipes/ptq.md index 898b775391e..25cc1510997 100644 --- a/modelopt_recipes/ptq.md +++ b/modelopt_recipes/ptq.md @@ -407,22 +407,13 @@ checkpoint's** quant config verbatim: fuse them), the Q6_K MLP `down_proj` layers → FP8 W8A8, embeddings → NVFP4 W4A16, `lm_head` → FP8 W8A16, and the F32 tensors (conv1d, norms) → BF16. - **`models/zai-org/GLM-5.3-Flash/ptq/nvfp4_experts_only-kv_fp8_cast`** is the - planned NVFP4 config for `zai-org/GLM-5.3-Flash`, a `glm5_next` VLM MoE with - **hybrid attention** — KDA (linear-attention) layers interleaved with NoPE - sparse-MLA layers. Routed experts → NVFP4 W4A4; KV cache → FP8 cast; - everything else BF16, including the dense MLP, shared experts, router gate, - both attention families, the vision tower and `lm_head`. Unlike the other - entries here it does **not** mix precisions across component types — it is the - general `nvfp4_experts_only-kv_fp8_cast` scheme pinned to this checkpoint, and - it captures two model facts instead: `layerwise.enable=false` is *required* - because the decoder layers nest under `model.language_model.layers` where - `layerwise_calibrate` cannot find them, and the MTP head — inlined as decoder - layer 45, which `num_hidden_layers: 45` never instantiates — is never - quantized and stays BF16. The `quant_cfg` is deliberately minimal: after - `base_disable_all` only the expert and KV entries remain, so the shared - `default_disabled_quantizers` unit is not imported (every pattern it disables - is already off and none is matched by `*.experts.*`, which needs a literal - `.experts.` and so skips `mlp.shared_experts.*`). + NVFP4 config for `zai-org/GLM-5.3-Flash`, a `glm5_next` VLM MoE with **hybrid + attention** — KDA (linear-attention) layers interleaved with NoPE sparse-MLA + layers. Routed experts → NVFP4 W4A4; KV cache → FP8 cast; everything else, + including the MTP block inlined as decoder layer 45, stays BF16. It carries + the experts-only scope rather than a mixed-precision map, and pins + `layerwise.enable=false`, which this VLM requires because its decoder layers + nest under `model.language_model.layers`. *Why special:* unlike any general recipe, these **mix FP8 and NVFP4 across different component types — or individual layers** — and hardcode the precise From 58dbd3b4973edc0c6e00b10758119334b04aee94 Mon Sep 17 00:00:00 2001 From: Shengliang Xu Date: Fri, 4 Sep 2026 11:50:20 -0700 Subject: [PATCH 3/3] Add the NVFP4 experts + dense-MLP PTQ recipe for zai-org/GLM-5.3-Flash Widens nvfp4_experts_only-kv_fp8_cast to cover the dense MLP as well. mlp_layer_types marks only layers 0-2 dense, so this adds 9 modules (mlp.gate_proj / up_proj / down_proj) on top of the routed experts; layers 3-44 carry mlp.experts..* instead. The vision tower reuses those leaf names across its 24 blocks, so the dense-MLP patterns also match 144 modules under model.visual.*. A trailing '*visual*' disable keeps them BF16, and it has to stay last because entries apply in order. That single rule is used instead of importing the shared default_disabled_quantizers unit: of that unit's patterns only '*visual*' changes anything for this model (the rest match no module here, or match modules base_disable_all already left off), and both spellings leave the same 74322 quantizers enabled. Also record in the experts-only recipe why it needs no such rule -- the vision tower is a dense ViT with no 'experts' in any module name, so '*.experts.*' cannot reach it. Signed-off-by: Shengliang Xu --- .../nvfp4_experts_dense_mlp-kv_fp8_cast.yaml | 105 ++++++++++++++++++ .../ptq/nvfp4_experts_only-kv_fp8_cast.yaml | 6 +- modelopt_recipes/ptq.md | 6 + 3 files changed, 116 insertions(+), 1 deletion(-) create mode 100644 modelopt_recipes/models/zai-org/GLM-5.3-Flash/ptq/nvfp4_experts_dense_mlp-kv_fp8_cast.yaml diff --git a/modelopt_recipes/models/zai-org/GLM-5.3-Flash/ptq/nvfp4_experts_dense_mlp-kv_fp8_cast.yaml b/modelopt_recipes/models/zai-org/GLM-5.3-Flash/ptq/nvfp4_experts_dense_mlp-kv_fp8_cast.yaml new file mode 100644 index 00000000000..36dc00ebe64 --- /dev/null +++ b/modelopt_recipes/models/zai-org/GLM-5.3-Flash/ptq/nvfp4_experts_dense_mlp-kv_fp8_cast.yaml @@ -0,0 +1,105 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Recipe for zai-org/GLM-5.3-Flash: NVFP4 on the routed experts and the dense MLP, FP8 KV cache. +# +# The wider sibling of `nvfp4_experts_only-kv_fp8_cast`. GLM-5.3-Flash is a `glm5_next` VLM +# MoE -- 45 decoder layers, 288 routed experts, and HYBRID attention: KDA (linear-attention) +# layers interleaved with NoPE sparse-MLA layers. It needs transformers >= 5.16.1 (earlier +# releases cannot parse the config). +# +# Quantized: the routed experts AND the dense MLP, W4A4 -- both the weight and the input +# quantizers are NVFP4 -- plus an FP8 KV cache in cast mode. Everything else stays BF16: +# shared experts, the router gate, both attention families, the vision tower, embeddings +# and lm_head. +# +# How little "dense MLP" adds: `mlp_layer_types` marks layers 0-2 `dense` and 3-44 `sparse`, +# so only 3 of the 45 layers have a plain MLP -- 9 modules in total (`mlp.gate_proj`, +# `mlp.up_proj`, `mlp.down_proj` each). The other 42 layers carry `mlp.experts..*`. +# +# The one rule that is not obvious is the trailing `*visual*` disable, and it is load-bearing. +# The vision tower reuses the same leaf names -- `model.visual.blocks..mlp.gate_proj` and +# friends, across 24 blocks -- so the dense-MLP patterns match 144 modules inside it. Entries +# apply in order, so the disable has to come LAST or the vision tower is quantized by accident. +# (The experts-only sibling needs no such rule: `*.experts.*` matches nothing under +# `model.visual.*`.) +# +# Two model-specific notes: +# +# * `layerwise.enable=false` is required, not incidental. This is a VLM, so the decoder +# layers nest under `model.language_model.layers` and layerwise_calibrate cannot find +# them. +# +# * The MTP head is inlined as decoder layer 45 and is NOT quantized. The config declares +# `num_hidden_layers: 45`, so transformers builds layers 0-44 and never instantiates +# layer 45; no quantizer attaches to it and it stays BF16. +# +# Like the sibling, the shared `default_disabled_quantizers` unit is deliberately not imported. +# Of its patterns only `*visual*` changes anything here; every other one either matches no +# module in this model or matches a module `base_disable_all` already left off (`lm_head`, the +# `mlp.gate.` routers), and none of them are re-enabled below. Two patterns that look riskier +# than they are: `*.experts.*` needs a literal `.experts.`, so `mlp.shared_experts.*` does not +# match; and `*mlp.gate_proj*` needs `mlp.gate_proj`, which the router `mlp.gate` lacks. + +imports: + base_disable_all: configs/ptq/units/base_disable_all + nvfp4: configs/numerics/nvfp4 + kv_fp8_cast: configs/ptq/units/kv_fp8_cast + +metadata: + recipe_type: ptq + description: >- + GLM-5.3-Flash: NVFP4 (W4A4) on the routed experts and on the dense MLP of layers 0-2, + plus an FP8 KV cache in cast mode using constant amax; max calibration. Shared experts, + router gate, KDA and MLA attention, vision tower, embeddings, lm_head and the inlined + MTP layer all stay BF16. +quantize: + algorithm: + method: max + # Max calibration is fast and does not typically need checkpointing. + # layerwise.enable=false required for VLMs where the decoder layers are nested under + # `model.language_model.layers` (layerwise_calibrate can't find them otherwise). + layerwise: {enable: false} + quant_cfg: + - $import: base_disable_all + # Routed experts: layers 3-44, 288 per layer. + - quantizer_name: '*.experts.*weight_quantizer' + cfg: + $import: nvfp4 + - quantizer_name: '*.experts.*input_quantizer' + cfg: + $import: nvfp4 + # Dense MLP: layers 0-2 -- and, until the last entry undoes it, the vision tower. + - quantizer_name: '*mlp.gate_proj*weight_quantizer' + cfg: + $import: nvfp4 + - quantizer_name: '*mlp.gate_proj*input_quantizer' + cfg: + $import: nvfp4 + - quantizer_name: '*mlp.up_proj*weight_quantizer' + cfg: + $import: nvfp4 + - quantizer_name: '*mlp.up_proj*input_quantizer' + cfg: + $import: nvfp4 + - quantizer_name: '*mlp.down_proj*weight_quantizer' + cfg: + $import: nvfp4 + - quantizer_name: '*mlp.down_proj*input_quantizer' + cfg: + $import: nvfp4 + - $import: kv_fp8_cast + # MUST stay last: keeps the vision tower BF16 after the dense-MLP patterns matched it. + - {quantizer_name: '*visual*', enable: false} diff --git a/modelopt_recipes/models/zai-org/GLM-5.3-Flash/ptq/nvfp4_experts_only-kv_fp8_cast.yaml b/modelopt_recipes/models/zai-org/GLM-5.3-Flash/ptq/nvfp4_experts_only-kv_fp8_cast.yaml index bfb3acc5f4f..5b61c83ff95 100644 --- a/modelopt_recipes/models/zai-org/GLM-5.3-Flash/ptq/nvfp4_experts_only-kv_fp8_cast.yaml +++ b/modelopt_recipes/models/zai-org/GLM-5.3-Flash/ptq/nvfp4_experts_only-kv_fp8_cast.yaml @@ -30,7 +30,11 @@ # unit is not imported: every pattern it turns off (router gates, `lm_head`, the linear-attn # `conv1d` and `in_proj_a`/`in_proj_b`, ...) is already off and is not matched by # `*.experts.*`. Note `*.experts.*` requires a literal `.experts.`, so `mlp.shared_experts.*` -# does not match it and the shared experts stay BF16. +# does not match it and the shared experts stay BF16. The same literal is why the vision tower +# needs no explicit disable here: it is a dense ViT whose 224 modules carry no `experts` in any +# name, so `*.experts.*` cannot reach `model.visual.*`. The wider +# `nvfp4_experts_dense_mlp-kv_fp8_cast` sibling does need one -- its `*mlp.*_proj*` patterns +# match the vision tower's identically named `gate_proj`/`up_proj`/`down_proj` leaves. # # Two model-specific notes: # diff --git a/modelopt_recipes/ptq.md b/modelopt_recipes/ptq.md index 25cc1510997..ec4a6955795 100644 --- a/modelopt_recipes/ptq.md +++ b/modelopt_recipes/ptq.md @@ -414,6 +414,12 @@ checkpoint's** quant config verbatim: the experts-only scope rather than a mixed-precision map, and pins `layerwise.enable=false`, which this VLM requires because its decoder layers nest under `model.language_model.layers`. +- **`models/zai-org/GLM-5.3-Flash/ptq/nvfp4_experts_dense_mlp-kv_fp8_cast`** + widens that scope to the dense MLP as well. `mlp_layer_types` marks only layers + 0-2 `dense`, so this adds just 9 modules (`mlp.gate_proj` / `up_proj` / + `down_proj`) on top of the routed experts. The vision tower reuses those same + leaf names, so a single `*visual*` disable is appended **last** to keep + `model.visual.*` in BF16 — the experts-only variant needs no such rule. *Why special:* unlike any general recipe, these **mix FP8 and NVFP4 across different component types — or individual layers** — and hardcode the precise