Describe the bug
Quantizing Mistral-24B with hf_ptq.py for B200/vLLM deployment. Per-tensor static FP8 (FP8_DEFAULT_CFG)
is net slower than BF16 in vLLM (prefill ~2× slower; decode faster), whereas NVFP4 (--qformat nvfp4) is
~2× faster than FP8 and beats BF16 on all metrics. Want to confirm the recommended recipe and understand two
export details.
Steps/Code to reproduce bug
Env: B200 sm_100, modelopt 0.46.0, vLLM 0.25.1 + FlashInfer 0.6.13. Calib 1024 samples @ seq 2048.
Quantization command (hf_ptq.py):
# per-tensor FP8:
python examples/hf_ptq/hf_ptq.py \
--pyt_ckpt_path <bf16-mistral-24b> \
--export_path ./mistral_fp8 --qformat fp8 \
--dataset <custom-calib-dataset> --calib_size 1024 --calib_seq 2048
# NVFP4 (identical, only --qformat and --export_path change):
python examples/hf_ptq/hf_ptq.py \
--pyt_ckpt_path <bf16-mistral-24b> \
--export_path ./mistral_nvfp4 --qformat nvfp4 \
--dataset <custom-calib-dataset> --calib_size 1024 --calib_seq 2048
Bench (vllm bench serve, random 300in / 64out, conc 64):
| Config |
req/s |
TTFT P50 (ms) |
TPOT P50 (ms) |
| BF16 |
44.4 |
547 |
13.7 |
| per-tensor FP8 (+FP8 KV) |
36.2 |
1010 |
7.1 |
| per-tensor FP8 (+BF16 KV) |
38.8 |
660 |
7.7 |
| NVFP4 |
84.1 |
313 |
6.8 |
Long prompts (2048in/64out, conc 32): per-tensor FP8 15.6 req/s vs NVFP4 23.1 req/s.
vllm bench command:
vllm bench serve --model <model-path> --host <host> --port <port> --dataset-name random --random-input-len 300 --random-output-len 64 --num-prompts 1000 --max-concurrency 64'
Note that I have issues on other models quantized in the same way and served through Tensorrt with the torch backend as well.
Expected behavior
I expect fp8 models to go faster than bf16, both with quantized and not quantized kv cache
System information
- Container used (if applicable): docker
- GPU name (e.g. H100, A100, L40S): B200
- Library versions (if applicable):
- Python: 3.12.3
- ModelOpt version or commit hash: 0.46.0
- CUDA: 13.0
- PyTorch: 2.13.0
- Transformers: 5.12.1
Describe the bug
Quantizing Mistral-24B with
hf_ptq.pyfor B200/vLLM deployment. Per-tensor static FP8 (FP8_DEFAULT_CFG)is net slower than BF16 in vLLM (prefill ~2× slower; decode faster), whereas NVFP4 (
--qformat nvfp4) is~2× faster than FP8 and beats BF16 on all metrics. Want to confirm the recommended recipe and understand two
export details.
Steps/Code to reproduce bug
Env: B200 sm_100, modelopt 0.46.0, vLLM 0.25.1 + FlashInfer 0.6.13. Calib 1024 samples @ seq 2048.
Quantization command (
hf_ptq.py):Bench (
vllm bench serve, random 300in / 64out, conc 64):Long prompts (2048in/64out, conc 32): per-tensor FP8 15.6 req/s vs NVFP4 23.1 req/s.
vllm bench command:
vllm bench serve --model <model-path> --host <host> --port <port> --dataset-name random --random-input-len 300 --random-output-len 64 --num-prompts 1000 --max-concurrency 64'Note that I have issues on other models quantized in the same way and served through Tensorrt with the torch backend as well.
Expected behavior
I expect fp8 models to go faster than bf16, both with quantized and not quantized kv cache
System information