🐛 [Bug] test_base_fp4_static_shapes still borderline-flaky with cosine_similarity

[apbose]

Bug Description

tests/py/dynamo/models/test_models_export.py::test_base_fp4_static_shapes (NVFP4 quantization of a single Linear layer) has been flaky across GPUs. History of attempted fixes on release/ngc/26.09:

1. Originally a strict elementwise torch.allclose(expected_output, outputs_trt, rtol=0.3, atol=0.3) — failed on GB200CX8 at max abs_diff=0.336.
2. Loosened to rtol=atol=0.35 — still failed, this time on GB200NVL at max abs_diff=0.362.
3. Switched to cosine_similarity(...) > COSINE_THRESHOLD (0.99), matching the rest of the file's model-correctness tests, reasoning that a strict elementwise check is dominated by a handful of outlier elements near FP4 quantization bin edges (mean abs_diff stayed low and stable, ~0.064, while max spiked) — still failed, this time with cosine similarity 0.9899 vs threshold 0.99, a razor-thin miss.

The global COSINE_THRESHOLD = 0.99 is used by every model-correctness test in this file, but all the others test fp16/fp32 models (ResNet18, MobileNet, BERT, EfficientNet). NVFP4 is 4-bit quantization — dramatically coarser than what that threshold was presumably calibrated for. Reusing the same bar for a 4-bit-quantized single linear layer may simply be inappropriate.

To Reproduce

pytest tests/py/dynamo/models/test_models_export.py::test_base_fp4_static_shapes -v

Repeatedly, across different Blackwell-class GPUs (GB200CX8, GB200NVL, have each individually failed this test with different symptoms in different runs).

Expected behavior

A stable pass/fail signal that isn't dominated by FP4 quantization roundoff noise — e.g. a dedicated (not globally-shared) threshold calibrated for 4-bit quantization, and/or averaging over multiple calibration/input seeds instead of a single random torch.randn draw.

Environment

• GPUs observed failing: GB200CX8, GB200NVL, R100 (each on different runs/symptoms)
• TensorRT: 11.3.x
• modelopt NVFP4 quantization (mtq.NVFP4_DEFAULT_CFG)

Additional context

This is the third attempted fix in the same CI cycle; each incremental loosening bought a bit more headroom but the test keeps finding a new GPU/run to fail on. Suspect this needs a fundamentally different tolerance strategy rather than another threshold bump.

[micwill755]

@apbose The issue is in test_base_fp4_static_shapes, the eager reference output was generated before mtq.quantize() was called, so the test compared an FP16 eager result against a TensorRT NVFP4 result. I moved the eager model invocation after quantization. The test now compares the ModelOpt NVFP4/QDQ reference against the TensorRT NVFP4 execution. This is consistent with the existing test_base_fp4_dynamic_shapes and test_nvfp4_nd_input_quantizer tests, which both generate their expected outputs after quantization.