Compiling a bfloat16 model triggers float32 precision PyTorch warning

[vadimkantorov]

System Info

5.0.0rc1, 2.9.1

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

/.../.venv/lib/python3.12/site-packages/torch/_inductor/compile_fx.py:312: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance.
  warnings.warn(

model = transformers.AutoModelForCausalLM.from_pretrained("PleIAs/Monad", attn_implementation = 'sdpa', dtype = torch.bfloat16)
model = torch.compile(model, mode = 'reduce_overhead')
model(...)

Is this warning because the model still does some fp32 matmul?

Expected behavior

no warning when bfloat16 model is used

[shantanugupta2004]

Hi @vadimkantorov, I have opened a PR for this issue. Please review and suggest any changes if required.

[ArthurZucker]

hey! I don't think this is related to us directly it has always been there