How does TransformerEngine interact with TensorRT/Onnx?

[tylerweitzman]

Hi,
We are often converting PyTorch models to other engines in order to optimize inference speed. Is it possible to convert a PyTorch model that is using TransformerEngine into TensorRT for faster inference? Or is it a choice either between nn.Linear with TensorRT right now?

[nzmora-nvidia]

Hi @tylerweitzman,

We are evaluating support for exporting TE models to ONNX and import from TensorRT. Support for FP8 GPT export will be rolled out first.
Can you share what kind of models and precisions are you interested in?