-
Notifications
You must be signed in to change notification settings - Fork 2.3k
Description
Need help about the usage of trtexec.
my trtexec version is on TensorRT V9.3 &&&& RUNNING TensorRT.trtexec [TensorRT v9300]
I have a pytorch model trained with AMP bfloat16 type . the model exported into onnx fp32 format, and then I use trtexec above to convert the onnx fp32 into trt fp16.
the model have six inputs, which are:
x dtype float32 shape_as: batchsize X seq_len X channels;
mask dtype bool shape_as: batchsize X seq_len
text dtype int32 shape_as: batchsize X seq_len_text;
t dtype float32 shape_as: batchsize,
cond1 dtype float32 shape_as: batchsize X cond1_channels;
cond2 dtype float32 shape_as: batchsize X seq_len X channels
and one output as follow:
estimator_output dtype float32 shape_as: batch_size X seq_len X channels;
the seq_len, and seq_len_text are dynamic dims specified in onnx exportation.
dynamic_axes = {
"x": {1, 'seq_len'},
"mask": {1: 'seq_len'},
"text": {1: "seq_len_text"},
"cond2": {1: "seq_len"},
"estimator_output": {1: "seq_len"}
}
Now I am writing the trtexec command to convert onnx fp32 into tensorrt fp16. I have to use --inputIOFormats to specify each input. the fmt part can be specified as chw, chw2, chw16 ..., but my input set channels as the last dim, and there is no such btc fomart in the fmt options, which one should I set in my case?
Another problem is, I had tried to convert the onnx fp32 above to bf16 tensorrt engine. the command is as follows:
/bin/trtexec --onnx=MyFp32.onnx --saveEngine=./resultengine_bf16.engine --bf16 --minShape=x:2x4x80,mask:2x4,text:2x2,cond2:2x4x80 --optShape=x:2x100x80,mask:2x100,text:2x50,cond2:2x100x80 --maxShapes=x:2x1000x80,mask:2x1000,text:2x500,cond2:2x1000x80 --inputIOFormats=fp16:chw,bool:chw,int32:chw,fp16:chw,fp16:chw,fp16:chw --outputIOFormats=fp16:chw
During execution, it report error like:
Error: Error: 9 skipping tactic 0x0000000000 due to exception Assertion type == myelinTypeInt32 .....