Skip to content

How to specify inputs fmt when using trtexec in my case? #4679

@JohnHerry

Description

@JohnHerry

Need help about the usage of trtexec.
my trtexec version is on TensorRT V9.3 &&&& RUNNING TensorRT.trtexec [TensorRT v9300]
I have a pytorch model trained with AMP bfloat16 type . the model exported into onnx fp32 format, and then I use trtexec above to convert the onnx fp32 into trt fp16.
the model have six inputs, which are:

x                        dtype float32  shape_as:  batchsize X seq_len X channels;   
mask                 dtype bool      shape_as:  batchsize X seq_len
text                   dtype int32      shape_as:  batchsize X  seq_len_text;
t                        dtype float32  shape_as:   batchsize,
cond1               dtype float32   shape_as:   batchsize X cond1_channels;
cond2               dtype float32  shape_as:    batchsize X seq_len X channels

and one output as follow:

estimator_output    dtype float32    shape_as:  batch_size X seq_len X channels;

the seq_len, and seq_len_text are dynamic dims specified in onnx exportation.

    dynamic_axes = {
       "x": {1, 'seq_len'},
      "mask": {1: 'seq_len'},
      "text": {1: "seq_len_text"},
     "cond2": {1: "seq_len"},
     "estimator_output": {1: "seq_len"}
    }

Now I am writing the trtexec command to convert onnx fp32 into tensorrt fp16. I have to use --inputIOFormats to specify each input. the fmt part can be specified as chw, chw2, chw16 ..., but my input set channels as the last dim, and there is no such btc fomart in the fmt options, which one should I set in my case?

Another problem is, I had tried to convert the onnx fp32 above to bf16 tensorrt engine. the command is as follows:

/bin/trtexec --onnx=MyFp32.onnx --saveEngine=./resultengine_bf16.engine --bf16 --minShape=x:2x4x80,mask:2x4,text:2x2,cond2:2x4x80 --optShape=x:2x100x80,mask:2x100,text:2x50,cond2:2x100x80 --maxShapes=x:2x1000x80,mask:2x1000,text:2x500,cond2:2x1000x80 --inputIOFormats=fp16:chw,bool:chw,int32:chw,fp16:chw,fp16:chw,fp16:chw --outputIOFormats=fp16:chw

During execution, it report error like:

Error: Error: 9  skipping tactic 0x0000000000 due to exception Assertion type == myelinTypeInt32 .....

Metadata

Metadata

Assignees

No one assigned

    Labels

    Module:SamplesIssues when using TensorRT samples under the samples/ directory, including usage with trtexec

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions