FP16 models does not work

[Platorius]

When i use Selurs Hybrid and using any FP16 model with backend TensorRT RTX (not TensorRT), then it does not work. Selur say, it is probably a vs-mlrt problem. It works nice with TensorRT (without RTX) as backend.

[WolframRhodium]

This is a known issue and I don't know any better solution.

[zelenooki87]

@WolframRhodium Hi,

I would like to request support for building TRT_RTX engines from FP16 or BF16 ONNX models, not only from FP32 ONNX models.

As far as I understand, TensorRT-RTX does not expose the same simple precision flag workflow as regular TensorRT/trtexec, so users cannot just pass an equivalent of --fp16 during engine build. For some models, manually converting the ONNX graph to FP16/BF16 first, then building the TRT_RTX model with strict optimizations, can give additional speed and memory benefits.

Would you consider either:

• allowing users to pass already-converted FP16/BF16 ONNX models to the TRT_RTX backend, or
• adding an optional FP32 → FP16/BF16 ONNX conversion step before engine building, possibly using ONNX tools or NVIDIA Model Optimizer / ModelOpt autocast?

Please choose whichever implementation you think fits vs-mlrt best. The important part would be to give users a way to use lower-precision ONNX graphs with TRT_RTX.

Thank you.

[WolframRhodium]

Both options are implemented. You can either build an fp16 engine (or even a nvfp4 one) yourself and then use the plugin interface directly, or install either onnxconverter-common or modelopt and then enable fp16 in backend.TRT_RTX.