Skip to content

Let a C++ application use the ExecuTorch TensorRT delegate from an install - #4458

Draft
shoumikhin wants to merge 1 commit into
pytorch:mainfrom
shoumikhin:executorch-wheel-consumable
Draft

Let a C++ application use the ExecuTorch TensorRT delegate from an install#4458
shoumikhin wants to merge 1 commit into
pytorch:mainfrom
shoumikhin:executorch-wheel-consumable

Conversation

@shoumikhin

@shoumikhin shoumikhin commented Aug 1, 2026

Copy link
Copy Markdown
Contributor

Torch-TensorRT can already export an ExecuTorch model that runs through TensorRT. What it cannot
do is give you a prebuilt delegate: the runtime piece that actually executes the TensorRT parts
of the model has to be compiled from source, which means a full C++ build with the TensorRT SDK
installed.

This adds a prebuilt delegate and a CMake package for it, so a C++ program can link it from the
installed wheel.

What you get

before:
  export a model through TensorRT       works
  run it from a C++ program             build the delegate from source first

after:
  torch_tensorrt/lib/libexecutorch_trt_backend.so.2   the delegate, prebuilt
  torch_tensorrt/share/cmake/                         so find_package works

How you use it

In your build, alongside the ExecuTorch runtime:

find_package(executorch REQUIRED)
find_package(torch_tensorrt_executorch REQUIRED)
target_link_libraries(my_app PRIVATE
  executorch::runtime
  executorch::cuda_backend
  torchtrt::executorch_backend
)

In Python, importing the subpackage loads the delegate so it registers itself:

import torch_tensorrt.executorch  # loads and registers the delegate

Version pairing

A delegate built against one ExecuTorch release cannot be assumed to work with another, because
the two share C++ types that are free to change between releases. So the delegate records the
ExecuTorch version it was built against and requires exactly that version, both in CMake and in
its Python metadata.

That is stricter than a normal lower bound, and deliberately so: a mismatched pair fails at
find_package time with a clear message rather than at run time with something confusing.

Status

The prebuilt delegate is off by default. Turning it on requires an ExecuTorch release that
publishes the GPU wheels it pins, which is not available yet, so a release build that enabled it
would produce a wheel whose dependency no package index can satisfy. Once those wheels publish,
enabling it is a build-flag change plus the release job wiring.

Everything else here is ready: the delegate builds, the CMake package exports correctly, the
version pairing is enforced, and the Python import path loads it.

Tested

Built the delegate on Linux aarch64 with a Jetson device and on x86_64, then from a clean
environment with no source checkout reachable:

  • ran find_package(torch_tensorrt_executorch) and linked a C++ program against the delegate
    alongside the ExecuTorch runtime and CUDA delegate;
  • confirmed the delegate survives the link and registers itself, even though nothing in the
    program references a symbol from it;
  • confirmed a mismatched ExecuTorch version is refused at configure time rather than accepted;
  • confirmed importing the Python subpackage loads the delegate, and that the export path still
    imports when no prebuilt delegate is present.

@meta-cla meta-cla Bot added the cla signed label Aug 1, 2026
@github-actions github-actions Bot added component: tests Issues re: Tests component: api [C++] Issues re: C++ API labels Aug 1, 2026
@shoumikhin
shoumikhin force-pushed the executorch-wheel-consumable branch 10 times, most recently from 8b572f6 to c2e8f0e Compare August 2, 2026 22:47
@github-actions github-actions Bot added the component: build system Issues re: Build system label Aug 2, 2026
@shoumikhin
shoumikhin force-pushed the executorch-wheel-consumable branch 16 times, most recently from 0ac7b67 to 0f8c1ad Compare August 3, 2026 16:37
@shoumikhin
shoumikhin force-pushed the executorch-wheel-consumable branch 5 times, most recently from 542bb47 to c30c1cd Compare August 4, 2026 16:59
@github-actions github-actions Bot added the component: api [Python] Issues re: Python API label Aug 4, 2026
@shoumikhin
shoumikhin force-pushed the executorch-wheel-consumable branch from c30c1cd to f23ba48 Compare August 5, 2026 04:13
Two problems in the new verification path.

The install check looked for the delegate under a hardcoded lib directory, but the
install honors the platform library directory, which is lib64 on several
distributions. A completely successful build would have been reported as a failed
install. The configure now pins the directory and the check searches for the file
rather than assuming where it landed.

The device-input path handed a device pointer to the runtime without checking how
that input is supplied. A memory-planned input is copied into the plan with a host
memcpy, so a device pointer there would be read from the host. Only a non-planned
input has its pointer aliased, which is what makes device memory safe. The runner
now reports that clearly instead of corrupting memory.

Test plan: confirmed the install check finds the library when it lands in either
lib or lib64, and that a host-side copy of a device pointer is what the runtime
would do for a memory-planned input.
@shoumikhin
shoumikhin force-pushed the executorch-wheel-consumable branch from f23ba48 to 4965bf5 Compare August 5, 2026 16:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cla signed component: api [C++] Issues re: C++ API component: api [Python] Issues re: Python API component: build system Issues re: Build system component: tests Issues re: Tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant