[shoumikhin]

What is wrong today

Importing executorch.runtime loads PyTorch whenever it is installed, even for a caller that only passes NumPy arrays and never asks for torch.Tensor outputs. On a small board that costs memory: running a pi0.5 bundle on a Jetson Orin Nano 8 GB took about 430 MB more peak memory, plus some swap, with PyTorch loaded. Today the only way out is setting EXECUTORCH_PYBINDINGS_TENSOR_OUTPUT=executorch before the first import.

What this change does

ExecuTorch no longer imports PyTorch on its own. A call returns torch.Tensor outputs only if the caller has already imported torch, and ExecuTorchResult outputs (readable by NumPy through DLPack) otherwise. The check only trusts a fully imported torch, so a thread importing torch during a call does not break it. EXECUTORCH_PYBINDINGS_TENSOR_OUTPUT still fixes the output type either way.

Behavior change: a script that never imports torch but calls torch methods on outputs now gets ExecuTorchResult and should add import torch. The two in-tree callers that did this are updated.

This is a draft to discuss the default. Keeping today's default and making this opt-in is the alternative.

The runtime log fixes that used to be in this pull request are now in #24602.

Tests

On Linux with an A100 GPU: with torch installed but not imported, import, load and execute never load torch, and outputs are ExecuTorchResult; after import torch, outputs are torch.Tensor; a thread importing torch during a call passes 10 of 10 runs; the environment variable overrides both ways.

cc @digantdesai @freddan80 @per @zingo @oscarandersson8218 @mansnils @Sebastian-Larsson @robell @rascani

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/24321

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure

As of commit 87e0d97 with merge base f1240a2 ():

NEW FAILURE - The following job has failed:

• pull / test-vulkan-operators-linux / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 127

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[shoumikhin]

@claude /pr-review

[claude]

Claude finished @shoumikhin's task in 20m 33s —— View job

---

Needs Discussion: picking the output type on every call from sys.modules lets it change partway through a process and changes a stable API's default with no deprecation step, and test_method_rejects_aliased_input_whose_storage_changed is now always skipped.

Detailed review

PR Review: #24321

Summary

This PR stops the portable Python bindings from importing PyTorch. Each call returns torch.Tensor outputs only if a fully imported torch is already in sys.modules, and ExecuTorchResult otherwise, unless EXECUTORCH_PYBINDINGS_TENSOR_OUTPUT fixes the type. Two design problems need a decision first: the output type now depends on global import state at the time of each call, and the default of a stable API changes without deprecation. Several test and doc defects need fixing whichever way that goes, including a test that is now always skipped. All 8 changed files were reviewed. Nothing was built or run; the findings come from reading the code.

API Design

• The output type depends on hidden global state that is read on every call (extension/pybindings/pybindings.cpp:175-183). Method.execute, Module.forward, get_outputs and get_attribute return whichever type matches whether anything in the process has imported torch yet, so the type can change between two calls on the same Method:

  • Code whose first execute() runs before torch is imported gets ExecuTorchResult. After a function-local import torch, or a dependency that imports torch (torchvision, executorch.exir), the same calls return torch.Tensor. Code that stacks or compares results then sees mixed types. The description's behavior-change note only covers scripts that never import torch.
  • A CuPy program that follows this page's advice (out.to_host(), docs/source/runtime-python-api-reference.rst:44) gets an AttributeError once any dependency imports torch, because torch.Tensor has no to_host.
  • A library built on executorch.runtime cannot control what it gets back. The only lever is an otherwise unused import torch, which this PR adds in pte_metadata.py:28. In the default mode _tensor_output is None, so nothing reports which type a call will return.

  Fix the choice once instead of on every call. That can be per process at import (today's behavior, with torch-free as the opt-in), or per Runtime/Program through an explicit option, with the env var as the process default. Then document the contract on Method.execute.

• The Method.execute docstring still states the old rule (runtime/__init__.py:174-176): "typically torch.Tensor objects. Without PyTorch, tensor outputs are ExecuTorchResult objects." .. autoclass:: Method renders it right above the new paragraph that contradicts it. The page intro (docs/source/runtime-python-api-reference.rst:3) still says execution "returns torch.Tensor values", and points to "Without PyTorch" only for running "without PyTorch installed". Fix this →

Backward Compatibility

• The default changes with no deprecation step or warning. executorch.runtime is not marked experimental, so it is Stable under docs/source/api-life-cycle.md:22-31. Stable APIs "can be changed, but not in a breaking way" unless they go through Deprecated first. Any caller that has torch installed but has not imported it yet now gets ExecuTorchResult, and the failure shows up late or not at all:

  • Duck-typed code breaks silently. Without the new import, the pattern at examples/arm/person_detection_example_cortex_m/utils/pte_metadata.py:35 (result.item() if hasattr(result, "item") else result) would put ExecuTorchResult objects into a dict[str, float | int].
  • Calling .item(), .numpy() or torch.* directly on outputs raises an AttributeError far from the cause.

  Either take the opt-in alternative the description mentions, or ship one release that warns once, then change the default in the next release. The warning would fire when an ExecuTorchResult is returned, the env var is unset, and importlib.util.find_spec("torch") finds torch (that call does not import it).

Thread Safety

• The handling of a thread that imports torch during a call relies on a private importlib detail, and the test only fakes it. loaded_torch_module() reads __spec__._initializing (extension/pybindings/pybindings_executorch_result.h:103-108). The only in-tree test sets that flag by hand on a types.ModuleType (extension/pybindings/test/test_pybindings.py:199-203), so it checks the assumption, not the race. The 10-of-10 threaded run from the description is not in the repo. Two options:
  • Add a deterministic threaded test: a fake torch package on a temporary sys.path whose __init__ blocks on an event while the main thread calls execute(). The broken case already builds a fake package this way.
  • Use PyImport_GetModule(). It is public C API and returns a strong reference. Since CPython 3.9 it waits for another thread's in-progress import instead of returning the partly imported module, so a call that races the import would get torch outputs rather than ExecuTorchResult. Low confidence on the waiting behavior: I could not run a check here.

Testing

• test_method_rejects_aliased_input_whose_storage_changed is now always skipped (extension/pybindings/test/test_pybindings.py:493). It skips unless _C._tensor_output == "torch", and _tensor_output is now None unless the env var is set (pybindings.cpp:4660-4665).
  • No CI job runs it. Only .github/workflows/pull.yml:199,210 (executorch, for the no-ATen suite) and .ci/scripts/test-cuda-build.sh:92-95 (the CUDA DLPack test) set the env var.
  • What it covers is still on by default, because this file imports torch: the aliased-input views from aliased_torch_output / module_aliased_torch_output, whose signatures this PR changes.
  • Fix: use if _C._tensor_output == "executorch":, the same flip the PR makes in test_pybindings_no_aten.py:479-483. Fix this →
• test_buffer_input_does_not_import_torch no longer checks the output type, and its result depends on test order (test_pybindings.py:157-160).
  • loaded_torch_module() stores torch in a static variable the first time it finds it (pybindings_executorch_result.h:97-111). If an earlier test in the same process already did that, this test gets a torch.Tensor even though it removed torch from sys.modules. Run alone, it gets ExecuTorchResult.
  • The loosened np.asarray check accepts both types. The "Decided for each call" comment (pybindings.cpp:175-176) is only true until torch is first found.
  • Separately, the torch branch of configure_portable_tensor_output() no longer calls (void)loaded_torch_module(); after the import (pybindings.cpp:203-204). Run alone with the env var set to torch, this test likely now raises "PyTorch tensor output was requested, but torch is not imported" (pybindings.cpp:1707-1714). It used to pass.
  • Fix: restore that call. Then either assert ExecuTorchResult in a fresh subprocess, or drop the test, since test_tensor_output_import_policy already covers torch installed but not imported. Fix this →
• assert "torch" in str(error) passes for almost any import failure (test_pybindings.py:212). "executorch" contains "torch", so an ImportError that names an ExecuTorch module passes too. Check each case's root message instead: "torch is unavailable" for no, "torch is broken" for broken, and the halted-import message for none. Fix this →
• The switch to torch outputs after import torch is only tested through module(...) (test_pybindings.py:236-239). Method.execute/get_outputs (pybindings.cpp:3859) and get_attribute (pybindings.cpp:3897) each make the decision where they are called, and none of them is tested across the switch. CUDA CI only runs the two fixed settings (.ci/scripts/test-cuda-build.sh:92), so the new default never runs on CUDA, even though hand_over_device_output now takes the per-call value. Fix this →

Code Quality

• The example imports torch only to change the output type (examples/arm/person_detection_example_cortex_m/utils/pte_metadata.py:27-28). Its caller deploy/generate_qparams_header.py imports only argparse, sys, pathlib and this module. It now loads all of PyTorch to read 8 scalars, which is the cost this PR is meant to remove. values[name] = np.asarray(result).item() works for ExecuTorchResult, CPU torch tensors and Python scalars, without torch. Fix this →
• The directory from tempfile.mkdtemp() is never deleted (test_pybindings.py:195), so each broken subprocess leaves a fake torch package behind in the temp directory. Create it under the test's TemporaryDirectory, for example by passing the path through argv. Fix this →

Recommendation

Needs Discussion

This draft asks for discussion of the per-call detection and of changing the default without deprecation. That decision shapes the rest of the change. Whichever way it goes, the always-skipped aliasing test, the stale Method.execute docstring and the weak test assertions need fixing before merge.

[shoumikhin]

@claude review this PR

[claude]

Claude finished @shoumikhin's task in 5m 10s —— View job

---

Needs Discussion: the output type still flips partway through a process depending on whether anything has imported torch yet, and a stable API's default changes with no deprecation step; the aliasing test is still always skipped and the Method.execute docstring is still stale.

Detailed review

PR Review: #24321

Summary

This PR stops the portable Python bindings from importing PyTorch. Each call returns torch.Tensor outputs only if a fully imported torch is already in sys.modules, and ExecuTorchResult otherwise. A separate commit (8903f4c) fixes the order, speed and dropping of runtime log lines that wait for the writer thread.

The three reviewers agreed on the main design problem: the output type depends on global import state that is read on every call. Some items from the earlier review are fixed: the policy test now checks Method.execute/get_outputs, and the env-var torch path does work, because loaded_torch_module() finds torch lazily. Others are still open: the always-skipped aliasing test, the stale docstring, the loosened buffer-input test, and the example's side-effect import.

All 8 files were reviewed. Nothing was built or run, so every finding comes from reading the code. The checkout is shallow, so the diff was taken against the PR's parent commit f1240a2.

Thread Safety

• The writer thread now waits with no time limit on a full non-blocking fd 2, and later main-thread lines pile up behind it (extension/pybindings/pybindings.cpp:226-231). poll(&stderr_fd, 1, -1) replaces the old drop-on-EAGAIN.
  • The batch stays unwritten if the pipe's reader is alive but never drains. Then own_deferred_lines_wait() stays true, so every later line from that thread is deferred behind it and eventually capped at the 1 MiB limit. That includes lines bound for a replaced, perfectly writable sys.stderr such as capsys or redirect_stderr.
  • There is no deadlock, because the GIL is not held and exit only waits 100 ms. Still, one stuck pipe can now silence all later redirected output.
  • A bounded wait, for example poll(..., 1000), that then drops the rest of the batch would keep the new no-drop behavior for slow readers and limit the damage from a dead one. poll returning an error (other than EINTR) should also end the loop instead of retrying forever.
  • (2 of 3 reviewers; suspected, not reproduced.)
• The "is torch still being imported" check relies on a private importlib detail, and no test runs a real concurrent import (extension/pybindings/pybindings_executorch_result.h:103-108).
  • __spec__._initializing is CPython-internal.
  • The initializing case (test_pybindings.py:218-222) only plants a fake module that never finishes importing. So the race from the description, and the switch to torch.Tensor once the import completes, are both untested.
  • If sys.modules["torch"] is a LazyLoader module, reading __spec__ runs the real import, so the bindings would import torch themselves. Low confidence: not run.
  • Consider PyImport_GetModule(), which is public API. Also consider a deterministic threaded test: a fake torch package on a temporary sys.path whose __init__ blocks on an event while the main thread calls execute().

Backward Compatibility

• The default of a stable API changes with no deprecation step. executorch.runtime is not marked experimental, so under docs/source/api-life-cycle.md it should go through Deprecated before a breaking change.
  • A caller that has torch installed but hasn't imported it yet now gets ExecuTorchResult, which has no .item(), .numpy() or torch.* interop. The AttributeError shows up far from the cause.
  • Duck-typed code fails silently. Without the new import, pte_metadata.py:35 (result.item() if hasattr(result, "item") else result) would put ExecuTorchResult objects into a dict[str, float | int].
  • The private _C._tensor_output now reports None by default instead of "torch"/"executorch" (test_pybindings.py:235), so nothing reports which type a call will return.
  • Either take the opt-in alternative from the description, or warn once for a release. The warning would fire when an ExecuTorchResult is returned, the env var is unset, and importlib.util.find_spec("torch") finds torch (that call does not import it).

API Design

• The output type can change between two calls on the same Method (extension/pybindings/pybindings.cpp:690-698).
  • Code whose first execute() runs before torch is imported gets ExecuTorchResult. After any later import of torch, the same call returns torch.Tensor. That import can come from a dependency (transformers, executorch.exir), a pytest plugin, or a function-local import torch.
  • The same script can also behave differently under a test runner than when run directly.
  • The old README promised: "importing PyTorch later does not change the output type". That guarantee is dropped and nothing replaces it.
  • A library on top of executorch.runtime cannot control the type except through a side-effect import torch or the process-wide env var.
  • Decide once instead of on every call: at import (today's behavior, with torch-free as an opt-in), or per Runtime/Program through an explicit option, with the env var as the default. Then document the contract on Method.execute.
  • (3 of 3 reviewers.)
• The Method.execute docstring still describes the old rule (runtime/__init__.py:174-176): "typically torch.Tensor objects. Without PyTorch, tensor outputs are ExecuTorchResult objects." .. autoclass:: Method renders it right next to the new wording that contradicts it. Fix this →

Testing

• test_method_rejects_aliased_input_whose_storage_changed is always skipped (extension/pybindings/test/test_pybindings.py:1433).
  • It skips unless _C._tensor_output == "torch", and that value is now None by default, even though this file imports torch and the outputs are torch tensors.
  • No CI job sets the env var to torch for this suite (only pull.yml sets executorch, and the CUDA script runs only the DLPack test). So the aliased-view checks behind aliased_torch_output now never run.
  • Skip only when it is "executorch", the same flip the PR makes in test_pybindings_no_aten.py.
  • (2 of 3 reviewers.) Fix this →
• test_buffer_input_does_not_import_torch checks neither contract for the output type, and its result depends on test order (test_pybindings.py:166-179).
  • loaded_torch_module() caches torch in a static the first time it sees it (pybindings_executorch_result.h:97-112). After an earlier test has run, popping sys.modules["torch"] still yields torch.Tensor. Run alone, the test gets ExecuTorchResult.
  • The loosened np.asarray check accepts both, so the test passes either way.
  • The "Decided for each call" comment (pybindings.cpp:690) is likewise true only until torch is first seen.
  • Either assert ExecuTorchResult in a fresh subprocess, or drop this check, since test_tensor_output_import_policy already covers torch installed but not imported. (2 of 3 reviewers.)
• Weak assertions in the policy-test script (test_pybindings.py:214, :231):
  • assert "torch" in str(error) also matches any message that names "executorch". Check each case's root message instead: "torch is unavailable", "torch is broken", and the halted-import message.
  • tempfile.mkdtemp() leaves a fake torch package behind on every broken run. Create it under the test's TemporaryDirectory, passed in through argv.
• Parts of commit 8903f4c have no test:
  • The writer stopping once finalization begins (the exiting/writer_entering handshake and the !python_still_running path).
  • Writing one line at a time so that lines stay whole on a pipe.
  • In test_lines_waiting_for_a_full_stderr, assertGreater(speed, 0.05) (:1205) catches the "hundreds of times slower" regression, but would pass a 20x slowdown. assertNotIn("runner lines: 0", ...) (:1202) proves only that some lines arrived, not that none were dropped. Comparing the drained count with the count of lines the runner emits would test the no-drop claim.

Code Quality

• The example loads all of PyTorch just to change the output type (examples/arm/person_detection_example_cortex_m/utils/pte_metadata.py:28). This is the cost the PR is meant to remove. values[name] = np.asarray(result).item() works for ExecuTorchResult, CPU torch tensors and scalars, without torch. Fix this →
• If the merge fails with bad_alloc, the lines come out in the order this commit set out to fix (pybindings.cpp:405-410). The exception is swallowed, the waiting main_thread_logs stay behind, and the call's own lines are written first. Minor, and memory exhaustion is rare. Defer the call's lines behind them in that branch instead of writing them directly.

Recommendation

Needs Discussion

The draft asks whether to change the default, and the reviewers agree that deciding the type on every call from sys.modules is the part to reconsider. Deciding once (at import, or per Runtime) gives callers a type they can rely on. Whichever way that goes, two things need fixing before merge: the always-skipped aliasing test and the stale Method.execute docstring.

[shoumikhin]

@claude review this PR

[claude]

Claude finished @shoumikhin's task in 5m 17s —— View job

---

Needs Discussion: the output type still changes partway through a process depending on whether anything has imported torch yet. The new LLM-runner setup (2b3c024) can be delayed indefinitely, and if it fails, its error is raised in unrelated main-thread code. Four items from earlier reviews are still open: the always-skipped aliasing test, the stale Method.execute docstring, the order-dependent buffer test, and the example's side-effect import torch.

Detailed review

PR Review: #24321

Summary

This PR stops the portable Python bindings from importing PyTorch. Each call returns torch.Tensor only if torch is already in sys.modules, and ExecuTorchResult otherwise. Commit 8903f4c fixes how waiting log lines are written, and commit 2b3c024 sets up log routing when only the LLM runner loads the library.

Three independent reviewers covered all 8 files, and every finding below was re-checked against the code. Nothing was built or run. Since the last review, commit 2b3c024 is new, and the earlier open items are still open.

Thread Safety

• The setup that 2b3c024 schedules at load can be delayed with no limit, can be dropped silently, and its error lands in unrelated code (extension/pybindings/pybindings.cpp:5994-5995, :674-685). Three reviewers reported this.
  • Delayed: a pending call runs only when Python's main thread next runs bytecode. Suppose the runner is first imported on a worker thread while the main thread is blocked in Thread.join(), a C event loop, or an embedder that never runs bytecode. Until the main thread runs bytecode again, main_thread_id stays 0, there is no atexit hook, and there is no fork handler. That is the same line loss the commit fixes, in a narrower window. A fork() in that window also gives the child no handler for the inherited DeferredLogs mutex.
  • Dropped: the return value of Py_AddPendingCall only feeds log_routing_scheduled, and nothing reads that variable. If the queue is full, setup never runs on the runner-only path.
  • Error placement: set_up_log_routing_after_load returns -1 with an error set. Python then raises that error at whatever main-thread bytecode runs next, which may sit inside a user's try/except with no connection to ExecuTorch. The comment at :638-639 says "A setup that raised runs again on the next call", but on the runner-only path nothing calls it again. Only an import of portable_lib (:5479) retries it.
  • Fix: in the pending call, report the error with PyErr_WriteUnraisable and return 0. Also call set_up_log_routing() from the first runner or binding entry that holds the GIL, so setup does not depend on the pending call alone.
  • Low confidence on the subinterpreter case: on 3.12+, the pending call targets the main interpreter, while set_up is a process-wide static.
• set_up = true is set before the Python calls that can release the GIL (pybindings.cpp:646). While the main thread is partway through setup, a worker importing portable_lib returns early and goes on with main_thread_id == 0. If the main thread's setup then fails, the worker's import has already succeeded with no routing. The comment acknowledges that another thread can enter here, but not this result. Suspected, and hard to hit.
• A non-blocking fd 2 whose reader never drains it now holds back all later lines for the rest of the process (pybindings.cpp:226-230). Three reviewers reported this.
  • poll(&stderr_fd, 1, -1) never returns. So writer_running stays true, and both deferred_lines_wait() and own_deferred_lines_wait() stay true.
  • From then on, every line logged outside a call and every line from a later call is queued behind the stuck batch. That includes lines bound for a writable replaced sys.stderr such as capsys or redirect_stderr. They are capped at 1 MiB and the oldest are dropped. Before this commit, EAGAIN dropped one batch and later lines still came through.
  • The return value of poll is not checked, so a POLLERR/POLLHUP combined with repeated EAGAIN would spin.
  • Fix: use a bounded poll such as 1 s, drop the rest of the batch on timeout, and leave the loop on a poll error. Also mention this case in the README's list of hangs.

API Design

• The output type can change between two calls on the same Method (pybindings.cpp:739-746, pybindings_executorch_result.h:97-112). Three reviewers reported this.
  • The type follows sys.modules["torch"] at each call. A later import by a dependency, a pytest plugin, a function-local import torch, or another thread changes what the same code gets back.
  • ExecuTorchResult has to_host() and no .item(), while torch.Tensor has the reverse, so code written against one type breaks after the switch. The pte_metadata.py change shows the workaround callers need.
  • The torch module is also cached in a static the first time it is found (pybindings_executorch_result.h:97). After that, removing torch from sys.modules no longer gives ExecuTorchResult, so "only if torch is already imported" is only true until the first lookup.
  • Fix: decide once, at import or per Runtime/Program, with the env var as the default. Or make the per-call rule opt-in, as the description already proposes. Either way, executorch.runtime is not marked experimental, so changing its default without a deprecation step goes against docs/source/api-life-cycle.md.
• The Method.execute docstring is still stale (runtime/__init__.py:174-176). It says outputs are "typically torch.Tensor objects. Without PyTorch, tensor outputs are ExecuTorchResult objects." Under the new rule, outputs are ExecuTorchResult whenever torch has not been imported, even if it is installed. This is still open from earlier reviews. Fix this →

Testing

• test_method_rejects_aliased_input_whose_storage_changed is still always skipped (extension/pybindings/test/test_pybindings.py:1457). It skips unless _C._tensor_output == "torch", but that value is now None by default, even though this file imports torch. Skip only when the value is "executorch". This is still open from earlier reviews. Fix this →
• test_buffer_input_does_not_import_torch accepts either output type, so whether it passes says nothing, and the type it gets depends on test order (test_pybindings.py:176-179). Because of the static cache above, it gets torch.Tensor after any earlier test that found torch, and ExecuTorchResult when run alone. Assert the exact type in a fresh subprocess. Two reviewers reported this.
• test_lines_waiting_for_a_full_stderr can stop guarding the fix on a slow machine:
  • The drain starts after a fixed 2 s sleep. If TextLLMRunner(...) construction takes about 1.2 s or more, as it can under pytest -n auto, draining has begun before the 0.5 s timing window. The speed ratio is then measured with nothing waiting, so the old spinning code would also pass. Start the drain from an event, after checking that lines are still waiting.
  • assertNotIn("runner lines: 0", ...) shows that some lines arrived, not that none were lost. Compare against the count the runner logs when stderr has room.
• No test covers the runner-only paths most likely to go wrong: a library first loaded on a worker thread, and a failed pending-call setup.
• The policy-test script still has two weak spots (test_pybindings.py:214, :231). tempfile.mkdtemp() leaves a fake torch package behind on every run. "torch" in str(error) also matches any message that names "executorch". Both are still open from earlier reviews.

Code Quality

• The example still loads all of PyTorch only to change the output type (examples/arm/person_detection_example_cortex_m/utils/pte_metadata.py:27-28). That is the cost this PR is meant to remove. values[name] = np.asarray(result).item() works for both output types, without torch. Fix this →
• A test comment gives the wrong reason (test_pybindings.py:692): "Python is exiting, so the runner's lines cannot wait for another thread." atexit hooks run last-registered first, so log_at_exit runs before the bindings' exit hook, while exiting is still false. The lines are written directly because fd 2 has room. The test itself is still valid.

Recommendation

Needs Discussion

The default still has to be decided: per-call detection from sys.modules, or a type fixed once. Whichever way that goes, two items block merge: the always-skipped aliasing test and the stale docstring. The 2b3c024 setup should also stop raising its error into unrelated main-thread code.