Make AppLauncher teardown truthful and re-entrancy-safe; document mGPU NCCL workaround

[hujc7]

Summary

• Consolidates AppLauncher process-lifecycle handling into one nested _SimulationAppLifecycle class: startup announcements (CI marker, Kit version diagnostics) plus the entire exit-path policy, with the policy table as the class docstring.
• Fixes every exit path that misreported failure as success or destroyed its own diagnostics (full failure-case table below). Absorbs the atexit exit-code fix from Fix AppLauncher exception exit status #6634.
• Documents the NCCL cuMem workaround for multi-GPU RTX training on NUMA-spanning GPU allocations (NCCL_CUMEM_HOST_ENABLE=0 first, NCCL_CUMEM_ENABLE=0 as fallback).
• The upstream SimulationApp exit-status fix (public reference: Preserve nonzero exit status through fast-shutdown close IsaacSim#717) has been merged (isaacsim.simulation_app >= 2.18.5): the signal handler now simply passes close(exit_code=128 + signum) — full teardown + truthful status on fixed builds, truthful status on older builds. The one remaining WORKAROUND(isaac-sim) is the SIGINT re-registration (upstream handler still exits 0 before user code unwinds); it fails loudly if drift disables it.
• Fixes AppLauncher atexit handler masks uncaught exceptions as exit code 0 #6573 (absorbed atexit exit-code fix, originally Fix AppLauncher exception exit status #6634 by @nblauch).
• Fixes [Bug Report] AppLauncher SIGTERM handler does not terminate Python workers #6530: the SIGTERM handler no longer returns to the interrupted execution path — the worker exits through close(exit_code=128 + signum) (full Kit teardown on isaacsim.simulation_app >= 2.18.5) or dies by the re-raised signal, so distributed workers terminate with a truthful status instead of surviving and spamming TCPStore Broken pipe errors.

Failure cases and how this PR addresses them

Root mechanism: Kit fast shutdown terminated the process with exit code 0 from inside SimulationApp.close(), so any death funneled through an unqualified close() was reported as success; additionally, the abort-signal handler was unguarded against re-entrancy.

How the process ends	Before this PR	After this PR
Unhandled Python exception	atexit close overwrote the pending failure with exit 0 (CI false-green)	exits 1 (sys.last_exc detected; absorbed from #6634; SystemExit documented as not yet covered)
Single SIGTERM (torchrun teardown, SLURM preemption, kill)	graceful close → exit 0; launcher marks the killed rank SUCCEEDED; surviving ranks hang until the NCCL watchdog	close(exit_code=128 + signum): full teardown + truthful status on isaacsim.simulation_app >= 2.18.5; truthful status on older builds; dies by the signal if close() returns
Second signal while a close is running (repeated SIGTERM; fault inside the replicator stop/wait)	handler re-entered close() → infinite recursion → SIGKILL-only shutdown, spurious SIGSEGV, logs flooded (~975 recursion frames/job observed on OSMO pods)	guard: re-entrant signal falls back to SIG_DFL
Signal racing the normal atexit close	nested full second teardown of a half-closed app	atexit arms the same guard
kill -ABRT	graceful close → exit 0	same truthful-close path as SIGTERM
Real SIGSEGV, main thread	Python handler can never run → process spins forever at 100% CPU, crash reporter clobbered (no minidump)	SIGSEGV no longer intercepted → default action, minidumps restored
Real SIGSEGV, worker thread	handler ran on the main thread → exit 0 for a crashed process	same: default action, truthful signal death
Ctrl-C	SimulationApp's handler exits 0 before user finally/KeyboardInterrupt code runs	Python default handler restored: KeyboardInterrupt unwinds user code, nonzero exit

Not addressed here (tracked elsewhere): sys.exit(N)/SystemExit still exits 0 (gap inside the #6634 mechanism, documented at the detection site).

Implementation notes

1. All exit-path logic lives in AppLauncher._SimulationAppLifecycle; the class docstring is the policy table, and each decision carries its rationale in place.
2. The signal handler passes the killed-by-signal status through close(exit_code=128 + signum). With the merged upstream fix (isaacsim.simulation_app >= 2.18.5) the app performs its full teardown and exits with that status; on older builds the status is preserved without the teardown; if close() returns (fast shutdown disabled), the handler re-raises with the default action. A TypeError fallback warns loudly if a future SimulationApp drops the parameter.
3. Docs: distributed camera training fails deterministically when the allocated GPUs span NUMA nodes and passes on a single-switch set; disabling NCCL cuMem host allocations rescues the failing shape in paired same-node experiments. Added to the multi-GPU NCCL troubleshooting section.

Testing. Six kitless unit tests (test_simulation_app_lifecycle.py: killed-by-signal status, re-entrancy both directions, exit-code selection, drift fallbacks on both paths) plus two real-Kit integration tests (test_app_launcher_exit_status.py: SIGTERM → truthful termination status with no handler recursion; unhandled exception → exits 1). The integration tests pass against both the pre-fix and the fixed (>= 2.18.5) SimulationApp builds; on develop's original behavior the SIGTERM test observes exit code 0 (the bug).

Type of change

• Bug fix (non-breaking change which fixes an issue)
• Documentation update

Checklist

• I have read and understood the contribution guidelines
• I have run the pre-commit checks with ./isaaclab.sh --format
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• I have added a changelog fragment under source/<pkg>/changelog.d/ for every touched package (do not edit CHANGELOG.rst or bump extension.toml — CI handles that)
• I have added my name to the CONTRIBUTORS.md or my name already exists there

[greptile-apps]

Greptile Summary

This PR consolidates all AppLauncher process-lifecycle handling into a new _SimulationAppLifecycle inner class, fixing several exit paths where Kit fast shutdown masked failures as exit code 0 and where repeated signals triggered infinite recursion. It also documents the NCCL_CUMEM_HOST_ENABLE=0 workaround for distributed RTX training failures on NUMA-spanning GPU configurations.

• Re-entrancy guard: _closing flag prevents a second close() call when a signal arrives while the app is already tearing down; both the atexit hook and the signal handler arm the same flag.
• Truthful exit status: _close_at_exit passes exit_code=1 when sys.last_exc is set; _on_abort_signal disables Kit fast shutdown so close() returns, then re-raises with SIG_DFL to produce the conventional 128+signum status.
• SIGSEGV removed: the Python-level handler is intentionally dropped so the carb crash reporter can produce minidumps and the main-thread spin-at-signal-delivery is avoided.

Confidence Score: 5/5

Safe to merge; the exit-path logic is well-reasoned, each decision is explicitly documented, and the two workarounds are clearly annotated as temporary with a deletion path tied to an upstream issue.

All changes are in the teardown/signal-handling path, exercised by the new unit and integration tests. The implementation carefully handles the re-entrant signal case, the fast-shutdown workaround, and exit-code preservation. The only findings are minor edge cases that do not affect correctness under normal conditions.

No files require special attention; the core logic in app_launcher.py is well-tested by the two new test files.

Important Files Changed

Filename	Overview
source/isaaclab/isaaclab/app/app_launcher.py	Core change: replaces ad-hoc signal/atexit setup with _SimulationAppLifecycle inner class; re-entrancy guard, truthful exit codes, and fast-shutdown workaround are well-designed; one minor API-boundary concern in the TypeError catch
source/isaaclab/test/app/test_simulation_app_lifecycle.py	Five kitless unit tests cover re-entrancy both directions, atexit guard, exit-code selection, and drift fallback — all targeting the specific failure modes described in the PR; test logic is sound
source/isaaclab/test/app/test_app_launcher_exit_status.py	Two integration tests launch a real Kit process and assert on exit status; the SIGTERM test startup-ready loop can hang indefinitely if the child never emits stdout
docs/source/features/multi_gpu.rst	Adds NCCL cuMem workaround documentation for distributed RTX training failures on NUMA-spanning GPU allocations; prose is clear and accurate
source/isaaclab/changelog.d/fix-abort-handler-reentrancy.rst	New changelog fragment accurately describes all exit-path fixes and the intentional SIGSEGV removal; no issues

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[AppLauncher.__init__] --> B[_SimulationAppLifecycle.__init__]
    B --> C[announce_startup]
    C --> D[install_exit_handlers]
    D --> E{Process ends how?}
    E -->|Normal exit| F[_close_at_exit]
    F --> G{sys.last_exc set?}
    G -->|Yes| H[app.close exit_code=1]
    G -->|No| I[app.close exit_code=0]
    E -->|SIGTERM or SIGABRT| J[_on_abort_signal]
    J --> K{_closing True?}
    K -->|Yes re-entrant| L[SIG_DFL + raise_signal]
    K -->|No| M[_closing=True disable fast shutdown]
    M --> N[app.close]
    N --> O[SIG_DFL + raise_signal 128+signum]
    E -->|SIGINT| P[default_int_handler KeyboardInterrupt raised]
    E -->|SIGSEGV| Q[Not intercepted carb minidumps]

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
    A[AppLauncher.__init__] --> B[_SimulationAppLifecycle.__init__]
    B --> C[announce_startup]
    C --> D[install_exit_handlers]
    D --> E{Process ends how?}
    E -->|Normal exit| F[_close_at_exit]
    F --> G{sys.last_exc set?}
    G -->|Yes| H[app.close exit_code=1]
    G -->|No| I[app.close exit_code=0]
    E -->|SIGTERM or SIGABRT| J[_on_abort_signal]
    J --> K{_closing True?}
    K -->|Yes re-entrant| L[SIG_DFL + raise_signal]
    K -->|No| M[_closing=True disable fast shutdown]
    M --> N[app.close]
    N --> O[SIG_DFL + raise_signal 128+signum]
    E -->|SIGINT| P[default_int_handler KeyboardInterrupt raised]
    E -->|SIGSEGV| Q[Not intercepted carb minidumps]

Loading

_{Reviews (2): Last reviewed commit: "Fail loudly if the SimulationApp workaro..." | Re-trigger Greptile}

[AntoineRichard]

[AI review] [P2] Make the startup timeout interrupt the pipe read

The loop over proc.stdout blocks until a line or EOF arrives, so the 300-second deadline is never evaluated if Kit hangs silently during startup. In that failure mode this test can stall the entire shard until the GitHub Actions job timeout.

Please use a bounded readiness mechanism such as select/selectors, a reader thread with a timeout, or equivalent, and preserve the child output in the timeout failure.

[hujc7]

Fixed in ddc38bd: readiness now waits on an Event fed by a daemon reader thread, so a silent startup hang or an early child exit fails within the 300 s deadline instead of stalling the shard, and the failure report preserves the child's stdout and stderr.

[kellyguo11]

let's avoid mentioning --enable_cameras, it shouldn't be necessary anymore

[kellyguo11]

should we use logging instead of print here?

[kellyguo11]

what happens if app.close() hits a different TypeError? could that ever become the case?