[FIX] Newton sensor CUDA graph capture fails on deformable geometry

[mmichelis]

Fixes nvbug 6675392.

Newton sensor CUDA graph capture fails on any task with deformable geometry, for example Isaac-Lift-Cloth-Franka-Camera:

RuntimeError: Conditional body graph contains an unsupported operation (memory allocation)
Newton sensor graph capture failed; falling back to eager execution.

NewtonManager wrapped every registered scene-query task in a wp.capture_if conditional body. CUDA does not permit memory-allocation nodes inside a conditional body graph, and wp.Mesh.refit() allocates a scratch buffer on every call. Cloth reaches that call through SensorTiledCamera.update -> sync_transforms -> RenderContext.update -> _sync_triangle_mesh, so capture fails and all sensor work falls back to eager execution. Rigid-only camera tasks are unaffected because wp.Bvh.refit() does not allocate.

The shared BVH refit and each scene-query task are now captured into their own top-level graph, where allocation nodes are legal. _update_sensor_tasks launches the refit graph when the state is dirty and then the graph of each requested task. This preserves the previous per-task selection semantics and removes the device flag array along with its per-call host-to-device copy.

The failure was not fatal, so it only ever surfaced as a logged traceback while training continued.

Type of change

• Bug fix (non-breaking change which fixes an issue)

Checklist

• I have run the pre-commit checks with ./isaaclab.sh --format
• I have added tests that prove my fix is effective or that my feature works
• I have added a changelog fragment under source/isaaclab_newton/changelog.d/

[greptile-apps]

Greptile Summary

This PR fixes Newton sensor CUDA graph capture for deformable geometry by replacing conditional task bodies with independently captured top-level graphs.

• Captures BVH refitting and each registered scene-query task into separate CUDA graphs.
• Launches only the requested task graphs while preserving dirty-state BVH refitting and eager fallback.
• Removes the device/host flag arrays used to select conditional graph bodies.
• Adds CUDA coverage for an allocating wp.Mesh.refit() task and updates shared sensor-graph assertions.

Confidence Score: 5/5

The PR appears safe to merge, with no concrete changed-code failure remaining.

Graph invalidation and publication keep the refit and task graph collections consistent, current callers request one registered task at a time, and capture failures retain the established eager-execution fallback.

Important Files Changed

Filename	Overview
source/isaaclab_newton/isaaclab_newton/physics/newton_manager.py	Replaces the conditional shared sensor graph with consistently invalidated, independently captured refit and task graphs while retaining eager fallback.
source/isaaclab_newton/test/physics/test_newton_manager_abstraction.py	Adds CUDA coverage proving that a task containing allocating wp.Mesh.refit() work can be captured as a top-level graph.
source/isaaclab_newton/test/sensors/test_newton_raycast_sensor.py	Updates integration coverage to assert that per-task sensor graphs are populated when CUDA graphs are enabled.
source/isaaclab_newton/changelog.d/fix-sensor-graph-allocating-task.rst	Documents the deformable-geometry capture failure and the move to independent task graphs.

Sequence Diagram

sequenceDiagram
    participant S as Sensor caller
    participant M as NewtonManager
    participant B as BVH refit graph
    participant T as Requested task graph
    S->>M: update_sensor_tasks(name)
    alt graphs not captured
        M->>M: Warm up refit and registered tasks
        M->>B: Capture top-level refit graph
        loop Each registered task
            M->>T: Capture independent top-level graph
        end
    end
    opt Sensor state is dirty
        M->>B: Launch refit graph
        B-->>M: Updated query acceleration structures
    end
    M->>T: Launch graph for name
    T-->>S: Updated sensor output

Loading

_{Reviews (1): Last reviewed commit: "Capture Newton scene-query tasks into pe..." | Re-trigger Greptile}

[isaaclab-review-bot]

Isaac Lab Review Bot

The change replaces the conditional sensor pipeline with standalone BVH-refit and per-task CUDA graphs, directly addressing Warp’s restriction on allocations inside conditional graph bodies while preserving eager fallback and requested-task execution.

• Design and architecture: Per-task top-level graphs are a proportionate design for allocation-capable scene-query work. Scheduling and invalidation remain encapsulated in NewtonManager. The main non-blocking tradeoff is that graph count and launch count now scale with the number of registered sensor tasks.
• API: No public API is changed. The renamed graph state and invalidation methods are private implementation details, affected internal call sites and tests are updated consistently, and the isaaclab_newton changelog fragment documents the user-visible fix.
• Implementation: The refit-before-task ordering, dirty-state handling, all-or-nothing capture fallback, failure latch, and clear/reset paths are internally consistent. The CUDA regression test exercises standalone capture of an allocating wp.Mesh.refit task, while the existing combined renderer/raycast test is adapted to the per-task graph representation.

No blocking issues. No inline issue met the actionable-evidence threshold; the assessment above records the review feedback.

Automated review; human maintainers own approval decisions.