Add animated Gaussian PPISP camera validation across renderers - #7548
Open
moennen wants to merge 18 commits into
Open
Add animated Gaussian PPISP camera validation across renderers#7548moennen wants to merge 18 commits into
moennen wants to merge 18 commits into
Conversation
The demo's camera trajectory never reached either renderer. CameraCfg left update_latest_camera_pose unset, so poses written by set_world_poses() stayed in the sensor view, and the pose seed authored a custom xformOp:transform that replaced the canonical translate/orient/scale stack the sensor frame views write to, silently discarding every later pose write. On Isaac RTX the seed was additionally composed against ancestors evaluated at the USD default time code, where animated rig xforms resolve as unauthored, so the rig transform was counted twice and the camera rendered from inside the scene geometry. Seed the pose through standardize_xform_ops before the sensors are built, sample the ancestor chain at the stage time code, and collapse the duplicated camera rig hierarchy to its static transform so per-frame pose writes are not overwritten by Fabric re-evaluating the animated USD xforms every frame. Replace --max_steps and --save_interval with --fps, --physics_dt, --num_frames, and --write_fps so the render rate no longer changes the physics rate, add --render_only and --profile, move image processing to Warp and NumPy, and flag-gate the Isaac RTX NuRec compositing overrides behind --isaacrtx_keep_compositing_defaults.
The two PPISP camera demos had drifted apart: the OVRTX script still stepped a fixed number of times at a static camera pose, so it could not play back a USD camera trajectory and its CLI no longer matched. Port the reworked demo so both scripts expose the same arguments, timing model, and output layout, and keep the PPISP sensor created before the baseline sensor since an OVRTX PPISP sensor created second on the same camera prims renders black. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Animated NuRec-style Gaussian tracks were rendered frozen: nothing pushed the sampled state to the renderers. Rigid tracks time-sample an ancestor Xform, deformable tracks time-sample the per-particle arrays, so each needs its own update path. Isaac RTX only picks up the state when it is re-authored on the duplicated-env stage, since direct Fabric writes are overwritten during population. OVRTX gets two public renderer hooks instead. Writing the per-particle arrays makes OVRTX stream the prim's geometry again, which the render product never waited for: it asked for AllLoadingFinished with OnlyOnFirstRequest, so only the first frame waited and the splats were missing from every later frame that rendered while a load was in flight. On the deformable asset that was five frames out of eight, with the prim loading three times across the run. Dropping OnlyOnFirstRequest fixes it. The qualifier was never a deliberate choice for animated content; it arrived verbatim with the initial OVRTX integration. The Newton renderer has no Gaussian path at all, so animated tracks raise there rather than silently rendering a static scene.
The Gaussian particle hook called write_array_attribute per frame, which rebuilds the binding descriptor and re-resolves the prim paths on every call. The sibling deformable-body and Newton-particle paths already hold a persistent bind_array_attribute handle for exactly this reason, and a binding is the prerequisite for the async zero-copy write those paths use. Bindings lock in their prim paths and element type, so they are cached per column and path set, and released in close() alongside the others. Output is bit-identical on the deformable asset.
The OVRTX demo resolved every animated Gaussian sample and camera pose from USD inside the profiled render loop, converted them per particle in Python, and wrote them to the renderer synchronously. The measured frame time was dominated by that host work rather than by the render. The Gaussian update hooks now take Warp arrays and write them through their persistent binding with DataAccess.ASYNC, so the renderer reads the caller's buffer in place and the write does not block the host. At most one write per attribute is in flight; the previous one is awaited when the next is issued, which is the buffer lifetime the caller has to respect. The demo resolves the whole animation once, vectorized, into pinned host staging and streams it to the renderer through a small device ring buffer, and prebakes the camera trajectory to the GPU. A playback frame now costs one host-to-device copy per column and no USD, host allocation or Python per particle. Rendered output is unchanged.
The OVRTX backend runs without Kit, so the settings manager had no carb.settings to delegate to and Camera could only warn that Gaussian tonemapping had to be disabled by hand. Rendering an ISP or HDR output over Gaussian splats therefore required exporting OVRTX_rtx_rtpt_gaussian_skipTonemapping_enabled=0 before launch, and produced display-referred radiance (or black frames) when it was forgotten. ovrtx 0.4 exposes no typed API for arbitrary RTX setting paths: RendererConfig covers only its own closed set, and the documented USD property for this setting is not wired up - the renderer reads it from the process-global carb settings rather than the per-render-product settings its neighbours use. Forward the settings through the ovrtx.settings.apply_settings extension instead, queued before the renderer is created, which is the only point OVRTX accepts them. Output is bit-identical to the environment-variable workaround.
The ovstage path copied every per-frame attribute write to host numpy and called wp.synchronize_device first, because ovstage.make_dltensor honours the lane-folded dtype override that ovstage columns require only on its host numpy branch. write_attribute accepts a prebuilt ovstage.DLTensor as-is, so the lane-folded description can be built from the warp array's own pointer, which removes the copy and the host wait from the object, camera, geometry and Gaussian writes. Each write now carries the array's warp stream so the copy-in is ordered behind the kernels that produced the data, matching what the legacy path already does. The description is what ovstage reads the buffer through, so the helper rejects a non-contiguous array and a row count the array cannot fill rather than letting either turn into an out-of-bounds read. Rendered output is unchanged: the NuRec Gaussian scene renders bit-identical frames on the ovstage path, and the default legacy path is untouched.
NVIDIA ships a maintained client for the settings extension as the ovrtx-extensions distribution, but it is published only to an NVIDIA-internal Python index while Isaac Lab installs ovrtx from the public one, so it cannot be depended on here. Record why in the module docstring, and adopt the one part of its behaviour that a public install can have: decoding ovrtx_get_last_error, which is public API, so a rejection reports what OVRTX disliked instead of a bare status number. The lookup now returns a callable that takes the token string and raises on failure, which moves the status check out of the caller.
The kit-less OVRTX demo was a near-copy of the Kit one: the scene duplication,
trajectory resampling, PPISP camera selection, and image comparison were
identical, so every fix had to be applied twice and the two drifted anyway.
Select the renderer with --renderer {newton_renderer,isaac_rtx,ovrtx} instead.
OVRTX runs kit-less, so the Kit app is only launched for the other two, and
the OVRTX-only Gaussian streaming and asset fetch stay behind that check.
Drop the private isaaclab_ppisp._demo_utils dependency in favor of local
helpers so the demo uses public API only, and collapse the camera trajectory
onto prebaked device tensors rather than resolving USD transforms per frame.
Create the PPISP camera before the baseline one, which OVRTX requires: only
the first sensor over a shared camera prim batch gets a bound HDR product.
The standalone script smoke matrix now reaches OVRTX through the merged
script's --renderer choices, so map that backend to the packages behind it.
It is a real alias of --visualizer, dropped by accident while merging the two demo sections.
UsdReplicateContext was only added automatically when Kit was available, so a kit-less run authored spawner-only assets under env_0 alone and left env_1..N missing entirely. The context replicates through pxr.Sdf and never needed Kit, and Kit runs already combined it with the backend physics context, so adding it whenever the cfg has a spawner makes kit-less runs match the Kit path. A kit-less renderer that reads the stage, such as OVRTX, rendered empty tiles for every env past the first.
The 20 m default suits a robot cell, but the demo's capture scenes span hundreds of metres, so duplicated envs interpenetrated and every env rendered a neighbour's geometry in front of its camera. Measure the scene instead, falling back to the old default when nothing is measurable. Gaussian splats are not UsdGeom.Boundable, so a USD bounding box reports an empty extent; the measurement covers the per-particle positions as well.
The PPISP invariants (HDR present, non-degenerate LDR mapping, vignetting, bounded output) are all satisfied by the background alone, so the multi-env coverage passed even when the renderer contributed no splats at all. Compare every tile against a control render whose gaussians sit outside the frustum so that case fails loudly, and author the render hint tokens NuRec exports write, with their exported values, so the synthesised asset matches real captures. That check shows OVRTX contributes no splats in any tile once more than one view tile is active, for the zDepth sort mode NuRec exports author and the renderer defaults to; isaac_rtx renders the same asset correctly, so mark only the OVRTX multi-tile test xfail(strict) with the measurements and leave the renderer bug visible rather than authoring a sort mode that hides it.
RTX drops all gaussian splat contribution, in every tile, when a render product is bound to more than one camera and the sort mode is zDepth. zDepth is what NuRec exports author and what RTX falls back to when the token is absent, so a real capture rendered through a multi-camera OVRTX render product showed no splats at all. cameraDistance derives the sort key in object space instead of from the per-view-tile WorldToView matrix and renders correctly at any camera count. Override sortingModeHint on every gaussian prim while preparing the stage, on the OVRTX path only; isaac_rtx renders zDepth correctly and is left alone. This is a temporary workaround for the renderer bug: it substitutes a different depth metric than the asset asked for, so splats may blend in a different order, and it carries a TODO to be removed with the RTX fix. The assets are unchanged, so the multi-tile gaussian test now doubles as the regression guard for the override and its xfail marker is dropped.
Union the input-scene bounds in defaultPrim space in the PPISP demo's resolve_env_spacing: Boundable bounds were unioned in world space while gaussian particle bounds were unioned in defaultPrim space, so a capture with a non-identity defaultPrim transform added that offset to the union as a spurious gap and over-spaced the env grid. Reuse gaussian_animation.require_default_prim for the three inline defaultPrim checks. Warn when RTX settings are recorded after ovrtx latched its settings, instead of dropping them silently: only the first apply_pending_rtx_settings call reaches OVRTX, so sensors built after the first renderer could not tell that their settings had no effect. Cache the ovstage gaussian path lists and queries per path set rather than rebuilding them on every transform and particle update, and release them at teardown. Document why the demo applies its RTX settings twice, and why the sortingModeHint override cannot be limited to the multi-camera case that misrenders.
moennen
requested review from
AntoineRichard,
StafaH,
fatimaanes,
huidongc,
kellyguo11,
marcodiiga,
ooctipus,
pbarejko and
rilei-nvidia
as code owners
September 3, 2026 21:57
github-actions
Bot
added
documentation
Contributor
Greptile SummaryThe PR unifies PPISP camera validation across Newton, Isaac RTX, and kit-less OVRTX, adding animated Gaussian playback, camera trajectories, renderer configuration, USD replication, and regression coverage.
Confidence Score: 4/5The OVRTX multi-camera path should be fixed before merging because an HDR or PPISP camera created after a non-HDR camera can silently render with the wrong Gaussian tonemapping state. OVRTX latches RTX settings during its first renderer construction, while equal renderer configurations are shared across cameras; a later HDR camera records its required setting but neither reconstructs the renderer nor re-runs the late-setting check. Files Needing Attention: source/isaaclab/isaaclab/sensors/camera/camera.py, source/isaaclab_ov/isaaclab_ov/renderers/ovrtx_settings.py Important Files Changed
Sequence DiagramsequenceDiagram
participant C1 as First OVRTX camera
participant RC as RenderContext
participant O as OVRTX renderer
participant C2 as Later HDR/ISP camera
participant S as SettingsManager
C1->>RC: get_renderer(cfg)
RC->>O: construct renderer
O->>S: flush pending RTX settings
C2->>S: "record skipTonemapping=false"
C2->>RC: get_renderer(equal cfg)
RC-->>C2: return cached renderer
Note over S,O: Late setting is never forwarded
Reviews (1): Last reviewed commit: "Fix ovstage availability guard after reb..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Comment on lines
203
to
208
| elif renderer_type == "ovrtx" and require_hdr_output: | ||
| from isaaclab.app.settings_manager import get_settings_manager | ||
|
|
||
| # OVRTX has no carb.settings to delegate to, so this is recorded in the settings manager | ||
| # and forwarded to the RTX runtime by the OVRTX renderer's constructor. | ||
| get_settings_manager().set_bool("/rtx/rtpt/gaussian/skipTonemapping/enabled", False) |
Contributor
There was a problem hiding this comment.
When a non-HDR OVRTX camera creates the shared renderer before an HDR or PPISP camera with the same renderer configuration, the later camera records skipTonemapping=False after OVRTX has latched its settings, while renderer reuse prevents another forwarding attempt. Gaussian radiance remains tonemapped, producing incorrect HDR and PPISP output without the intended late-setting warning.
Knowledge Base Used: Simulation, rendering, and sensors
![isaaclab-review-bot[bot]](https://avatars.githubusercontent.com/u/157846462?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Isaac Lab Review Bot
The renderer-unified PPISP demo, animated Gaussian playback, OVRTX update hooks, settings forwarding, and kit-less replication changes are broadly coherent. Two integration gaps remain: Gaussian animation times are excluded whenever camera animation exists, and OVRTX tonemapping settings can be silently recorded too late when a renderer instance is reused.
- Design and architecture: The split between renderer-agnostic track playback, Isaac RTX USD re-authoring, and explicit OVRTX renderer hooks is coherent. However, animation scheduling currently treats Gaussian times as a fallback rather than combining them with camera times, which can truncate Gaussian-only portions of a scene containing both animation types.
- API: The new OVRTX Gaussian update APIs and buffer-lifetime contracts are consistent with their consumers. The OVRTX settings path remains order-dependent: an ISP/HDR camera created after an existing plain OVRTX camera can record the required tonemapping setting without constructing a renderer, so the value is neither applied nor reported as dropped.
- Implementation: Camera trajectory prebaking, device ring-buffer playback, lane-folded DLTensor handling, and teardown paths were traced. The remaining fixes are to incorporate Gaussian-authored times when camera samples also exist and to detect or flush late OVRTX settings when an existing renderer is reused.
Minor fixes needed. Posted 2 actionable findings inline.
Automated review; human maintainers own approval decisions.
| ) | ||
| else: | ||
| # A scene may animate only its Gaussians, in which case their time samples drive the frames. | ||
| trajectory_times = gaussian_anim.collect_authored_times(source_stage, gaussian_tracks) |
Contributor
There was a problem hiding this comment.
🟡 Warning · Implementation — Gaussian times used only as camera fallback
collect_authored_times documents that the demos union Gaussian sample times with the camera trajectory, but this call sits in the else branch. When a scene animates both, frame_time_codes span only the camera range, so Gaussian motion outside it is silently truncated. Either union both ranges before resampling, or correct the helper's docstring to describe fallback-only use.
| elif renderer_type == "ovrtx" and require_hdr_output: | ||
| from isaaclab.app.settings_manager import get_settings_manager | ||
|
|
||
| # OVRTX has no carb.settings to delegate to, so this is recorded in the settings manager |
Contributor
There was a problem hiding this comment.
🟡 Warning · Api — Removed warning hides unapplied OVRTX tonemapping setting
The setting is recorded here, but it only reaches OVRTX through apply_pending_rtx_settings() in OVRTXRenderer.init. render_context.get_renderer reuses a renderer for an equal cfg, so an ISP/HDR camera created after a plain OVRTX camera records the value without constructing a renderer: it is neither applied nor reported as dropped, and Gaussians render tonemapped silently. Warn (or re-flush) when an OVRTX renderer already exists.
moennen
force-pushed
the
nicolasm/deformable-gaussian-splats-support
branch
from
September 3, 2026 22:21
46162d4 to
d981101
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Important
Confirm the pull request base before submitting. This pull request targets
develop.This pull request extends PPISP camera validation to static and animated Gaussian scenes across the Newton, Isaac RTX, and OVRTX renderers.
The changes provide a consistent workflow for comparing PPISP and baseline camera outputs, including tiled multi-environment scenes and kit-less OVRTX execution.
Fixes # (issue)
Type of change
Release backport
developScreenshots
Not applicable. This change adds automated image-generation and comparison workflows.
Changes
Dependencies
Testing
run-cicomment.Checklist
Docker and GPU tests run on demand. Push the commits you want tested, then
comment
run-cion the pull request.pre-commitcheckssource/<pkg>/changelog.d/for all touched packagesCONTRIBUTORS.md