[Newton] Enable gravity compensation via Newton's inverse-dynamics API#6408
Merged
kellyguo11 merged 17 commits intoJul 23, 2026
Merged
Conversation
Newton's inverse-dynamics feature (newton-physics/newton#2753, closing the request in newton-physics/newton#2497) provides the primitive that gravity_compensation_forces was waiting for on the Newton backend. - Implement the accessor with eval_inverse_dynamics(GRAVITY_FORCE) and a flat-buffer gather into the per-view contract layout of (N, num_joints + num_base_dofs); buffers are pre-allocated so reads stay CUDA-graph-capture safe. - Bump the pinned Newton commit to the feature's merge commit; the transitive MuJoCo stack moves from the 3.8 to the 3.10 series. - Replace the strict-xfail gap test with positive coverage: shape contracts (fixed/floating base), a Jacobian-transpose gravity identity (incl. non-identity root pose), FK-staleness refresh, and the static-equilibrium accuracy test mirrored from the PhysX suite. - Drop stale unsupported-on-Newton guidance from task-space actions.
The bumped Newton commit requires mujoco-warp 3.10 (bumped upstream in newton-physics/newton#2980, which the inverse-dynamics merge sits on), while isaacsim-core 6.0.0.1 exact-pins mujoco-warp==3.8.0.3. The isaaclab[isaacsim] wheel-install path therefore cannot co-resolve any Newton commit that provides the inverse-dynamics API, as caught by the install CI on this PR. Keep the develop Newton pin and land the accessor forward-compatible instead: articulation init already degrades gracefully on older Newton builds, and the new gravity-compensation tests now skip when the installed Newton lacks the API. They activate automatically with the next Newton pin bump once Isaac Sim moves to the MuJoCo 3.10 series.
Follow the centralized dependency scheme (isaac-sim#6009): the pin moves in [tool.isaaclab.versions] with the mirrored [tool.uv] override. The pinned commit b24dc255 is the merge commit of the upstream inverse-dynamics feature (newton-physics/newton#2753). Newton's MuJoCo requirement moved to the 3.10 series upstream (newton-physics/newton#2980) while isaacsim-core 6.0.0.1 exact-pins mujoco-warp==3.8.0.3, so mujoco/mujoco-warp join override-dependencies — Isaac Lab owns the Newton stack, the same rationale as the existing newton and torch overrides. The previously failing isaaclab[all,isaacsim] wheel resolution now succeeds from the repo root with the overrides applied. With the pin guaranteeing the API, drop the availability guard in the articulation data and the skip gating in the tests; the accessor and tests are unconditional like the sibling Jacobian and mass-matrix accessors.
Add a two-phase end-effector hold test: with gravity acting and compensation off, the OSC hold must sag past a 1.2 cm floor; flipping gravity_compensation mid-episode (read per compute call) must recover the hold to under 5 mm and by at least 10x. This validates the new Newton gravity_compensation_forces accessor end-to-end through the controller loop, complementing the force-level Jacobian cross-check. Measured on the pinned stack: 23.7 mm uncompensated sag vs 1.65 mm compensated, deterministic across runs. The floor assertion keeps the setup discriminating if the task stiffness is ever raised. The Franka home-pose helper gains a disable_gravity parameter since FRANKA_PANDA_HIGH_PD_CFG ships with per-body gravity disabled, and the stale 'Newton lacks a gravity-comp primitive' docstring in the sibling OSC sentinel now points at the new test.
Extend phase 2 from 400 to 1200 steps so the compensated hold is measured after the residual limit cycle settles: at 400 steps the tail mean still carried ~2x convergence transient (1.65 mm vs 0.71 mm settled). Tighten the precision bound from 5 mm to 2.5 mm (3.5x margin over the deterministic settled floor). Document the measured cross-backend floors at the shared task stiffness of 500: Newton ~0.71 mm vs OVPhysX ~8 um, converging to 36 um at stiffness 2000. The residual is a limit cycle from the RNEA feed-forward being integrated by a different engine (MJWarp), not a static gravity mismatch — uncompensated sag agrees with PhysX to ~1%.
Long-run probing showed phase 2 does not settle to a stationary state: after ~1000 steps of decay it enters a slow 0.4-1.0 mm envelope oscillation indefinitely. A single tail-mean measurement therefore depends on where the window lands on the envelope. Bound every 200-step window mean in the second half of a 2000-step phase 2 instead (worst measured window 1.01 mm vs the 2.5 mm bound). Phase 1 does reach a stationary band by ~step 200; add an explicit stationarity guard (tail-half means within 25%, measured drift 2.2%) so an insufficient phase length fails loudly instead of silently measuring a transient.
Probing isolated the compensated-hold oscillation to MJWarp implicitfast integration error at a single 1/120 substep: the residual scales linearly with the solver step, worsens with higher damping gains, vanishes in zero gravity, and is unaffected by how stale the per-control-step torque is — with 4 substeps the same frozen torques hold dead-still at ~1 um, quieter than OVPhysX at ~8 um. This supersedes the previous conclusion that Newton needs stiffer task gains than PhysX for a quiet hold; it needs finer solver substeps. Add a panda_fine sim config (panda with num_substeps=4, documented as load-bearing) and switch the test to it. Both phases now reach true steady states, so drop the windowed-envelope assertions for simple stationarity-guarded tail means, shorten phase 2 from 2000 to 600 steps, and tighten the precision bound from 2.5 mm to 0.1 mm (100x margin over the measured floor; detects sub-percent compensation-force errors).
…avity-compensation # Conflicts: # source/isaaclab_newton/isaaclab_newton/assets/articulation/articulation_data.py # source/isaaclab_newton/test/assets/test_articulation.py
hujc7
marked this pull request as ready for review
July 16, 2026 23:57
Contributor
Greptile SummaryThis PR implements
Confidence Score: 3/5Safe to merge for the common identity-ordering case, but silently wrong for any articulation configured with a non-identity joint ordering. The gravity compensation value fed into OSC would be scrambled across joints whenever a user has configured a custom joint ordering on Newton — the Jacobian and mass matrix are permuted to user order, but the new gravity vector is not, so the feed-forward cancels the wrong joints' gravity load. All other sibling accessors handle this through the existing joint_user_to_backend path; the new kernel has no equivalent. The new tests are thorough for the default ordering but do not cover the non-identity ordering case, so the gap would not surface in CI as currently written. kernels.py (gather_dof_force_rows signature) and the corresponding launch site in articulation_data.py (gravity_compensation_forces property) need the same joint-ordering parameters and permutation logic present in gather_mass_matrix_rows / gather_jacobian_rows. Important Files Changed
Sequence Diagram%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
participant Caller
participant ArticulationData
participant NewtonView as root_view
participant InvDynBuf as _inverse_dynamics_buf
participant Kernel as gather_dof_force_rows
participant Output as _gravity_compensation_forces_ta
Caller->>ArticulationData: .gravity_compensation_forces
ArticulationData->>ArticulationData: _ensure_fk_fresh()
ArticulationData->>NewtonView: eval_inverse_dynamics(state_0, GRAVITY_FORCE, _inverse_dynamics_buf)
NewtonView-->>InvDynBuf: writes flat model-wide gravity_force[]
ArticulationData->>Kernel: "wp.launch(gather_dof_force_rows, dim=(num_instances, num_joints+num_base_dofs))"
Note over Kernel: src[view_dof_starts[i] + d] to dst[i, d] - no joint_user_to_backend reorder
Kernel-->>ArticulationData: _gravity_compensation_forces_buf filled
ArticulationData-->>Output: ProxyArray wrapper
Output-->>Caller: (num_instances, num_joints + num_base_dofs)
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
participant Caller
participant ArticulationData
participant NewtonView as root_view
participant InvDynBuf as _inverse_dynamics_buf
participant Kernel as gather_dof_force_rows
participant Output as _gravity_compensation_forces_ta
Caller->>ArticulationData: .gravity_compensation_forces
ArticulationData->>ArticulationData: _ensure_fk_fresh()
ArticulationData->>NewtonView: eval_inverse_dynamics(state_0, GRAVITY_FORCE, _inverse_dynamics_buf)
NewtonView-->>InvDynBuf: writes flat model-wide gravity_force[]
ArticulationData->>Kernel: "wp.launch(gather_dof_force_rows, dim=(num_instances, num_joints+num_base_dofs))"
Note over Kernel: src[view_dof_starts[i] + d] to dst[i, d] - no joint_user_to_backend reorder
Kernel-->>ArticulationData: _gravity_compensation_forces_buf filled
ArticulationData-->>Output: ProxyArray wrapper
Output-->>Caller: (num_instances, num_joints + num_base_dofs)
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Comment on lines
+739
to
+766
| @wp.kernel | ||
| def gather_dof_force_rows( | ||
| src: wp.array(dtype=wp.float32), | ||
| view_dof_starts: wp.array(dtype=wp.int32), | ||
| dst: wp.array2d(dtype=wp.float32), | ||
| ): | ||
| """Copy per-view articulation DoF forces from a flat model-sized buffer into a view-sized buffer. | ||
|
|
||
| Flat-layout analogue of :func:`gather_mass_matrix_rows` for per-DoF outputs | ||
| written in :attr:`newton.Model.joint_qd` layout (e.g. the gravity | ||
| compensation force from ``newton.eval_inverse_dynamics``). The DoF axis is | ||
| preserved in full (including the leading 6 free-root entries for | ||
| floating-base articulations), matching the cross-library industry | ||
| convention used by PhysX, Pinocchio, Drake, MuJoCo, RBDL, OCS2, and | ||
| iDynTree. | ||
|
|
||
| The gather is in-place on a pre-allocated ``dst`` buffer, so the kernel | ||
| launch is safe under CUDA graph capture. | ||
|
|
||
| Args: | ||
| src: Input flat DoF buffer. Shape is (model.joint_dof_count,). | ||
| view_dof_starts: Flat DoF start index of each view articulation. Shape | ||
| is (num_instances,). | ||
| dst: Output DoF-force buffer for this view. Shape is | ||
| (num_instances, num_joints + num_base_dofs). | ||
| """ | ||
| i, d = wp.tid() | ||
| dst[i, d] = src[view_dof_starts[i] + d] |
Contributor
There was a problem hiding this comment.
gather_dof_force_rows
gather_mass_matrix_rows and gather_jacobian_rows both accept joint_user_to_backend, num_base_dofs, and has_joint_ordering and apply the permutation backend_dof = num_base_dofs + joint_user_to_backend[user_dof - num_base_dofs] when the ordering is non-identity. gather_dof_force_rows has none of these parameters and always reads src[view_dof_starts[i] + d] in raw backend order. When an articulation is configured with a non-identity joint ordering, gravity_compensation_forces would return torques in Newton's internal joint order while mass_matrix and body_com_jacobian_w return data in the user-facing order, silently scrambling the OSC gravity feed-forward term across all joints.
The articulation-reordering series added a user/backend joint ordering to the sibling Jacobian and mass-matrix gathers. Route the same permutation through gather_dof_force_rows so gravity compensation forces come back in user joint order, and extend the Jacobian-identity test with a reversed-ordering variant that fails on a backend-order gather.
…avity-compensation # Conflicts: # pyproject.toml
Newton PR isaac-sim#3530 (post-v1.4.0, on the 1.5 dev line that develop pins) replaced the InverseDynamics container, EvalType flags, and Model.inverse_dynamics() factory with caller-owned output arrays on eval_inverse_dynamics_passive(); each optional output is computed only when its array is passed. The gravity accessor now allocates one flat model-wide gravity-force buffer and requests just that output. Newton allocates its RNEA scratch internally per call through Warp's stream-ordered mempool allocator, which records as allocation nodes under CUDA graph capture; verified on this stack by Newton's own graph-capture test and a composed capture of the accessor's eval + gather path.
The compensated hold settles at the ~1 um solver noise floor, where the 25% relative tail-agreement check is GPU-dependent jitter roulette. A 10 um absolute floor (10% of the 0.1 mm verdict threshold) keeps the guard discriminating for real non-stationarity while ignoring jitter that cannot flip the verdict.
hujc7
requested review from
kellyguo11,
matthewtrepte,
maxkra15 and
mmichelis
as code owners
July 21, 2026 19:52
Replace the host-side numpy precompute of per-view flat DoF start offsets with two integer indirections in the gather kernel over the device-resident Model.articulation_start / Model.joint_qd_start topology arrays — no host round-trip at init and one fewer persistent buffer, matching the device-native style of the sibling gathers.
- Extend the heterogeneous-scene test to read gravity compensation on both views: the flat-buffer gather's padded-layout failure mode is numerically invisible in homogeneous scenes, and drop the shape test's overclaim about covering it. - Rename the init-captured Model topology refs from _sim_bind_* to _model_* — they are not rebound by _create_simulation_bindings, so the prefix broke the file's naming convention. - Guard the rotated-root identity against vacuous agreement under a hypothetical FK-invalidation regression, and refresh the OSC precision calibration provenance to the validated pin.
Caching Model.articulation_start / Model.joint_qd_start at buffer creation added a staleness hazard for no benefit: the eval computes against the view's model, so the gather now reads the topology arrays from that same object per call and can never mix models across a rebuild.
kellyguo11
approved these changes
Jul 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
1. Summary
gravity_compensation_forceson the Newton backend via Newton's inverse-dynamics API, closing the lastNotImplementedErroramong the task-space dynamics accessors introduced in [Newton] Backend-agnostic task-space accessors for IK/OSC #5400. Gravity compensation in the OSC and Pink IK actions now works on Newton.eval_inverse_dynamics_passive, Refine experimental inverse dynamics API newton-physics/newton#3530): caller-owned output arrays, no container/flags. No dependency changes — develop's current Newton pin (81cdcfc2) already provides the API.gather_dof_force_rowskernel (greptile review finding), keeping gravity forces consistent with the Jacobian / mass-matrix accessors on reordered articulations.2. Design notes
eval_inverse_dynamics_passive(state, gravity_force=<flat model-wide buffer>)computes only the requested output; a gather kernel extracts each view's DoF segment (start =joint_qd_start[articulation_start[art_id]], host-precomputed once) and applies the joint-ordering permutation, mirroring the sibling accessors.test_eval_inverse_dynamics_passive_cuda_graph_capturepasses, and a composed capture of the accessor's eval + gather path replays correctly.coriolis_force) is available upstream but intentionally not surfaced — no IsaacLab consumer exists yet.3. Test plan
pytest source/isaaclab_newton/test/assets/test_articulation.py -k gravity_compensation— 11 passed against develop's pinned stack (newton81cdcfc2+ mujoco-warp 3.10.0.2): shape contracts (fixed/floating base), Jacobian-transpose gravity identityg(q) == -Σ J_com^T (m·g)incl. rotated + lifted floating root pose and reversed joint ordering, FK-staleness refresh, static equilibrium (≤5e-3 rad drift over 100 τ_gc-only steps), and the two-phase OSC precision test../isaaclab.sh -fpre-commit clean.