Rewrite advanced benchmark framework guide by AntoineRichard · Pull Request #6627 · isaac-sim/IsaacLab

AntoineRichard · 2026-07-20T10:42:29Z

Description

Rewrites the advanced benchmark guide around the supported Python API.

The guide:

leads with typed runtime, startup, training, and play requests;
documents verified request configuration and typed result access;
explains formatter output, recorder behavior, process lifetime, and synchronized-timing caveats;
replaces stale framework examples with an executable custom benchmark;
hands isolated asset and sensor questions to the micro-benchmark guide.

Dependencies

Documentation dependency: Rewrite benchmark user guide #6595, which introduces the CLI-first user guide.
API dependency: Promote benchmarks to library API #6564, which introduces the public typed benchmark API documented here.
Transitive dependency through both branches: Improve benchmark sampling and synchronized timing diagnostics #6474.
Satisfied prerequisite: Refactor train and play into rl subpackage #6553 is already merged.

The focused documentation comparison is:
AntoineRichard/IsaacLab@antoiner/benchmark-user-guide...antoiner/benchmark-framework-guide

Type of change

Documentation update

Validation

PYTHONPATH=source/isaaclab ./isaaclab.sh -p -m pytest source/isaaclab/test/benchmark/test_api.py -q (18 passed).
Small Newton MJWarp public-API probes for runtime, startup, RSL-RL training, and trained-policy play.
Extracted and executed the custom benchmark example.
./isaaclab.sh -d (warning-free).
./isaaclab.sh -f

Checklist

I have read and understood the contribution guidelines
I have run the pre-commit checks
I have made corresponding documentation changes
My changes generate no new warnings
Existing benchmark tests cover the documented API
A changelog fragment is not required for this documentation-only PR
My name already exists in CONTRIBUTORS.md

Exclude configurable warmup frames from runtime metrics and compute throughput from total measured work over total wall time. Use longer defaults so the quick benchmark is less sensitive to startup and short sampling windows.

Add an opt-in runtime benchmark probe around the environment's SimulationContext step calls. Report host time inside and outside that boundary while preserving the original method after measurement.

Measure environment and synchronized simulation stepping across runtime, training, and play benchmarks. Store the resulting Isaac Lab overhead in benchmark schema 1.1 while leaving regular training and play entry points uninstrumented.

Require an explicit benchmark flag before wrapping environment and simulation steps. Disabled benchmarks now avoid method replacement and device synchronization while schema 1.1 continues to represent measured runs.

Delegate synchronization and elapsed-time measurement to the existing Isaac Lab Timer. Keep the interception layer focused on recording per-step samples so environment, simulation, and derived overhead metrics include standard deviation and peak values.

Record the env.step boundary in every benchmark workflow without adding explicit synchronization. Keep the nested synchronized simulation probe optional, and expose environment-step FPS in schema 1.2 so runtime, play, and training results can be compared on the same scope.

Name the opt-in instrumentation after the Isaac Lab overhead it reports instead of the implementation detail used to derive it.

Synchronize before environment and simulation timing begins so policy and action-generation work queued earlier is not attributed to Isaac Lab overhead.

Recalculate throughput standard deviation around the aggregate effective mean so benchmark MeanStd values use one consistent center.

Validate workload counts before launch and require positive environment timing samples so empty synchronized runs fail with actionable errors instead of division by zero.

Run the training and play smoke matrix without synchronized overhead measurement, retain one focused synchronized replay, and verify pending work before env.step is excluded from its metric.

Keep the first runtime step outside throughput regardless of overhead instrumentation and floor derived overhead at zero when nested wall timers cross due to measurement noise.

Rename the disputed overhead metric as a serialized synchronized step breakdown and expose explicit measurement modes. Mark every instrumented timing and rate as diagnostic, validate the additive partition, and document why outside-simulation time is not removable overhead.

Follow-up polish from review of the runtime/synchronized-timing changes; no behavior change. - Drop the redundant ``dest=``/``default=False`` on runtime.py's ``--measure_synchronized_step_breakdown`` (argparse derives both) so the flag matches the eight RL adapter scripts. - Remove the single-use ``timer_context`` alias and enter the step-timing recorder alongside ``BenchmarkMonitor`` on one line, matching the adapters' context-manager pattern. - Reorder the ``build_runtime`` Args block to signature order (``aggregate_throughput`` before ``frames_per_environment_step``). - Clarify ``MeanStd.mean``/``std``: for effective-throughput fields these are the aggregate rate and the dispersion about the effective mean, not the arithmetic sample mean and standard deviation.

Bring PR isaac-sim#6553 on top of the benchmark timing work from PR isaac-sim#6474 so the benchmark library can dispatch the new RL entrypoints directly.

Move the supported benchmark workflows into isaaclab.benchmark and expose typed in-process dispatch for runtime, startup, training, and play benchmarks. Keep deprecated script and test namespace shims so existing callers can migrate without an immediate break.

Use one device-wide synchronization per timing boundary and keep mock backend state preallocated between samples. Initialize ordering and dynamics buffers so PhysX and Newton accessors are measured without hidden skips or allocation work.

Replace per-file Python shims with lazy module proxies so legacy imports remain compatible while Git can recognize the benchmark implementation moves. Retain type forwarding stubs and document removal in Isaac Lab 3.1.

Add a warmup_steps parameter to EnvironmentStepTimingRecorder that drops the first N ``env.step()`` calls (CUDA graph capture, lazy kernel compilation) from the recorded environment-step timing. In serialized mode the skipped steps' simulation call-count and time are rolled back so the recorded aggregate stays consistent. Expose it as an opt-in ``--warmup_steps`` flag (default 0, validated non-negative, recorded in metadata) on the eight RL train/play benchmark adapters. ``runtime.py`` is unchanged: its ``--warmup_frames`` already excludes the startup step before the recorder is installed. Add recorder unit tests for the host-return and serialized exclusion paths and a parse-level test rejecting a negative ``--warmup_steps``.

Preserve the destination Warp dtype when copying mock binding data so structured pose and spatial-vector buffers can be exercised by asset data benchmarks.

Cover articulation, rigid object, and rigid object collection method and data access performance using kitless mock bindings comparable to the Newton benchmark matrix.

Apply the repository Ruff format to the newly tracked benchmark scripts and keep fill-ratio selection conditions within the line-length limit.

Measure synchronized updates, submission latency, and backend reads for contact, frame-transformer, IMU, PVA, joint-wrench, and ray-caster sensors without charging physics stepping to sensor timing.

Explain why isolated asset and sensor benchmarks exist, how to run each backend workload, and how to interpret and compare their outputs.

# Conflicts: # scripts/benchmarks/rl_games/benchmark_rl_games_play.py # scripts/benchmarks/rl_games/benchmark_rl_games_train.py # scripts/benchmarks/rsl_rl/benchmark_rsl_rl_play.py # scripts/benchmarks/rsl_rl/benchmark_rsl_rl_train.py # scripts/benchmarks/runtime.py # scripts/benchmarks/sb3/benchmark_sb3_play.py # scripts/benchmarks/sb3/benchmark_sb3_train.py # scripts/benchmarks/skrl/benchmark_skrl_play.py # scripts/benchmarks/skrl/benchmark_skrl_train.py

# Conflicts: # docs/source/testing/micro_benchmarks.rst # scripts/benchmarks/benchmark_load_robot.py

Make runtime and startup help bypass the required task check so the unified benchmark CLI can render its full options. Cover the forwarded-argv path that exposed the parser ordering bug.

# Description PR isaac-sim#6505 removed the Kit extension startup that previously re-imported `isaaclab_physx` after Kit loaded. When task configuration imports the PhysX package before Kit, the compatibility hook returns early and is never retried. The original Isaac Sim `SimulationManager` STOP callback then invalidates the tensor view shared with `PhysxManager`, causing Franka and other PhysX articulation environments to fail during their first reset. This PR retries the existing idempotent compatibility takeover from `PhysxManager.initialize()`, where Kit is available and PhysX is the selected backend. It also updates the import-order contract and adds an integration regression test that imports `PhysxCfg` before launching Kit. This fix is independent of isaac-sim#6337. ## Type of change - Bug fix (non-breaking change which fixes an issue) - Documentation update ## Screenshots N/A ## Validation - Regression RED on unmodified develop: module alias remained the original Isaac Sim `SimulationManager` after `SimulationContext` initialization. - Regression GREEN: `1 passed`. - `Isaac-Reach-Franka`, 16 environments, PhysX: environment creation, reset, and first step completed with exit code 0 and without invalidated-view errors. - `./isaaclab.sh -p tools/changelog/cli.py check develop` - `./isaaclab.sh -f` ## Checklist - [x] I have read and understood the contribution guidelines - [x] I have run the pre-commit checks with `./isaaclab.sh -f` - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove the fix is effective - [x] I have added a changelog fragment for every touched package - [x] My name already exists in `CONTRIBUTORS.md`

…-user-guide # Conflicts: # docs/source/api/lab_rl/isaaclab_rl.rst # scripts/benchmarks/play.py # scripts/benchmarks/rl_games/benchmark_rl_games_play.py # scripts/benchmarks/rl_games/benchmark_rl_games_train.py # scripts/benchmarks/rsl_rl/benchmark_rsl_rl_play.py # scripts/benchmarks/rsl_rl/benchmark_rsl_rl_train.py # scripts/benchmarks/sb3/benchmark_sb3_play.py # scripts/benchmarks/sb3/benchmark_sb3_train.py # scripts/benchmarks/skrl/benchmark_skrl_play.py # scripts/benchmarks/skrl/benchmark_skrl_train.py # scripts/benchmarks/training.py # scripts/reinforcement_learning/ray/tuner.py # source/isaaclab_rl/isaaclab_rl/entrypoints/api.py # source/isaaclab_rl/isaaclab_rl/entrypoints/dispatch.py # source/isaaclab_rl/test/test_entrypoints.py

Keep typed training and play requests equivalent to the CLI added by PR isaac-sim#6474. Translate warm-up steps through dispatch so callers do not need backend-specific arguments.

Bring the latest PR isaac-sim#6564 typed training and play request contract into the benchmark documentation stack.

Define the stable three-page benchmark journey, validated Python API examples, cross-PR audit scope, and the boundary for a later micro-benchmark documentation pass.

Break the approved design into validated workflow probes, supported API documentation, framework extension guidance, navigation integration, and final PR verification.

Scope sys.argv to the typed request while benchmark workflows run so nested AppLauncher parsing sees the same arguments as the workflow entry point. Restore the caller arguments after dispatch.

Lead with the supported typed Python workflows, document verified request and result contracts, and provide an executable lower-level extension example. Align API reference coverage and testing navigation with the three-part benchmark journey.

greptile-apps · 2026-07-20T15:21:18Z

Too many files changed for review. (150 files found, 100 file limit)

Bypass the limit by tagging @greptile-apps to review.

AntoineRichard and others added 30 commits July 13, 2026 09:48

Improve runtime benchmark sampling

97a00b1

Exclude configurable warmup frames from runtime metrics and compute throughput from total measured work over total wall time. Use longer defaults so the quick benchmark is less sensitive to startup and short sampling windows.

Measure simulation-step host time

8d0b99d

Add an opt-in runtime benchmark probe around the environment's SimulationContext step calls. Report host time inside and outside that boundary while preserving the original method after measurement.

Report synchronized benchmark overhead

7fb24c1

Measure environment and synchronized simulation stepping across runtime, training, and play benchmarks. Store the resulting Isaac Lab overhead in benchmark schema 1.1 while leaving regular training and play entry points uninstrumented.

Make synchronized benchmark timing optional

9521321

Require an explicit benchmark flag before wrapping environment and simulation steps. Disabled benchmarks now avoid method replacement and device synchronization while schema 1.1 continues to represent measured runs.

Rename benchmark overhead flag

ee04ef9

Name the opt-in instrumentation after the Isaac Lab overhead it reports instead of the implementation detail used to derive it.

Exclude pending work from overhead timing

4a59a05

Synchronize before environment and simulation timing begins so policy and action-generation work queued earlier is not attributed to Isaac Lab overhead.

Center throughput deviation on effective mean

ab28226

Recalculate throughput standard deviation around the aggregate effective mean so benchmark MeanStd values use one consistent center.

Reject empty benchmark workloads

e648170

Validate workload counts before launch and require positive environment timing samples so empty synchronized runs fail with actionable errors instead of division by zero.

Cover optional benchmark timing paths

7a2a118

Run the training and play smoke matrix without synchronized overhead measurement, retain one focused synchronized replay, and verify pending work before env.step is excluded from its metric.

Stabilize benchmark timing diagnostics

c292852

Keep the first runtime step outside throughput regardless of overhead instrumentation and floor derived overhead at zero when nested wall timers cross due to measurement noise.

Refactor train and play into rl subpackage

35a3777

Fix changelog fragment

2e8f35a

Merge RL entrypoint refactor dependency

b763ab5

Bring PR isaac-sim#6553 on top of the benchmark timing work from PR isaac-sim#6474 so the benchmark library can dispatch the new RL entrypoints directly.

Fix articulation data micro-benchmarks

4421ee0

Use one device-wide synchronization per timing boundary and keep mock backend state preallocated between samples. Initialize ordering and dynamics buffers so PhysX and Newton accessors are measured without hidden skips or allocation work.

Centralize deprecated benchmark aliases

63092f3

Replace per-file Python shims with lazy module proxies so legacy imports remain compatible while Git can recognize the benchmark implementation moves. Retain type forwarding stubs and document removal in Isaac Lab 3.1.

Fix OVPhysX mock structured reads

08abc71

Preserve the destination Warp dtype when copying mock binding data so structured pose and spatial-vector buffers can be exercised by asset data benchmarks.

Add OVPhysX asset micro-benchmarks

095acc4

Cover articulation, rigid object, and rigid object collection method and data access performance using kitless mock bindings comparable to the Newton benchmark matrix.

Format OVPhysX asset benchmarks

a12af09

Apply the repository Ruff format to the newly tracked benchmark scripts and keep fill-ratio selection conditions within the line-length limit.

Add OVPhysX sensor micro-benchmarks

8628973

Measure synchronized updates, submission latency, and backend reads for contact, frame-transformer, IMU, PVA, joint-wrench, and ray-caster sensors without charging physics stepping to sensor timing.

Add Newton sensor micro-benchmarks

827345e

Document micro-benchmark workflows

b9d681f

Explain why isolated asset and sensor benchmarks exist, how to run each backend workload, and how to interpret and compare their outputs.

Merge micro-benchmark documentation stack

9970c5e

# Conflicts: # docs/source/testing/micro_benchmarks.rst # scripts/benchmarks/benchmark_load_robot.py

Allow benchmark help without task

e260c36

Make runtime and startup help bypass the required task check so the unified benchmark CLI can render its full options. Cover the forwarded-argv path that exposed the parser ordering bug.

AntoineRichard added 16 commits July 17, 2026 16:35

Rewrite benchmark user guide

216ce31

Explain startup iteration cost

6031ec5

Fix benchmark schema provenance

f02ca89

Add benchmark workflow examples

64a9295

Merge commit 'fba71b97d1a' into antoiner/benchmark-library-api

ce78a6e

Expose RL benchmark warm-up in Python API

ed5eae8

Keep typed training and play requests equivalent to the CLI added by PR isaac-sim#6474. Translate warm-up steps through dispatch so callers do not need backend-specific arguments.

Merge benchmark API warm-up update

ba72445

Bring the latest PR isaac-sim#6564 typed training and play request contract into the benchmark documentation stack.

Design advanced benchmark framework guide

777c463

Define the stable three-page benchmark journey, validated Python API examples, cross-PR audit scope, and the boundary for a later micro-benchmark documentation pass.

Plan advanced benchmark framework guide

63b2e63

Break the approved design into validated workflow probes, supported API documentation, framework extension guidance, navigation integration, and final PR verification.

Fix typed benchmark launcher arguments

ae91002

Scope sys.argv to the typed request while benchmark workflows run so nested AppLauncher parsing sees the same arguments as the workflow entry point. Restore the caller arguments after dispatch.

Merge typed benchmark dispatch fix

e6eeb54

Rewrite advanced benchmark framework guide

a218391

Lead with the supported typed Python workflows, document verified request and result contracts, and provide an executable lower-level extension example. Align API reference coverage and testing navigation with the three-part benchmark journey.

Merge typed benchmark dispatch fix

aa37b66

Merge updated benchmark user guide

ab2800b

github-actions Bot added documentation Improvements or additions to documentation isaac-lab Related to Isaac Lab team labels Jul 20, 2026

This was referenced Jul 20, 2026

Unify benchmark documentation journey #6629

Open

Rewrite benchmark user guide #6595

Open

AntoineRichard marked this pull request as ready for review July 20, 2026 15:21

AntoineRichard requested a review from a team July 20, 2026 15:21

AntoineRichard requested review from StafaH, kellyguo11, marcodiiga, matthewtrepte, myurasov-nv, ooctipus and pv-nvidia as code owners July 20, 2026 15:21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Rewrite advanced benchmark framework guide#6627

Rewrite advanced benchmark framework guide#6627
AntoineRichard wants to merge 51 commits into
isaac-sim:developfrom
AntoineRichard:antoiner/benchmark-framework-guide

AntoineRichard commented Jul 20, 2026 •

edited

Loading

Uh oh!

greptile-apps Bot commented Jul 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

AntoineRichard commented Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Dependencies

Type of change

Validation

Checklist

Uh oh!

greptile-apps Bot commented Jul 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

AntoineRichard commented Jul 20, 2026 •

edited

Loading