Unify benchmark documentation journey#6629
Open
AntoineRichard wants to merge 67 commits into
Open
Conversation
Exclude configurable warmup frames from runtime metrics and compute throughput from total measured work over total wall time. Use longer defaults so the quick benchmark is less sensitive to startup and short sampling windows.
Add an opt-in runtime benchmark probe around the environment's SimulationContext step calls. Report host time inside and outside that boundary while preserving the original method after measurement.
Measure environment and synchronized simulation stepping across runtime, training, and play benchmarks. Store the resulting Isaac Lab overhead in benchmark schema 1.1 while leaving regular training and play entry points uninstrumented.
Require an explicit benchmark flag before wrapping environment and simulation steps. Disabled benchmarks now avoid method replacement and device synchronization while schema 1.1 continues to represent measured runs.
Delegate synchronization and elapsed-time measurement to the existing Isaac Lab Timer. Keep the interception layer focused on recording per-step samples so environment, simulation, and derived overhead metrics include standard deviation and peak values.
Record the env.step boundary in every benchmark workflow without adding explicit synchronization. Keep the nested synchronized simulation probe optional, and expose environment-step FPS in schema 1.2 so runtime, play, and training results can be compared on the same scope.
Name the opt-in instrumentation after the Isaac Lab overhead it reports instead of the implementation detail used to derive it.
Synchronize before environment and simulation timing begins so policy and action-generation work queued earlier is not attributed to Isaac Lab overhead.
Recalculate throughput standard deviation around the aggregate effective mean so benchmark MeanStd values use one consistent center.
Validate workload counts before launch and require positive environment timing samples so empty synchronized runs fail with actionable errors instead of division by zero.
Run the training and play smoke matrix without synchronized overhead measurement, retain one focused synchronized replay, and verify pending work before env.step is excluded from its metric.
Keep the first runtime step outside throughput regardless of overhead instrumentation and floor derived overhead at zero when nested wall timers cross due to measurement noise.
Rename the disputed overhead metric as a serialized synchronized step breakdown and expose explicit measurement modes. Mark every instrumented timing and rate as diagnostic, validate the additive partition, and document why outside-simulation time is not removable overhead.
Follow-up polish from review of the runtime/synchronized-timing changes; no behavior change. - Drop the redundant ``dest=``/``default=False`` on runtime.py's ``--measure_synchronized_step_breakdown`` (argparse derives both) so the flag matches the eight RL adapter scripts. - Remove the single-use ``timer_context`` alias and enter the step-timing recorder alongside ``BenchmarkMonitor`` on one line, matching the adapters' context-manager pattern. - Reorder the ``build_runtime`` Args block to signature order (``aggregate_throughput`` before ``frames_per_environment_step``). - Clarify ``MeanStd.mean``/``std``: for effective-throughput fields these are the aggregate rate and the dispersion about the effective mean, not the arithmetic sample mean and standard deviation.
Bring PR isaac-sim#6553 on top of the benchmark timing work from PR isaac-sim#6474 so the benchmark library can dispatch the new RL entrypoints directly.
Move the supported benchmark workflows into isaaclab.benchmark and expose typed in-process dispatch for runtime, startup, training, and play benchmarks. Keep deprecated script and test namespace shims so existing callers can migrate without an immediate break.
Use one device-wide synchronization per timing boundary and keep mock backend state preallocated between samples. Initialize ordering and dynamics buffers so PhysX and Newton accessors are measured without hidden skips or allocation work.
Replace per-file Python shims with lazy module proxies so legacy imports remain compatible while Git can recognize the benchmark implementation moves. Retain type forwarding stubs and document removal in Isaac Lab 3.1.
Add a warmup_steps parameter to EnvironmentStepTimingRecorder that drops the first N ``env.step()`` calls (CUDA graph capture, lazy kernel compilation) from the recorded environment-step timing. In serialized mode the skipped steps' simulation call-count and time are rolled back so the recorded aggregate stays consistent. Expose it as an opt-in ``--warmup_steps`` flag (default 0, validated non-negative, recorded in metadata) on the eight RL train/play benchmark adapters. ``runtime.py`` is unchanged: its ``--warmup_frames`` already excludes the startup step before the recorder is installed. Add recorder unit tests for the host-return and serialized exclusion paths and a parse-level test rejecting a negative ``--warmup_steps``.
Preserve the destination Warp dtype when copying mock binding data so structured pose and spatial-vector buffers can be exercised by asset data benchmarks.
Cover articulation, rigid object, and rigid object collection method and data access performance using kitless mock bindings comparable to the Newton benchmark matrix.
Apply the repository Ruff format to the newly tracked benchmark scripts and keep fill-ratio selection conditions within the line-length limit.
Measure synchronized updates, submission latency, and backend reads for contact, frame-transformer, IMU, PVA, joint-wrench, and ray-caster sensors without charging physics stepping to sensor timing.
Explain why isolated asset and sensor benchmarks exist, how to run each backend workload, and how to interpret and compare their outputs.
# Conflicts: # scripts/benchmarks/rl_games/benchmark_rl_games_play.py # scripts/benchmarks/rl_games/benchmark_rl_games_train.py # scripts/benchmarks/rsl_rl/benchmark_rsl_rl_play.py # scripts/benchmarks/rsl_rl/benchmark_rsl_rl_train.py # scripts/benchmarks/runtime.py # scripts/benchmarks/sb3/benchmark_sb3_play.py # scripts/benchmarks/sb3/benchmark_sb3_train.py # scripts/benchmarks/skrl/benchmark_skrl_play.py # scripts/benchmarks/skrl/benchmark_skrl_train.py
# Conflicts: # docs/source/testing/micro_benchmarks.rst # scripts/benchmarks/benchmark_load_robot.py
…er/ovphysx-asset-micro-benchmarks
…ovphysx-sensor-micro-benchmarks
Share synchronized latency sampling across PhysX and OVPhysX sensor workloads. Emit summary and JSON results with observer-floor, submission, completion, and native-read statistics so runs are directly comparable and failures remain visible.
…/newton-sensor-micro-benchmarks # Conflicts: # docs/source/testing/micro_benchmarks.rst
Use the shared synchronized latency runner so Newton sensor results include consistent statistics, observer cost, metadata, and machine-readable output across backends.
Bring the guide onto the hardened benchmark stack and document the shared sensor timing boundaries, observer floor, structured outputs, and reproducibility requirements.
Keep the designated-workstation requirement public without exposing the private local SSH helper.
Integrate the micro-benchmark guide with the CLI-first user guide and advanced framework reference, and make the intended reader path explicit.
…enchmark-documentation-navigation
github-actions
Bot
added
documentation
7 tasks
AntoineRichard
marked this pull request as ready for review
July 20, 2026 15:21
AntoineRichard
requested review from
StafaH,
kellyguo11,
marcodiiga,
matthewtrepte,
myurasov-nv,
ooctipus and
pv-nvidia
as code owners
July 20, 2026 15:21
Contributor
|
Too many files changed for review. ( Bypass the limit by tagging |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Unifies the benchmark documentation into one explicit reader journey:
The testing index presents those pages in that order. Cross-references describe the measurement boundaries and prevent isolated latency from being interpreted as end-to-end throughput.
Dependencies
This is the final integration PR for the benchmark documentation journey.
Type of change
Validation
./isaaclab.sh -p -m pytest source/isaaclab/test/benchmark(184 passed)../isaaclab.sh -d(warning-free)../isaaclab.sh -fChecklist
CONTRIBUTORS.md