Merge branch 'main' into feat/runname_display

Author: ClemensSchwarke

.flake8

@@ -1,40 +1,21 @@
 repos:
-  - repo: https://github.com/python/black
-    rev: 23.10.1
+  - repo: https://github.com/astral-sh/ruff-pre-commit
+    rev: v0.14.0
     hooks:
-      - id: black
-        args: ["--line-length", "120", "--preview"]
-  - repo: https://github.com/pycqa/flake8
-    rev: 6.1.0
-    hooks:
-      - id: flake8
-        additional_dependencies: [flake8-simplify, flake8-return]
+      - id: ruff-check
+      - id: ruff-format
   - repo: https://github.com/pre-commit/pre-commit-hooks
     rev: v4.5.0
     hooks:
-      - id: trailing-whitespace
       - id: check-symlinks
       - id: destroyed-symlinks
       - id: check-yaml
+      - id: check-toml
       - id: check-merge-conflict
       - id: check-case-conflict
       - id: check-executables-have-shebangs
-      - id: check-toml
-      - id: end-of-file-fixer
       - id: check-shebang-scripts-are-executable
       - id: detect-private-key
-      - id: debug-statements
-  - repo: https://github.com/pycqa/isort
-    rev: 5.12.0
-    hooks:
-      - id: isort
-        name: isort (python)
-        args: ["--profile", "black", "--filter-files"]
-  - repo: https://github.com/asottile/pyupgrade
-    rev: v3.15.0
-    hooks:
-      - id: pyupgrade
-        args: ["--py37-plus"]
   - repo: https://github.com/codespell-project/codespell
     rev: v2.2.6
     hooks:

.pre-commit-config.yaml

@@ -0,0 +1,38 @@
+cff-version: 1.2.0
+title: "RSL-RL: A Learning Library for Robotics Research"
+message: "If you use this work, please cite the following paper."
+repository-code: "https://github.com/leggedrobotics/rsl_rl"
+license: BSD-3-Clause
+version: 3.1.3
+type: software
+authors:
+  - family-names: Schwarke
+    given-names: Clemens
+  - family-names: Mittal
+    given-names: Mayank
+  - family-names: Rudin
+    given-names: Nikita
+  - family-names: Hoeller
+    given-names: David
+keywords:
+  - reinforcement learning
+  - robotics
+  - control
+  - RSL-RL
+preferred-citation:
+  type: article
+  authors:
+    - family-names: Schwarke
+      given-names: Clemens
+    - family-names: Mittal
+      given-names: Mayank
+    - family-names: Rudin
+      given-names: Nikita
+    - family-names: Hoeller
+      given-names: David
+    - family-names: Hutter
+      given-names: Marco
+  title: "RSL-RL: A Learning Library for Robotics Research"
+  journal: "arXiv preprint"
+  doi: 10.48550/arXiv.2509.10771
+  url: "https://arxiv.org/abs/2509.10771"

CITATION.cff

@@ -17,12 +17,14 @@ Please keep the lists sorted alphabetically.
 
 ---
 
-* Mayank Mittal
 * Clemens Schwarke
+* Mayank Mittal
 
 ## Authors
 
+* Clemens Schwarke
 * David Hoeller
+* Mayank Mittal
 * Nikita Rudin
 
 ## Contributors

CONTRIBUTORS.md

@@ -1,14 +1,15 @@
-# RSL RL
+# RSL-RL
 
-A fast and simple implementation of RL algorithms, designed to run fully on GPU.
-This code is an evolution of `rl-pytorch` provided with NVIDIA's Isaac Gym.
+A fast and simple implementation of learning algorithms for robotics. For an overview of the library please have a look at https://arxiv.org/pdf/2509.10771.
 
 Environment repositories using the framework:
 
 * **`Isaac Lab`** (built on top of NVIDIA Isaac Sim): https://github.com/isaac-sim/IsaacLab
-* **`Legged-Gym`** (built on top of NVIDIA Isaac Gym): https://leggedrobotics.github.io/legged_gym/
+* **`Legged Gym`** (built on top of NVIDIA Isaac Gym): https://leggedrobotics.github.io/legged_gym/
+* **`MuJoCo Playground`** (built on top of MuJoCo MJX and Warp): https://github.com/google-deepmind/mujoco_playground/
+* **`mjlab`** (built on top of MuJoCo Warp): https://github.com/mujocolab/mjlab
 
-The main branch supports **PPO** and **Student-Teacher Distillation** with additional features from our research. These include:
+The library currently supports **PPO** and **Student-Teacher Distillation** with additional features from our research. These include:
 
 * [Random Network Distillation (RND)](https://proceedings.mlr.press/v229/schwarke23a.html) - Encourages exploration by adding
   a curiosity driven intrinsic reward.
@@ -21,8 +22,6 @@ information.
 **Affiliation**: Robotic Systems Lab, ETH Zurich & NVIDIA <br/>
 **Contact**: cschwarke@ethz.ch
 
-> **Note:** The `algorithms` branch supports additional algorithms (SAC, DDPG, DSAC, and more). However, it isn't currently actively maintained.
-
 
 ## Setup
 
@@ -56,8 +55,7 @@ For documentation, we adopt the [Google Style Guide](https://sphinxcontrib-napol
 We use the following tools for maintaining code quality:
 
 - [pre-commit](https://pre-commit.com/): Runs a list of formatters and linters over the codebase.
-- [black](https://black.readthedocs.io/en/stable/): The uncompromising code formatter.
-- [flake8](https://flake8.pycqa.org/en/latest/): A wrapper around PyFlakes, pycodestyle, and McCabe complexity checker.
+- [ruff](https://github.com/astral-sh/ruff): An extremely fast Python linter and code formatter, written in Rust.
 
 Please check [here](https://pre-commit.com/#install) for instructions to set these up. To run over the entire repository, please execute the following command in the terminal:
 
@@ -70,20 +68,14 @@ pre-commit run --all-files
 
 ## Citing
 
-**We are working on writing a white paper for this library.** Until then, please cite the following work
-if you use this library for your research:
+If you use this library for your research, please cite the following work:
 
 ```text
-@InProceedings{rudin2022learning,
-  title = 	 {Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning},
-  author =       {Rudin, Nikita and Hoeller, David and Reist, Philipp and Hutter, Marco},
-  booktitle = 	 {Proceedings of the 5th Conference on Robot Learning},
-  pages = 	 {91--100},
-  year = 	 {2022},
-  volume = 	 {164},
-  series = 	 {Proceedings of Machine Learning Research},
-  publisher =    {PMLR},
-  url = 	 {https://proceedings.mlr.press/v164/rudin22a.html},
+@article{schwarke2025rslrl,
+  title={RSL-RL: A Learning Library for Robotics Research},
+  author={Schwarke, Clemens and Mittal, Mayank and Rudin, Nikita and Hoeller, David and Hutter, Marco},
+  journal={arXiv preprint arXiv:2509.10771},
+  year={2025}
 }
 ```
 

README.md

@@ -1,21 +1,21 @@
 runner:
   class_name: OnPolicyRunner
-  # -- general
-  num_steps_per_env: 24  # number of steps per environment per iteration
-  max_iterations: 1500  # number of policy updates
+  # General
+  num_steps_per_env: 24  # Number of steps per environment per iteration
+  max_iterations: 1500  # Number of policy updates
   seed: 1
-  # -- observations
-  obs_groups: {"policy": ["policy"], "critic": ["policy", "privileged"]} # maps observation groups to types. See `vec_env.py` for more information
-  # -- logging parameters
-  save_interval: 50  # check for potential saves every `save_interval` iterations
+  # Observations
+  obs_groups: {"policy": ["policy"], "critic": ["policy", "privileged"]} # Maps observation groups to sets. See `vec_env.py` for more information
+  # Logging parameters
+  save_interval: 50  # Check for potential saves every `save_interval` iterations
   experiment_name: walking_experiment
   run_name: ""
-  # -- logging writer
+  # Logging writer
   logger: tensorboard  # tensorboard, neptune, wandb
   neptune_project: legged_gym
   wandb_project: legged_gym
 
-  # -- policy
+  # Policy
   policy:
     class_name: ActorCritic
     activation: elu
@@ -25,45 +25,46 @@ runner:
     critic_hidden_dims: [256, 256, 256]
     init_noise_std: 1.0
     noise_std_type: "scalar"  # 'scalar' or 'log'
+    state_dependent_std: false
 
-  # -- algorithm
+  # Algorithm
   algorithm:
     class_name: PPO
-    # -- training
+    # Training
     learning_rate: 0.001
     num_learning_epochs: 5
     num_mini_batches: 4  # mini batch size = num_envs * num_steps / num_mini_batches
     schedule: adaptive  # adaptive, fixed
-    # -- value function
+    # Value function
     value_loss_coef: 1.0
     clip_param: 0.2
     use_clipped_value_loss: true
-    # -- surrogate loss
+    # Surrogate loss
     desired_kl: 0.01
     entropy_coef: 0.01
     gamma: 0.99
     lam: 0.95
     max_grad_norm: 1.0
-    # -- miscellaneous
+    # Miscellaneous
     normalize_advantage_per_mini_batch: false
 
-    # -- random network distillation
+    # Random network distillation
     rnd_cfg:
-        weight: 0.0  # initial weight of the RND reward
-        weight_schedule: null # note: this is a dictionary with a required key called "mode". Please check the RND module for more information
-        reward_normalization: false  # whether to normalize RND reward
-        # -- learning parameters
-        learning_rate: 0.001  # learning rate for RND
-        # -- network parameters
-        num_outputs: 1  # number of outputs of RND network. Note: if -1, then the network will use dimensions of the observation
-        predictor_hidden_dims: [-1] # hidden dimensions of predictor network
-        target_hidden_dims: [-1]  # hidden dimensions of target network
+        weight: 0.0  # Initial weight of the RND reward
+        weight_schedule: null # This is a dictionary with a required key called "mode". Please check the RND module for more information
+        reward_normalization: false  # Whether to normalize RND reward
+        # Learning parameters
+        learning_rate: 0.001  # Learning rate for RND
+        # Network parameters
+        num_outputs: 1  # Number of outputs of RND network. Note: if -1, then the network will use dimensions of the observation
+        predictor_hidden_dims: [-1] # Hidden dimensions of predictor network
+        target_hidden_dims: [-1]  # Hidden dimensions of target network
 
-    # -- symmetry augmentation
+    # Symmetry augmentation
     symmetry_cfg:
-      use_data_augmentation: true  # this adds symmetric trajectories to the batch
-      use_mirror_loss: false  # this adds symmetry loss term to the loss function
-      data_augmentation_func: null # string containing the module and function name to import
+      use_data_augmentation: true  # This adds symmetric trajectories to the batch
+      use_mirror_loss: false  # This adds symmetry loss term to the loss function
+      data_augmentation_func: null # String containing the module and function name to import
       # Example: "legged_gym.envs.locomotion.anymal_c.symmetry:get_symmetric_states"
       #
       # .. code-block:: python
@@ -73,4 +74,4 @@ runner:
       #        obs: Optional[torch.Tensor] = None, actions: Optional[torch.Tensor] = None, cfg: "BaseEnvCfg" = None, obs_type: str = "policy"
       #     ) -> Tuple[torch.Tensor, torch.Tensor]:
       #
-      mirror_loss_coeff: 0.0 #coefficient for symmetry loss term. If 0, no symmetry loss is used
+      mirror_loss_coeff: 0.0 # Coefficient for symmetry loss term. If 0, no symmetry loss is used

config/example_config.yaml

@@ -1,6 +1,6 @@
-The MIT License (MIT)
+MIT License
 
-Copyright (c) 2018 Łukasz Langa
+Copyright (c) Microsoft Corporation
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal