Conversation
build_football_scene attaches one MicroDuck per player with mujoco.MjSpec onto a procedural pitch with walls, goals, a ball, cameras and a STAND keyframe, and returns the MJCF as text. MicroDuckFootballEnv is the multi-agent env over that scene: per-duck joint actions, MicroDuck proprioception plus team-relative match features, goal, ball-progress, approach, fall and action-rate rewards, goals terminate, fallen ducks respawn on their kickoff slot. MicroDuckSkillEnv runs a trained MicroDuck controller under a policy that picks one of its tasks per duck and per decision period. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…r as controller Shared-parameter actor and centralized critic over MicroDuckFootballEnv, self-play, KL-adaptive PPO, deterministic match evaluation, broadcast video, unified checkpoints and rlrender factories. By default the ducks are driven by the run 9 walker checkpoint through MicroDuckSkillEnv; the end-to-end joint-level mode is one flag away. ppo_mujoco.make_actor_critic rebuilds a walker without an env. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…tball scene Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
MjSpec.attach copies the child's meshes under every player's prefix, so a 5-a-side scene carried 380 meshes (2.15 M vertices, 4.3 M faces) where 38 suffice. Point the other players' geoms at the first copy and delete the rest: the compiled model keeps the same bodies, masses, inertias, poses and trajectories (checked over 50 steps) with a tenth of the mesh tables. The MuJoCo WASM viewer, which ran out of its 2 GB heap on the copies, loads the 5-a-side scene now. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The backend rendered with mujoco-torch's pure-PyTorch ray caster, which tests every ray against every mesh triangle: on the football scene (430 k triangles once the meshes are shared, 4.3 M before) a single 640 x 360 frame exhausted memory. Use the path of the MJX backend instead: copy qpos and qvel into an MjData and draw with mujoco.Renderer. Same contract, a uint8 (num_envs, H, W, 3) tensor, and the frames now match the other two backends pixel for pixel. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The football tests run on every installed backend again. The README and the env docstring now say that mujoco-torch needs a build from its main branch (the 0.2.0 release cannot step a scene with more than one duck) and that the vectorized backends are the ones for a GPU host. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ample The default walker is now the six-task, 128-unit checkpoint of the torchrl/microduck-skills Hugging Face repository (the one the skills tutorial deploys), pinned by revision and sha256, instead of the run 9 checkpoint on the fork's assets branch. The skill selection keeps the five locomotion tasks and leaves the jump out. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
TransformedEnv(TransformedEnv(env, t)) raised "Invalid transform type ... NoneType": the unwrap branch checked the type of the transform before its None guard, so the guard was dead code. Handle None first and keep the inner transforms only, as the non-unwrapping branch already did with an empty Compose. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The low-cost collision scene marks the mesh geoms as boxes and relies on the compiler flag fitaabb to fit them; MicroDuckEnv compiles that scene directly. build_football_scene attached the robot into a fresh MjSpec whose compiler never received the flag, so MuJoCo fell back to the inertia-box fit and the feet came out thinner and lower than the ones the walkers were trained on: a duck walking forward alone fell every two seconds. Copy the robot's compiler fitting flags into the football spec. The exported scene carries the fitted boxes explicitly (MjSpec.to_xml does not write fitaabb), and the compiled 1v1 model now matches MicroDuckEnv's geom for geom. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Respawning a fallen duck on its kickoff slot was a free teleport home: for
a -1 penalty a defender saved ten seconds of walking, and the slot sits in
front of its own goal. Falls now cost time and position instead: the duck
lies down for respawn_delay_s (default 1 s) with its actions ignored, then
stands up where it fell, facing the goal it attacks (respawn_mode
"in_place"; "kickoff" keeps the old rule). The fall penalty is still paid
once, and ("agents", "fallen") reports the duck as down until it stands.
The example exposes both knobs.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
MultiAgentMLP(centralized=True, share_params=True) returns one value for every agent, which cannot tell the two teams of a zero-sum match apart, so the baseline removed none of the variance of the team-signed terms. The critic now reads each duck's own observation, which already describes the whole match in the duck's team frame (policy.centralized_critic restores the previous critic). The KL-adaptive learning rate gets a ceiling (ppo.max_learning_rate, 1e-3): the first run's rate climbed to 1e-2 while the policy collapsed to standing still. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
spawn_noise, yaw_noise, joint_reset_noise_scale and ball_noise are constructor arguments of MicroDuckFootballEnv; the config now carries them so a run can spread kickoffs over the pitch or align the ducks without code changes. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…rics With the get-up delay the fallen flag stays up for every decision a duck lies on the ground, so falls_per_duck counted ten flags per fall. Count the steps on which the flag rises instead, in the evaluation and collection metrics alike. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…es not reset _reset_no_buffers filled the slots of workers that were not being reset with the caller's input minus the reset signal. The collector resets through maybe_reset, which passes the reset signal alone, so those slots came back empty and the stacked result was a ragged lazy stack: any transform reading an observation on reset failed, and iterating the keys raised. The parent now keeps the last output each worker sent (from steps and resets) and uses it, updated with whatever the caller supplied, for the workers it leaves untouched. Envs built with metadata_from_workers=True run without buffers, so this is the default path of the MuJoCo custom envs' worker batching. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The collector batch of the worker-batched env is (envs, 1, T), so the rising-edge count ran along the singleton dimension and still reported down time. The helper now takes the time dimension explicitly. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Paying every duck to approach the ball produced a scrum: all ten piled onto the ball, fell over each other and the ball rarely came out. The new crowd term charges each duck, per second, for every other duck within CROWD_RADIUS (0.2 m), so a team spreads out and collisions, the main cause of falls, become costly on their own. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Charging every nearby duck made the whole team keep away from the ball: a single opponent within reach cost more than approaching the ball paid, and the policy dispersed and stopped scoring. Contesting the ball against opponents is the game; only teammates piling up is the failure to discourage. The term now counts teammates within CROWD_RADIUS and its default weight drops to -0.2 per second per teammate. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Symmetric self-play from scratch stalls in a scrum or, with crowding penalties, in mutual avoidance: the pushing contest between two identical teams has no gradient toward scoring. policy.opponent_skill makes the red team execute one fixed skill (0: standing statues) while the shared actor still acts for blue, and ppo.train_team=blue removes red's rows from the update by zeroing their advantage and value target, the networks keeping their per-duck width. Blue can then learn to reach the ball, dribble and score before self-play resumes from that checkpoint. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Zeroing red's rows and letting the loss normalize over every row inflated blue's advantages, pushed the update KL past its target and drove the KL-adaptive learning rate to its floor within a few iterations. In train_team=blue mode the advantage is normalized over blue's rows and the loss's own normalization is off. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
OpponentPolicy replaces OpponentSkill: the red team executes either a fixed skill or a frozen copy of the actor (policy.opponent_checkpoint, "self" for the initial parameters), refreshed every ppo.opponent_refresh_interval iterations for fictitious self-play. The executed actions' log-probabilities are recomputed under the actor so every duck's importance ratio starts at one. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
MicroDuckFootballEnv(approach_players=k) pays the approach_ball term to the k ducks of each team closest to the ball, so one or two chase it instead of the whole team piling onto it. Exposed as env.approach_players in the MAPPO example. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The iteration callback now runs at the end of the iteration, so an evaluation that falls on a refresh iteration measures the policy against the previous opponent rather than against a copy of itself. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ic up MicroDuckFootballEnv(progress_players=k) pays ball_progress to the k ducks of each team closest to the ball, so the duck pushing it gets the credit instead of the whole team. The MAPPO example gains ppo.critic_warmup_iterations: iterations that update the critic alone before the actor trains, for warm-started actors. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
ppo.reference_kl_coeff adds that weight times the KL divergence from a frozen copy of the initial actor to the PPO loss, so a behavior-cloned or otherwise warm-started policy stays close to its prior while the critic and the reward shape it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
MicroDuckFootballEnv(knockout=True) terminates the match when every duck of a team is down at the same time, as a win for the other team: a new knockout output (1 blue wins, -1 red wins) and a one-off knockout reward term with the goal's default weight. With respawn=False this makes every fall count for the rest of the match. The MAPPO example exposes env.knockout and reports knockouts in its metrics. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Revision 43ffe3b725b6a6853ba8e4a54deb06ae17606e5e of torchrl/microduck-skills replaces walker.ckpt with the walker trained for football (standing, forward 0.25 m/s, backward -0.2 m/s, sidestep left and right 0.2 m/s; no jump). football.yaml pins that revision and its sha256, the README no longer describes the default football walker as the one the skills tutorial deploys, and the skills tutorial streams navigation.mp4 from the new revision (the file is unchanged there). The tutorial keeps its pinned six-skill walker and navigation pair. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ess credit With respawn disabled a duck lying next to the ball kept one of the approach_players / progress_players slots for the rest of the match, so its standing teammates earned nothing for going to the ball. Ducks that are down now rank last. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The scene declared the ball's inertial frame at the body position, so the center of mass sat at the top of the sphere: a rolling ball behaved like an eccentric wheel, speeding up from 0.5 to 1.8 m/s on its own, bouncing and dying within 13 cm. The inertial frame now sits at the body origin and the ball's rolling friction is 0.001, so a 0.5 m/s ball loses about 0.07 m/s per second and stops within a metre. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…rough it Only the robot's feet (collision class 1) collided with the pitch; the body and leg boxes are class 2 and passed through the floor, walls and posts, so a fallen duck sank 10 cm under the grass. The pitch geoms now accept both classes. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
vmoens
added
Documentation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/4375
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit 0b39a13 with merge base a2acf99 ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
meta-cla
Bot
added
the
CLA Signed
6 tasks
vmoens
added this pull request to stack #4333
September 14, 2026 07:14
vmoens
changed the title
This was referenced Sep 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
MicroDuck football now lives in the optional games zoo, keeping evolving game rules, scenes, selectors and recipes outside TorchRL. This PR provides the installed skill-loading interface and a short football walkthrough using TorchRL controllers, collection, PPOTrainer, evaluation and rendering. The linked six-game catalog tracks implemented mechanics, pipeline validation and measured learned behavior separately. Later game/sensor development uses its own pinned revisions above this initial football integration.
load_microduck_walkerreconstructs and freezes the actor from checkpoint metadata and returns its ordered task library, validating the hash and deployment action scale. Required model-construction helpers stay in TorchRL and are reused by its skill examples. No zoo code is required to train or load skills, and no runtime code importsexamples.microduck. URL-specific cache directories keep distinctwalker.ckptartifacts from colliding.The integration also preserves explicit local-collector policy mappings for learner/opponent wrappers and excludes masked samples from PPO's adaptive-KL diagnostic. Football-specific library exports, API entries, optimization code and tests move to the zoo; reusable backend/controller fixes and regressions remain here. No released API is removed: these football APIs were introduced in this unmerged PR.
Stack: remains based on #4330. Skills and prior training stay in TorchRL. The head-calibration/reward/turning work in #4361 and controller/trainer scopes in #4329/#4332 are preserved. Later skill-training, sensor and EWMA changes will stack above this PR.
Validation:
dd0dcb5bc7be41a4e4e842133dc5d224bce5cfa8; zoo CI pins the matching TorchRL implementation commitd4ae6c256ecc576deaf63b223820f9fd7159363d.Existing HF files and URLs are unchanged. Old actor exports are warm starts, while resumable trainer snapshots retain optimizer/scheduler/reference/opponent/hook state. Native MuJoCo resume starts at a fresh episode boundary. Full game tests and future game-training pilots belong to the zoo; this integration makes no new football-learning claim.