[Algorithm] GTrXL PPO with compact window replay and transformer tutorial

[vmoens]

GTrXL PPO needs its pre-window memory to remain aligned with sampled observations without allocating a transformer-memory snapshot for every replayed transition. This adds a runnable hidden-velocity CartPole implementation and transformer tutorial on top of the recurrent-state stack and the merged #4193 API.

Changes

• Export GTrXL, preserving the configurable state container and existing per-step-state path. Add parallel relative attention for compact windows with the same rolling causal horizon, GRU-style gates, episode masks and detached initial memory.
• Extend the explicit TransformerModule path to accept [B] records containing observations [B,T,F] and one [B] initial state. It returns [B,T,D] features and one final carry. Existing module-owned cache behavior remains unchanged.
• Store replay along the outer window batch dimension: initial_state.memory is [B,L,M,D], while the transitions child has batch [B,T]. Existing tensor/memmap storages and SamplerWithoutReplacement select whole records. Compute advantages before packing; mask padded positions in PPO.
• Add Hydra configuration, a SOTA smoke entry, three reproducible learning runs, a separate execution benchmark, API documentation and an executable tutorial explaining required input/output keys, shapes, primers, slice boundaries and memory costs.

Scope and dependencies

Stacked on #4326, which depends on #4325 and #4324. The stack includes the exact merged #4193 commit a2acf99c7. TensorDict #1766 has merged; #1789 remains a prerequisite. All draft CI installers pin its compiled typed-state compatibility fix at 2d252bfae380f150a6136930d7d7a5833d68ecf1. The stack has been rebased onto TorchRL main 44c871df8.

The container decision remains open: state_cls is configurable and the example offers td, tc and ttd. No existing GRU/LSTM default changes. The example's TypedTensorDict schema does not finalize a library-wide state class.

Compact replay requires whole-window sampling. Arbitrary new offsets require per-step carries, which remain supported by the existing path. Packing currently happens after standard collection, so this saves replay allocation and compact learner state, not the collector's peak memory. Supplied carry survives parameter updates and can be stale, as with recurrent replay.

Validation and measurements

• 113 transformer tests passed, including existing cache semantics, typed states, serial/parallel environments, tensor/memmap storage, nested/multidimensional batches, resets, snapshot ownership, intermediate slices, compact step/window parity, finite gradients and detached initial memory.
• All 113 transformer tests also pass with PyTorch 2.15.0.dev20260911, including the previously failing TypedTensorDict compact-window compilation. Compact windows passed eager and fullgraph aot_eager compilation, float64, bfloat16 and CPU autocast.
• The memmap SOTA smoke test includes padded windows and passed; the tutorial executed end to end; all nine GTrXL doctests passed. Full pre-commit run --all-files and diff checks passed, including flake8 plugins and Sphinx heading checks.
• Three seeds ran 501,760 transitions each. Final deterministic evaluation returns were 53.25, 46.25 and 24.0. These are preliminary, unstable learning results, not a tuned or solved CartPole baseline. Full curves and configurations are committed.
• Default replay root-memory payload is 256 KiB instead of 8 MiB (32-fold reduction). The separate CPU benchmark compares identical plain TensorDict payloads and weights; it measures execution/layout differences rather than container overhead. See benchmarks/results/gtrxl-learning.md for absolute timings and limitations.

CUDA execution of the new compact kernel has not been tested locally.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/4335

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 8e72d16 with merge base 44c871d ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.