[Refactor] MicroDuck prior PPOTrainer and PPO-EWMA recipes

[vmoens]

MicroDuck prior training now uses PPOTrainer for optimization while retaining complete-episode collection, recurrent minibatches, per-task GAE, evaluation and portable best/latest exports. The Hydra recipe selects ordinary PPO or PPO-EWMA and saves resumable learner, optimizer, proximal actor/updater, RNG, scheduler and evaluation state. Skill/prior training remains in TorchRL and requires no zoo installation.

This also fixes two concrete update errors: the proximal actor no longer advances for gradient accumulation or skipped nonfinite optimizer steps, and prior GAE reserves a bootstrap row for each complete episode instead of discarding a suffix when several episodes truncate.

Stacked above #4375 (football-example). The optional games zoo consumes the same trainer and delayed-actor objective in its own recipe.

Validation: native-MuJoCo whole-episode/per-task/recurrent/resume regressions passed; nine focused trainer/updater tests passed; the matching zoo suite passed 39 tests (one accelerated-backend skip). Installed-package PPO-EWMA tag smoke and recurrent prior PPO-EWMA smoke produced finite losses and resumable checkpoints. Fixed-rollout game tests compare GAE and an optimizer update against explicit calculations, and verify frozen opponents remain unchanged across PPO and EWMA resume. CI runs both prior recipe smoke commands and the installed zoo integration tutorial.

Simulator continuation starts from fresh episodes after restart; fixed-data optimizer continuation is checked exactly. No training comparison or claim that EWMA improves learning is made.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/4378

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 8c81e5c with merge base a2acf99 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.