[LLM] WS4 Post-Training Observability: Implement PostTrainingLogger

[coder-jayp]

Description

This PR implements the Observability workstream (WS4) from RFC #3948. It introduces a modular PostTrainingLogger that unifies metric emission across TorchRL's post-training algorithms, making the observability stack consumable by external training loops.

Key Changes:

• PostTrainingLogger: New backend-agnostic logger for tracking loss components, buffer utilization, and inference staleness. Uses getattr duck-typing to safely ingest both GRPOLossOutput and SFTLossOutput.
• Recipe Refactoring: Updated GRPO and Expert Iteration (EI) recipes to delegate to the new logger.
• Deprecation Handling: EI legacy metric keys are temporarily emitted in parallel with explicit DeprecationWarnings (targeted for removal in v0.15.0 per CLAUDE.md).
• Docs & Tests: Added full Sphinx documentation and comprehensive unit testing in test_loggers.py.

Motivation and Context

Solves the need for component-level interoperability in TorchRL's LLM stack. Users can now drop this logger into their own existing loops without migrating their entire workflow.

close #3948

• I have raised an issue to propose this change (required for new features and bug fixes)

Types of changes

• New feature (non-breaking change which adds core functionality)
• Documentation (update in the documentation)
• Example (update in the folder of examples)

Checklist

• I have read the CONTRIBUTION guide (required)
• My change requires a change to the documentation.
• I have updated the tests accordingly (required for a bug fix or a new feature).
• I have updated the documentation accordingly.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/3950

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit f2a8042 with merge base 92a3f7f ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[github-actions]

⚠️ PR Title Label Error

PR title must start with a label prefix in brackets (e.g., [BugFix]).

Current title: WS4 Post-Training Observability: Implement PostTrainingLogger

Supported Prefixes (case-sensitive)

Your PR title must start with exactly one of these prefixes:

Prefix	Label Applied	Example
[Algorithm]	new algo	[Algorithm] Add new RL objective
[BE]	BE	[BE] Improve error messages
[Benchmark] or [Benchmarks]	Benchmarks	[Benchmark] Add collector benchmark
[BugFix]	BugFix	[BugFix] Fix memory leak in collector
[Example] or [Examples]	Examples	[Example] Add training script
[Feature]	Feature	[Feature] Add new optimizer
[Doc] or [Docs]	Documentation	[Doc] Update installation guide
[Refactor]	Refactoring	[Refactor] Clean up module imports
[CI]	CI	[CI] Fix workflow permissions
[Test] or [Tests]	Tests	[Tests] Add unit tests for buffer
[Trainer] or [Trainers]	Trainers	[Trainer] Add trainer config
[Environment] or [Environments]	Environments	[Environments] Add Gymnasium support
[Data]	Data	[Data] Fix replay buffer sampling
[LLM]	llm/	[LLM] Add reward model integration
[Minor]	small change	[Minor] Fix typo in error message
[Performance] or [Perf]	Performance	[Performance] Optimize tensor ops
[BC-Breaking]	bc breaking	[BC-Breaking] Remove deprecated API
[Deprecation]	Deprecation	[Deprecation] Mark old function
[Algorithm] or [Algorithms]	new algo	[Algorithm] Add new objective
[Quality]	Quality	[Quality] Fix typos and add codespell
[Versioning]	versioning	[Versioning] Bump release version
[WIP]	WIP	[WIP] Draft implementation

Note: Common variations like singular/plural are supported (e.g., [Doc] or [Docs]).