Add GSPOLoss (Group Sequence Policy Optimization)

[smellslikeml]

Add GSPOLoss — Group Sequence Policy Optimization

Draft — internal review on the fork. Review artifact + validation-service test vehicle; not an upstream submission. Upstream to pytorch/rl is gated on the Facebook CLA (a human signs + submits) and a rebase-check against the fast-moving grpo.py PRs (pytorch#4363 / pytorch#4340).

What. Adds GSPOLoss to torchrl/objectives/llm/grpo.py — the sequence-level policy-optimization objective from GSPO (Group Sequence Policy Optimization, Qwen — arXiv:2507.18071). GSPO defines the importance ratio at the sequence likelihood level (a length-normalized / geometric-mean log-ratio over the response tokens) and clips at the sequence level, vs GRPO's token-level ratio. The objectives/llm/ surface has GRPO/DAPO/CISPO (all token-level) but no sequence-level mode.

Provenance / license. The sequence-ratio math is ported with attribution from the importance_sampling_level == "sequence" branch of huggingface/trl (Apache-2.0 → compatible with TorchRL's MIT). Credited to the trl source + the paper in the class docstring. No trl trainer/Liger machinery copied.

Design (house style — variant-per-subclass, mirrors CISPOLoss).

• class GSPOLoss(GRPOLoss) + class GSPOLossOutput(LLMLossOutput) added in grpo.py (not a new file, not a string mode); output_type set accordingly.
• Overrides _log_weight: takes the inherited token-level log-ratio + response mask from super()._log_weight(...), reduces to a per-sequence length-normalized log-ratio (tok_log_weight · mask).sum(-1) / mask.sum(-1).clamp(1), and broadcasts it back over the sequence — so sequence-level clipping is inherited for free from GRPOLoss._compute_policy_objective (not re-implemented).
• No new accepted keys (inherits _AcceptedKeys / set_keys()); GRPO/DAPO/CISPO behavior untouched; pure PyTorch/TensorDict, no new deps.

Validation.

• CPU synthetic-TensorDict test in test/llm/test_llm_objectives.py, mirroring test_cispo_clips_importance_weight (with an inline batch-2 ragged-mask TensorDict, since _policy_loss_data hardcodes a single all-ones sequence), using the trl formula as the in-test oracle.
• Independently reproduced: the _log_weight reduction is bit-identical to the trl reference formula on a ragged-mask batch, padded tokens are correctly excluded, and the cancellation property holds (token ratios 1.2 and 1/1.2 → sequence ratio exactly 1.0).
• Deferred (GPU/real-model): full RL-training convergence + the paper's MoE-stability headline — see VALIDATION.md.

Scope. 6 files: the loss + output class in grpo.py, its objectives/llm/__init__.py export, the test, two docs entries (llms_objectives.rst + the llms.rst index), and VALIDATION.md. No changes to the trainer, collectors, envs, or existing objectives.

Attribution. Drafted with Outrider (remyx-ai[bot], GLM-5.2) + Claude Code; human-reviewed and the sequence-ratio math independently validated. Co-authored on the commit.

[github-actions]

⚠️ PR Title Label Error

PR title must start with a label prefix in brackets (e.g., [BugFix]).

Current title: Add GSPOLoss (Group Sequence Policy Optimization)

Supported Prefixes (case-sensitive)

Your PR title must start with exactly one of these prefixes:

Prefix	Label Applied	Example
[Algorithm]	new algo	[Algorithm] Add new RL objective
[BE]	BE	[BE] Improve error messages
[Benchmark] or [Benchmarks]	Benchmarks	[Benchmark] Add collector benchmark
[BugFix]	BugFix	[BugFix] Fix memory leak in collector
[Example] or [Examples]	Examples	[Example] Add training script
[Feature]	Feature	[Feature] Add new optimizer
[Doc] or [Docs]	Documentation	[Doc] Update installation guide
[Refactor]	Refactoring	[Refactor] Clean up module imports
[CI]	CI	[CI] Fix workflow permissions
[Test] or [Tests]	Tests	[Tests] Add unit tests for buffer
[Trainer] or [Trainers]	Trainers	[Trainer] Add trainer config
[Environment] or [Environments]	Environments	[Environments] Add Gymnasium support
[Data]	Data	[Data] Fix replay buffer sampling
[LLM]	llm/	[LLM] Add reward model integration
[Minor]	small change	[Minor] Fix typo in error message
[Performance] or [Perf]	Performance	[Performance] Optimize tensor ops
[BC-Breaking]	bc breaking	[BC-Breaking] Remove deprecated API
[Deprecation]	Deprecation	[Deprecation] Mark old function
[Algorithm] or [Algorithms]	new algo	[Algorithm] Add new objective
[Quality]	Quality	[Quality] Fix typos and add codespell
[Versioning]	versioning	[Versioning] Bump release version
[WIP]	WIP	[WIP] Draft implementation

Note: Common variations like singular/plural are supported (e.g., [Doc] or [Docs]).