Alibaba's Qwen team published the algorithm that actually made Qwen3's reinforcement learning scale: GSPO (Group Sequence Policy Optimization). If you've worked with GRPO (the algorithm DeepSeek introduced and most Chinese labs adopted), you know it has a dirty secret — it collapses during long training runs, especially with MoE models. GSPO is Qwen's answer: sequence-level clipping instead of token-level, which turns out to solve three problems at once.

This isn't an incremental improvement paper. GSPO is the core RL algorithm behind every Qwen3 model — Instruct, Coder, and Thinking variants. The paper finally explains why Qwen3's reasoning capabilities felt like a step change from Qwen2.5.

What's New

The Problem with GRPO at Scale

GRPO (Group Relative Policy Optimization) works well for short training runs. But when you try to scale it — longer training, bigger models, MoE architectures — three failure modes emerge:

1. Irreversible model collapse. During extended training, GRPO's token-level importance ratios accumulate noise. Individual token probabilities can spike or crater, and once the model enters a collapse regime, it doesn't recover. This puts a hard ceiling on how much compute you can throw at RL.

2. MoE expert activation volatility. In MoE models, which experts activate for a given input can change between the old policy (used for generation) and the current policy (being optimized). GRPO computes token-level ratios assuming stable routing — when routing changes, the ratios become meaningless. Qwen previously patched this with "Routing Replay" (cache and replay expert activation patterns), but that adds memory overhead and limits MoE capacity.

3. Precision sensitivity. Token-level optimization is sensitive to floating-point precision differences between the generation infrastructure and the training infrastructure. This forces teams to maintain identical precision across their entire stack, complicating deployment.

Architecture

GSPO: Sequence-Level Everything

GSPO's core idea is to compute the importance ratio at the sequence level rather than the token level:

$$s_i(\theta) = \left(\frac{\pi_\theta(y_i|x)}{\pi_{\theta_{old}}(y_i|x)}\right)^{1/|y_i|}$$

This is the geometric mean of per-token ratios, normalized by sequence length. Then clipping, rewarding, and optimization all happen at this sequence level.

Why does this single change fix all three problems?

Collapse prevention: Sequence-level ratios smooth out individual token spikes. A single degenerate token probability gets diluted across the full sequence length, preventing the cascade that leads to collapse.

MoE stability: When expert routing changes between old and current policy, individual token probabilities can shift dramatically. But the overall sequence likelihood is much more stable — routing changes that increase some token probabilities tend to decrease others. GSPO inherently averages this out, eliminating the need for Routing Replay entirely.

Precision tolerance: Small floating-point differences at the token level compound across long sequences in GRPO. At the sequence level, these differences largely cancel out, making GSPO fundamentally more tolerant of precision mismatches between generation and training infrastructure.

The Counterintuitive Result

Here's what's surprising: GSPO clips two orders of magnitude more tokens than GRPO (measured by what fraction of tokens have their gradients clipped). Conventional wisdom says more clipping = less learning signal = worse performance. But GSPO achieves higher training efficiency despite this.

The explanation: GRPO's token-level optimization objective is noisy and inefficient. Many tokens that GRPO lets through with unclipped gradients are providing misleading learning signals. GSPO's sequence-level approach is more aggressive about clipping, but the signals it does pass through are more reliable. Quality over quantity.

Benchmarks

Scaling Results

The paper demonstrates GSPO on Qwen3-30B-A3B (a MoE model) compared against GRPO:

The key claim: GSPO delivers continuous performance improvement through increasing training compute, regularly updating the query set, and extending generation length. This is what "scalable RL" actually means — you can keep spending compute and keep getting returns.

Benchmarks on the cold-start model (fine-tuned from Qwen3-30B-A3B-Base):

What It Means for Developers

If you're doing RL post-training: GSPO is a drop-in replacement for GRPO with the same basic structure (sample group, compute advantages, clip ratios). The change is computing ratios at sequence level instead of token level. Implementation complexity is comparable.

If you're training MoE models with RL: This eliminates the need for Routing Replay, which was a significant infrastructure burden. You can now train MoE models with standard RL infrastructure.

If you're scaling RL compute: GSPO's key property is that it doesn't collapse. You can train longer, with more compute, and expect continued improvement. This changes the economics of RL post-training — it's no longer "find the sweet spot before collapse" but "spend compute, get performance."

For Qwen3 users: This explains why Qwen3's reasoning models (especially Thinking) feel notably stronger than Qwen2.5 — the RL training could actually scale to where the compute budget wanted to take it.

Developer Impact

Bottom Line

GSPO is one of those papers where the core idea is simple (sequence-level instead of token-level) but the implications cascade: no more training collapse, no more MoE instability, simpler infrastructure requirements. It powered all of Qwen3's RL training and the results speak for themselves. If you're doing RL post-training on large models — especially MoE — this should be your starting algorithm. Paper only for now, no code released, but the algorithm is straightforward to implement from the paper.

Resources

GSPO powers the RL training of all Qwen3 models, available via the Qwen API and on HuggingFace.