Group Sequence Policy Optimization | Summary

Contents

Summary
1 Introduction
2 Preliminaries
3 Motivation
4 Algorithm
5 Experiments and Discussion
6 Conclusion
Appendix
Brief Thoughts

This article explains the key points of Group Sequence Policy Optimization.

2025-07-24 (arXiv)
Zheng, Chujie, Liu, Shixuan, Li, Mingze, Chen, Xiong-Hui, Yu, Bowen, Gao, Chang, Dang, Kai, Liu, Yuqiong, Men, Rui, Yang, An, et al.
Qwen Team, Alibaba Inc.
Paper

Summary

Reward-based training of language models can become unstable as responses grow longer. Starting from a model fine-tuned from Qwen3-30B-A3B-Base, Group Sequence Policy Optimization (GSPO), which weights updates using whole-response probabilities, achieves better reward and benchmark performance at matched training compute and consumed queries than Group Relative Policy Optimization (GRPO), which uses per-token probability ratios, even when GRPO uses Routing Replay to reuse the data-generating model’s expert selections during updates.
GSPO combines rewards normalized within a group of responses to the same query with a length-normalized ratio of response probabilities under the current and data-generating models. It applies clipping, a rule that suppresses selected updates when this ratio crosses a threshold, to whole responses. The evaluated Mixture-of-Experts (MoE) model, which activates only a subset of its expert networks for each token, trains stably with GSPO without Routing Replay.
The reported average fraction of clipped tokens is 0.15 for GSPO and 0.0013 for GRPO, despite GSPO’s higher training efficiency. The evidence consists primarily of training curves for one named starting model; the paper does not provide numerical compute budgets, repeated-run uncertainty, or measured infrastructure savings.

1 Introduction

The introduction presents training stability as a prerequisite for scaling reinforcement learning (RL), training that improves a policy, or response-generating model, using rewards. The authors attribute GRPO collapse to noisy token-level importance weights, ratios between current-policy and old-policy token probabilities, whose effects accumulate over long responses and interact with clipping. They propose aligning the optimization unit with the whole response receiving the reward.

2 Preliminaries

The policy πθ is an autoregressive language model, meaning that each token probability is conditioned on the query and preceding tokens. For a query x and response y, its likelihood is πθ(y|x) = ∏ₜ₌₁^|y| πθ(yₜ|x,y<ₜ), and a verifier assigns \(r(x,y)\in[0,1]\). Proximal Policy Optimization (PPO), an algorithm that constrains updates using clipped probability ratios, estimates token advantages, measures of relative desirability, with a separate value model that predicts expected rewards; GRPO removes that model by assigning every token in a response the same group-normalized reward.

GRPO computes the response advantage as its reward minus the group mean, divided by the group standard deviation. G denotes the number of responses generated for each query.
The paper identifies the value model’s memory, computation, and estimation reliability as practical limitations of PPO. Its displayed objectives omit the Kullback–Leibler (KL) regularization term, a penalty for policy divergence, because this is outside the paper’s focus.

3 Motivation

Splitting a large batch of generated responses into several gradient-update mini-batches creates an off-policy setting: later updates optimize a policy different from the one that generated the data. Importance sampling, reweighting samples to account for a change in their generating distribution, motivates the probability ratios used in these updates. The authors argue that GRPO’s single sampled token at each prefix provides a noisy correction and report collapse that persisted despite checkpoint rollback, clipping adjustments, longer generation, or changed queries.

This is the authors’ proposed explanation for the observed instability. The section motivates sequence-level optimization but does not establish a general impossibility result for token-level importance sampling.

4 Algorithm

The algorithm section defines GSPO’s sequence-level objective, compares its gradient weighting with GRPO, and introduces GSPO-token for adjustable token-level advantages. Both GSPO variants use one length-normalized sequence likelihood ratio to determine the numerical importance weight for a response.

4.1 GSPO: Group Sequence Policy Optimization

For response i, GSPO defines sᵢ(θ) = (πθ(yᵢ|x)/πθold(yᵢ|x))^(1/|yᵢ|), the geometric mean of its token-level probability ratios. Its objective averages min(sᵢ(θ)Âᵢ, clip(sᵢ(θ),1−ε,1+ε)Âᵢ) over responses and queries, where Âᵢ is the group-normalized reward and ε specifies the clipping range. Clipping acts on entire responses; the authors use length normalization to reduce ratio variance and keep responses of different lengths within a comparable numerical range.

Equation (7) equivalently computes the ratio by exponentiating the mean token log-probability difference between current and old policies.
Without length normalization, probability changes at a few tokens can cause large sequence-ratio fluctuations. Because their ratio definitions differ, GSPO and GRPO typically use clipping ranges of different orders of magnitude.

4.2 Gradient Analysis

With clipping omitted from the derivation, GSPO multiplies the mean token log-likelihood gradient by the response-wide factor sᵢ(θ)Âᵢ. GRPO instead weights each token’s gradient by its own probability ratio, even though all tokens share the same response advantage. GSPO therefore gives every token within a response the same importance weight; the authors identify GRPO’s unequal weights as a source of accumulating instability.

The paper notes that GRPO’s gradient-contributing token weights can range over (0,1+ε] for positive advantages and [1−ε,+∞) for negative advantages. Clipping does not make all retained token weights nearly equal.

4.3 GSPO-token: A Token-level Objective Variant

GSPO-token allows advantages to vary by token, motivated by multi-turn RL, training on interactions containing multiple exchanges. It defines sᵢ,ₜ(θ) = sg[sᵢ(θ)]πθ(yᵢ,ₜ|x,yᵢ,<ₜ)/sg[πθ(yᵢ,ₜ|x,yᵢ,<ₜ)], where sg stops gradient propagation while retaining the numerical value, corresponding to the detach operation in the PyTorch machine learning framework. The ratio remains numerically equal to the response ratio while gradients flow through the individual token probability.

When all token advantages equal the response advantage, GSPO-token and GSPO have identical objective values, clipping conditions, and theoretical gradients. The paper derives this equivalence but does not report a separate experiment using customized token advantages.

5 Experiments and Discussion

The experiments compare training reward and mathematics and programming performance, then examine clipping fractions and changes in expert selection. The infrastructure discussion proposes avoiding recomputation of old-policy likelihoods but does not report a systems benchmark.

5.1 Empirical Results

The starting point is a cold-start model fine-tuned from Qwen3-30B-A3B-Base, with each batch of generated responses divided into four mini-batches for gradient updates. Evaluation uses AIME’24, a mathematics benchmark, with average Pass@1, single-attempt success rate, over 32 samplings; LiveCodeBench, a programming benchmark, for 202410–202502 with average Pass@1 over 8 samplings; and CodeForces, a programming benchmark, with Elo Rating, a competitive performance rating. GSPO uses left and right clipping ranges of 3e-4 and 4e-4, respectively, while the tuned GRPO baseline uses 0.2 and 0.27 and includes Routing Replay.

Figure 1 shows higher reward and benchmark performance for GSPO at comparable training compute. The text states that the comparison also matches consumed queries and that GSPO continues improving as compute increases, queries are updated, and generation length is extended.
The compute axes have no numerical tick labels, and the paper provides no tabulated endpoint scores or uncertainty estimates. The curves support a qualitative efficiency advantage without establishing an exact compute saving.

Training reward and benchmark performance versus training compute for a cold-start model fine-tuned from Qwen3-30B-A3B-Base, comparing GSPO with GRPO using Routing Replay.

5.2 Curious Observation on Clipping Fractions

Figure 2 reports average clipped-token fractions of 0.15 for GSPO and 0.0013 for GRPO, a difference of roughly two orders of magnitude. GSPO achieves higher training efficiency despite excluding a much larger fraction of tokens from gradient estimation through response-level clipping. The authors interpret this as evidence of less noisy learning signals, although clipping fractions alone do not directly measure gradient variance.

Average fractions of clipped tokens during GSPO and GRPO reinforcement learning training.

5.3 Benefit of GSPO for MoE Training

The authors report expert-activation volatility, changes in the experts selected for the same response after policy updates. For the 48-layer Qwen3-30B-A3B-Base model, roughly 10% of activated experts differ between old and new policies after each RL gradient update for the same generated sample. Routing Replay stabilizes GRPO by caching old-policy expert selections and reusing them when computing current-policy token ratios; Figure 3 shows reward improving with this strategy and declining without it.

Reusing routing decisions incurs memory and communication overhead and can constrain the model’s available expert capacity, according to the authors.
GSPO trains stably without Routing Replay in Figure 1. The authors attribute this to reduced sensitivity of the sequence-level ratio to individual-token likelihood fluctuations.

GRPO training reward versus training compute with and without Routing Replay for an MoE model.

5.4 Benefit of GSPO for RL Infrastructure

Training engines, software systems that execute model training, such as Megatron, and inference engines, systems that execute generation, such as SGLang and vLLM, can produce different numerical likelihoods. The authors argue that GSPO’s sequence-level ratios tolerate these discrepancies well enough to use inference-engine likelihoods directly, avoiding recomputation with the training engine. They identify partial rollout, generation performed in portions, multi-turn RL, and frameworks with separate training and inference components as potential beneficiaries, without reporting latency, memory, or throughput measurements.

6 Conclusion

The conclusion presents sequence-level likelihood ratios, clipping, rewards, and optimization as GSPO’s central design. The authors state that GSPO was used in training the latest Qwen3 models, but this paper does not isolate its quantitative contribution to those models’ overall improvements.

Appendix

The supplied seven-page paper contains no appendix; it ends with the conclusion and references.

Brief Thoughts

The strongest practical evidence is the combination of better training curves against GRPO with Routing Replay and stable GSPO training without replay. Length normalization changes the raw sequence likelihood ratio, so the importance-sampling motivation does not establish that the resulting update is an unbiased distribution correction. Broader applicability remains uncertain because the experiments name one starting model and omit numerical compute budgets, detailed training data, repeated-run uncertainty, and separate evaluations of infrastructure savings or customized GSPO-token advantages.