Soft Adaptive Policy Optimization | Summary

Contents

Summary
1 Introduction
2 Preliminaries
3 Soft Adaptive Policy Optimization
4 A Gating-Function Perspective on SAPO’s Connections to GRPO and GSPO
5 Experiments
- 5.1 Controlled Experiments
- 5.2 Qwen3-VL Training
6 Conclusion
Appendix
Brief Thoughts

This article explains the key points of Soft Adaptive Policy Optimization.

2025-11-25 (arXiv)
Gao, Chang, Zheng, Chujie, Chen, Xiong-Hui, Dang, Kai, Liu, Shixuan, Yu, Bowen, Yang, An, Bai, Shuai, Zhou, Jingren, Lin, Junyang.
Qwen Team, Alibaba Inc.
Paper
Github
Project Page

Summary

Reward-based training of language models can become unstable when token probabilities change unevenly between response generation and parameter updates. Soft Adaptive Policy Optimization (SAPO), a method that smoothly attenuates updates as these probabilities change, sustains useful training longer and achieves higher single-response success rates than Group Sequence Policy Optimization (GSPO), which clips updates by response, and Group Relative Policy Optimization with routing replay (GRPO-R2), which clips updates by token and reuses expert-routing decisions, in controlled mathematical reasoning experiments under comparable training budgets.
SAPO replaces hard clipping, which zeroes gradients, the derivatives used to update model parameters, beyond an advantage-dependent threshold, with a sigmoid-based gate, a smooth bounded update weight controlled by the current-to-generation token probability ratio. The advantage is a response’s reward relative to its group, normalized by the group’s reward standard deviation. SAPO preserves the unclipped gradient when the ratio equals 1 and uses a larger temperature, a parameter controlling how rapidly weights decay, for negative advantages.
Controlled experiments use a cold-start checkpoint, the initialization before the compared reward-based training runs, fine-tuned from Qwen3-30B-A3B-Base. A separate Qwen3-VL-30B-A3B comparison on text and multimodal tasks, tasks involving multiple input modalities, shows higher aggregate validation performance under equal compute budgets without the baseline collapse seen in the controlled comparison. The paper presents training curves but no tabulated endpoint scores, repeated-run uncertainty estimates, or complete compute specifications.

1 Introduction

Reinforcement learning (RL), training a policy using rewards, improves reasoning in large language models (LLMs), models that generate token sequences. Group-based policy optimization samples several responses per query, normalizes their rewards within the group, and weights updates using importance ratios: probabilities under the current policy divided by probabilities under the behavior policy that generated the responses.

Mixture-of-Experts (MoE) models, architectures that route computation through selected expert components, can exhibit greater token-ratio variation because routing differs across tokens; long responses can further amplify this heterogeneity.
Tight clipping discards useful learning signals, while loose clipping admits noisy off-policy updates, updates using responses generated by a different policy. SAPO aims to retain informative tokens while smoothly suppressing large deviations.
The introduction acknowledges that all compared methods may eventually become unstable. Its empirical claim concerns a longer useful training period and higher accuracy before divergence.

2 Preliminaries

An autoregressive policy, a model that predicts each token conditioned on preceding tokens, factorizes response likelihood into token probabilities. Group Relative Policy Optimization (GRPO) assigns every token in response i the same group-normalized advantage Âᵢ = (Rᵢ − mean(R))/std(R), then averages clipped token objectives over responses and their tokens.

The token importance ratio is rᵢ,ₜ(θ) = πθ(yᵢ,ₜ | q, yᵢ,<ₜ)/πθold(yᵢ,ₜ | q, yᵢ,<ₜ). Here q is the query, θ denotes current parameters, θold denotes behavior-policy parameters, and Rᵢ is response i’s reward.
GSPO instead clips the length-normalized response ratio sᵢ(θ) = exp(meanₜ log rᵢ,ₜ(θ)), the geometric mean of token ratios. Length normalization reduces variance and gives responses of different lengths a consistent numerical scale.
Both methods use the minimum of the original and clipped advantage-weighted objectives. Positive advantages lose their gradient above the upper ratio threshold; negative advantages lose it below the lower threshold.

3 Soft Adaptive Policy Optimization

SAPO maximizes the expected response-and-token average of fᵢ,ₜ(rᵢ,ₜ(θ))Âᵢ,ₜ, where fᵢ,ₜ(x) = (4/τᵢ,ₜ)σ(τᵢ,ₜ(x − 1)) and σ(x) = 1/(1 + e⁻ˣ). The temperature τᵢ,ₜ is τpos for positive advantages and τneg otherwise.

Differentiation gives the importance-weighted log-policy gradient multiplied by wᵢ,ₜ = 4pᵢ,ₜ(1 − pᵢ,ₜ), where pᵢ,ₜ = σ(τᵢ,ₜ(rᵢ,ₜ − 1)). This weight peaks at 1 when the probability ratio equals 1 and decreases smoothly as the ratio moves away from 1.
The factor 4/τᵢ,ₜ makes the gradient at ratio 1 equal to the unclipped gradient for every temperature. Larger temperatures narrow the region of substantial gradient weight.
Figure 1 illustrates this soft trust region, the range of policy changes receiving substantial update weight. Moderately changed tokens retain gradients, while extreme deviations receive progressively smaller weights.

Surrogate objective values and gradient weights under positive advantage for unclipped updates, hard clipping, and temperature-controlled soft gates.

The discussion Why Different Temperatures for Positive and Negative Advantages motivates τneg > τpos through gradients on logits, the scores converted into token probabilities. Positive advantages increase the sampled token’s score and decrease unsampled scores; negative advantages reverse these directions and therefore increase the scores of many unsampled tokens.

The paper argues that a large vocabulary contains many irrelevant alternatives at a given generation step, so negative updates can increase the scores of undesirable tokens, especially when responses are off-policy.
Stronger attenuation for negative advantages aims to reduce this instability while retaining their contributions to exploration and regularization. The controlled temperature ablation, a comparison that varies this parameter, tests that rationale.

4 A Gating-Function Perspective on SAPO’s Connections to GRPO and GSPO

The paper places SAPO, GRPO, and GSPO in a unified surrogate objective, the function optimized using sampled responses, whose algorithm-specific function fᵢ,ₜ determines how importance ratios affect updates. SAPO uses a sigmoid surrogate, GRPO uses an advantage-dependent clipped token ratio, and GSPO uses an advantage-dependent clipped sequence ratio.

For positive advantages, GRPO uses min(rᵢ,ₜ, 1 + ε); for nonpositive advantages, it uses max(rᵢ,ₜ, 1 − ε), where ε > 0 is the clipping range.
GSPO uses the same sign-dependent clipping structure with a sequence ratio shared across tokens. Its formulation uses sg[·], the stop-gradient operation that holds a quantity fixed during differentiation.
For SAPO and GRPO, each log-policy gradient is weighted by f′ᵢ,ₜ(rᵢ,ₜ)rᵢ,ₜÂᵢ. This expression separates the smooth or binary gradient gate from the remaining policy-gradient terms.

4.1 SAPO–GSPO Connection: Reduction to a Sequence-Level Soft Gate

SAPO’s exact token gate is \(f’_{i,t}(r)=\operatorname{sech}^{2}\!\left(\frac{\tau_i}{2}(r-1)\right)\), where sech is the hyperbolic secant, the reciprocal of the hyperbolic cosine. The proposed connection to sequence-level optimization assumes (A1) small policy changes, rᵢ,ₜ ≈ 1, and (A2) low within-response variance of log token ratios.

Under (A1), r − 1 is approximated by log r, giving gτ(z) = sech²(τz/2). Under (A2), averaging this smooth gate over tokens approximates evaluating it at the response’s mean log ratio, log sᵢ.
A second-order Taylor expansion bounds the scalar gate discrepancy by Dᵢ(θ) ≤ (τᵢ²/4)Varᵢ(θ), where Varᵢ is within-response log-ratio variance.
The paper then presents an approximate GSPO-like sequence gradient with a continuous gate. The stated bound controls the average scalar gate discrepancy; it does not separately bound the error of the full vector gradient approximation.

The authors examine (A1) and (A2) using more than 10⁵ sequences and 10⁹ tokens from off-policy mini-batches of cold-start Qwen3-30B-A3B and Qwen3-4B checkpoints. Figures 2 and 3 show ratios sharply concentrated around 1, log-ratio variance typically below 0.02, and scalar gate discrepancies consistent with the theoretical bound.

Qwen3-30B-A3B, the MoE checkpoint, has a wider variance distribution than Qwen3-4B, the dense checkpoint whose computation does not use sparse expert routing. The paper suggests expert-routing heterogeneity as a possible explanation.
These observations support the assumptions for most sampled responses, particularly in the dense model. They do not establish that the assumptions hold throughout every training run.
When heterogeneous or outlier tokens violate the assumptions, SAPO continues to compute token-specific weights. It can suppress extreme tokens while preserving other tokens in a response that GSPO might clip as a whole, depending on the sequence ratio and advantage sign.

Token importance ratios, within-response log-ratio variance, and gate approximation discrepancy for Qwen3-30B-A3B.

Token importance ratios, within-response log-ratio variance, and gate approximation discrepancy for the dense Qwen3-4B model.

4.2 SAPO–GRPO Connection: Smooth Token Gates vs. Hard Token Clipping

GRPO’s gradient gate is 1 for positive-advantage tokens with rᵢ,ₜ ≤ 1 + ε and 0 above that threshold; for nonpositive advantages, it is 1 when rᵢ,ₜ ≥ 1 − ε and 0 below that threshold. SAPO replaces this sign-dependent binary gate with a smooth weight centered at ratio 1.

Near the behavior policy, SAPO preserves responsive gradients. As deviations grow, it reduces update weights continuously instead of introducing a clipping discontinuity.
SAPO avoids abrupt zeroing at a finite clipping threshold, although its weights can still become very small for extreme deviations. For zero advantage, the advantage-weighted gradient is zero regardless of the gate.

4.3 Summary

The three methods primarily differ in their treatment of tokens whose current-to-behavior probability ratios deviate from 1. SAPO applies smooth token-specific attenuation and can preserve informative parts of a response, whereas GRPO and GSPO impose sign-dependent hard clipping at token and sequence levels, respectively.

5 Experiments

The experiments evaluate optimization behavior in controlled mathematical reasoning training and Qwen3-VL training on mixed text and multimodal tasks. Training reward and validation trajectories provide the main evidence, with a separate temperature ablation testing the asymmetric gate design.

5.1 Controlled Experiments

Controlled experiments start from a cold-start model fine-tuned from Qwen3-30B-A3B-Base on mathematical reasoning queries. Validation reports average Pass@1, the success rate of an individual generated response, over 16 samples on the mathematical reasoning benchmarks AIME25, HMMT25, and BeyondAIME; each rollout batch, a batch of generated responses, is divided into four mini-batches for gradient updates.

SAPO uses τpos = 1.0 and τneg = 1.05. GSPO and GRPO-R2 use the hyperparameter configurations from Zheng et al. (2025); GRPO-R2 adds routing replay to GRPO, while SAPO does not require it.
Figure 4 shows SAPO maintaining higher reward and validation performance after the baselines deteriorate. GSPO and GRPO-R2 decline sharply before the end of SAPO’s plotted trajectory.
The results are presented as curves rather than exact endpoint tables. The paper does not report multiple random seeds, confidence intervals, or a numerical compute breakdown for this comparison.

Training reward and mathematical reasoning validation trajectories for SAPO, GSPO, and GRPO-R2 from a Qwen3-30B-A3B-Base-derived cold-start model.

The temperature ablation fixes τpos = 1.0 and compares τneg = 1.05, τneg = 1.0, and τneg = 0.95. Figure 5 shows the longest stable trajectory with τneg = 1.05, later collapse with equal temperatures, and earlier deterioration with τneg = 0.95.

This comparison supports stronger attenuation of negative-advantage updates in the tested setting.
Only three nearby negative temperatures are examined, so the evidence does not determine an optimal temperature across architectures, tasks, or training regimes.

Training reward and mathematical reasoning validation trajectories for three SAPO negative-temperature settings.

5.2 Qwen3-VL Training

Qwen3-VL, the model family trained on text and multimodal tasks in this study, combines mathematics, coding, and logical reasoning training with a fixed sampling ratio for each task within a batch. Large rollout batches are divided into two mini-batches for gradient updates.

The explicit three-method comparison starts from the same preliminary cold-start Qwen3-VL-30B-A3B checkpoint. It evaluates AIME25 using Pass@1 with 32 samples, LiveCodeBench v6, a code-generation benchmark, using Pass@1 with 8 samples, ZebraLogic, a logical-reasoning benchmark, and MathVision, a multimodal mathematical-reasoning benchmark.
Figure 6 reports training reward and the mean validation score across the four benchmarks. SAPO finishes above GSPO and GRPO-R2 under the reported equal compute budgets.
The paper states that SAPO improves models across sizes and both dense and MoE architectures, but the displayed comparative curves cover Qwen3-VL-30B-A3B. Per-benchmark scores, exact task sampling ratios, and numerical batch sizes are not supplied.

Training reward and aggregate validation trajectories for SAPO, GSPO, and GRPO-R2 from a preliminary Qwen3-VL-30B-A3B cold-start checkpoint.

6 Conclusion

The conclusion attributes SAPO’s gains to smooth token-adaptive gating and asymmetric temperatures that more strongly attenuate negative-advantage gradients. The reported mathematical reasoning comparison supports longer stable training, while both that comparison and the Qwen3-VL comparison support higher validation performance within the evaluated budgets.

The presented results do not establish immunity to eventual divergence or quantify gains for every model size and architecture mentioned in the conclusion.

Appendix

The supplied nine-page PDF contains no appendix. Its theoretical derivations, empirical assumption checks, and temperature ablation appear in the main body.

Brief Thoughts

The controlled comparisons and temperature ablation support smooth gating with stronger attenuation of negative-advantage updates in the tested training setup. The Qwen3-VL comparison extends the evidence to a mixed-task setting, where SAPO achieves higher aggregate validation performance without baseline collapse. The sequence-coherence argument is approximate: the scalar gate bound is explicit, while the subsequent full-gradient reduction lacks a separate error bound. Reproducibility and generality are limited by the absence of exact final scores, repeated-run variability, complete training configurations, and comparative results for each claimed model scale. The Qwen3-VL aggregate curve does not reveal whether every individual benchmark improves.