Train Long, Think Short: Curriculum Learning for Efficient Reasoning | Summary

Contents

Summary
1 Introduction
2 Related Work
3 Methodology
4 Experiments
5 Limitations
6 Conclusion
Appendix
Brief Thoughts

This article explains the key points of Train Long, Think Short: Curriculum Learning for Efficient Reasoning.

2025-08-12 (arXiv)
Hammoud, Hasan Abed Al Kader, Alhamoud, Kumail, Hammoud, Abed, Bou-Zeid, Elie, Ghassemi, Marzyeh, Ghanem, Bernard.
King Abdullah University of Science and Technology (KAUST), Saudi Arabia, Massachusetts Institute of Technology (MIT), Cambridge, MA, USA, Princeton University, Princeton, NJ, USA
Paper
Github

Summary

The paper studies how to reduce the number of tokens, the units of generated text, that large language models use to solve mathematical problems while preserving answer accuracy. For QWEN-2.5-7B trained on GSM8K, a grade-school arithmetic dataset, gradually reducing the target from 256 to 87 tokens achieves 86.20% accuracy versus 82.71% for fixed-target training at 87 tokens, with nearly identical average token usage.
The method combines curriculum learning, progressively tightening training constraints, with Group Relative Policy Optimization (GRPO), which normalizes rewards across candidate responses to the same question. Rewards combine automatically verified answer correctness, proximity to the current token target, and structural tags separating reasoning from the final answer.
Ablations, comparisons that vary individual design choices, show that reward shape and budget trajectory affect the accuracy–length trade-off: a triangular reward that peaks at the target length achieves 57.9% reported average accuracy at 135 tokens, compared with 55.0% at 94 tokens for a flat-band reward that gives maximum reward to all lengths below the target. Evidence is restricted to one model and mathematical benchmarks, and reported outputs often exceed the nominal 87-token target.

1 Introduction

The introduction distinguishes supervised fine-tuning, learning from annotated examples, from reinforcement learning, optimizing generated responses through reward feedback. Chain-of-thought means the intermediate reasoning steps generated before an answer; the proposed curriculum initially permits longer traces and then encourages their compression.

The contribution is a changing training target. The authors propose that allowing longer responses early helps the model explore solution strategies before learning to express them concisely.
The implementation uses torchtune, the training software named in the paper. Code and checkpoints are provided at https://github.com/hammoudhasan/curriculum_grpo.

Related work covers test-time scaling, increasing computation during inference, and methods for controlling reasoning length. Examples include self-consistency, selecting the most consistent answer among sampled reasoning paths; TokenSkip, suppressing low-utility tokens; and Chain of Draft, prompting concise intermediate drafts.

Thinkless selects between short and long reasoning paths according to problem difficulty, while ConciseRL incorporates a learned conciseness score into reinforcement-learning rewards.
Length Controlled Policy Optimization (LCPO) trains models to follow a user-specified length budget in the prompt. The present paper studies how decreasing the training target shapes a model intended to produce short responses.
The authors describe their resulting model as staying within roughly 5% of its target on average without runtime budget hints. Later measurements exceed that range on several benchmarks, so this statement does not establish general adherence to a hard limit.

3 Methodology

The methodology adds three reward components to GRPO: correctness verified from the final answer, length adherence to the current curriculum target, and valid reasoning/answer formatting. Figure 1 illustrates the proposed progression from a verbose solution to a shorter solution with the same answer.

Curriculum Learning GRPO Overview: length rewards, decreasing token targets, and an illustrative compressed solution.

3.1 GRPO Preliminaries

For each prompt, the old policy, the model distribution used to sample responses before an update, generates a group of G responses, each assigned a scalar reward. GRPO computes a group-relative advantage \(A_i=\frac{r_i-\mu}{\sigma}\), a normalized reward indicating how a response compares with its group, where μ and σ are the group reward mean and stabilized standard deviation.

The objective uses the probability ratio between the updated and old policies. Clipping that ratio within a window controlled by ε limits its contribution to the update.
A Kullback–Leibler divergence penalty, measuring deviation from a reference policy, is weighted by β. It regularizes the clipped reward-based objective.

3.2 Prompt Structure

The training prompt requests reasoning inside <think></think> and the final answer inside <answer></answer>, and explicitly asks for exactly {token_budget} tokens. The associated reward guides generated length toward the target; reported measurements show that the request does not ensure exact output length.

3.3 Reward Decomposition and Weighting

The total reward is r_i = λ_c r_i^correct + λ_ℓ R_len(ℓ_i) + λ_f R_fmt(a_i), with nonnegative weights for correctness, length, and formatting. The automated verifier math-verify checks the answer extracted from the answer tags, and the correctness reward is R_cor c_i for a binary acceptance indicator c_i.

The triangular length reward increases linearly from zero at length 0 to R_max at the target L, decreases linearly to zero at 2L, and remains zero above 2L. It rewards matching the target rather than minimizing length.
The formatting reward adds separate positive contributions for well-formed, non-overlapping reasoning and answer spans.
Partial correctness scores are described as an optional extension. The PDF does not establish that they were used in the experiments.

3.4 Curriculum Token Budget

The curriculum sets B(t) = max(1, B_0 γ^{⌊t/T⌋}), where B_0 is the initial token budget, γ ∈ (0,1) is the decay factor, and T is the interval between budget updates. Setting the length-reward target L to B(t) progressively tightens the incentive during training.

4 Experiments

The experiments ask whether curriculum training improves the accuracy–length trade-off at a matched final target and whether gains hold across easier and harder training data. Four further questions test correctness-versus-length weighting, the number of decay points, triangular versus flat-band rewards, and exponential versus linear schedules.

4.1 Setup

All experiments use QWEN-2.5-7B with GRPO group size G = 8. The comparisons are the unchanged base model, fixed-budget GRPO targeting 87 tokens, and curriculum GRPO decreasing its target from 256 to 87 tokens.

Separate training conditions use all 7,473 GSM8K grade-school problems or 500 MATH500 competition-level problems drawn from the MATH dataset. The main checkpoint plots show training through 600 steps.
Evaluation uses zero-shot prompting, without task demonstrations, on GSM8K, SVAMP, GSM+, MATH500, and College Math. The paper describes SVAMP as perturbed word problems, GSM+ as adversarial GSM8K variants, and College Math as university-level mathematics.
The setup names MATH500 as both training and evaluation data without specifying a disjoint split. Its reported result therefore does not establish performance on unseen MATH500 problems.

4.2 Curriculum Learning Vs. Fixed Budget

For GSM8K-trained models, curriculum training raises GSM8K accuracy from 82.71% to 86.20%. Section 4.2 reports average token usage of 88.8 for curriculum training versus 87.0 for fixed-budget training; the base model achieves 83.55% at 258.4 tokens.

On out-of-distribution evaluations, benchmarks differing from the training distribution, the text reports accuracy increases from 77.67% to 85.00% on SVAMP and from 62.75% to 67.58% on GSM+.
For MATH500-trained models, the text reports accuracy increasing from 38.80% to 43.40%, while average reasoning length decreases from 179.3 to 137.1 tokens. The fixed-budget MATH500 marker in Figure 2 appears inconsistent with the stated 38.80%.
Figure 2 does not support superiority on every benchmark: the final curriculum point is below the fixed-budget point for College Math after GSM8K training and for GSM+ after MATH500 training.

Curriculum versus fixed-budget training on GSM8K and MATH500, evaluated across five mathematical benchmarks.

4.3 Reward Weight Ablations: Correctness Vs. Length

The ablation compares length-heavy weights λ_c = 0.3, λ_ℓ = 0.6 with correctness-heavy weights λ_c = 0.6, λ_ℓ = 0.3. Section 4.3 reports final GSM8K accuracy of 85.37% at 92.3 tokens for length-heavy training and 87.34% at 93.5 tokens for correctness-heavy training.

The prose values imply a gain of 1.97 percentage points for 1.2 additional tokens on average when correctness receives greater weight.
The text reports correctness-heavy gains of 1–2 points on SVAMP and GSM+. The final GSM8K point in Figure 3 appears near 87.3%, conflicting with the stated length-heavy result of 85.37%; the numerical comparison requires reconciliation.

Length-heavy reward weighting and its checkpoint accuracy–token trade-offs.

Correctness-heavy reward weighting across five mathematical benchmarks.

4.4 Effect of Curriculum Schedule

The schedule ablation fixes initial and final targets at S_0 = 256 and S_f = 87 while varying the number of decay points n. It defines \(I=\frac{T}{n+1},\qquad S_k=S_0d^k,\qquad d=\left(\frac{S_f}{S_0}\right)^{1/n}\); here T denotes total training steps rather than the update interval used in Section 3.4.

With T = 600, n = 1 gives d ≈ 0.340 and interval 300; n = 3 gives d ≈ 0.700 and interval 150; n = 7 gives d ≈ 0.857 and interval 75. Each schedule reaches the same final target, although the step at which it reaches that target differs.
Table 1 reports 57.9% average accuracy for intervals 150 and 75, with 135 and 115 average tokens, respectively. Interval 300 gives 52.4% at 248 tokens.
The single late drop achieves 86.8% on GSM8K and 88.0% on SVAMP but only 9.8% on MATH500. The comparison demonstrates schedule sensitivity without directly measuring the proposed exploration-and-compression mechanism.

Decay rate ablation for exponential schedules with fixed initial and final token targets.

4.5 Effect of Length Reward Function

The flat-band alternative gives maximum length reward to every output up to L, then decreases linearly to zero at 2L. It removes the triangular reward’s increasing incentive to use additional tokens below the target.

Table 2 reports an average of 94 tokens and 55.0% accuracy for the band reward, compared with 135 tokens and 57.9% for the triangular reward.
Triangular rewards achieve higher accuracy on every listed benchmark. The largest gap is on MATH500: 37.4% at 201 tokens versus 30.8% at 112 tokens.
These results show higher accuracy with longer outputs under the triangular reward. They are compatible with the authors’ exploration-before-compression interpretation, but do not directly measure the discovery or retention of reasoning strategies.

Triangular versus Band length reward, also described as flat-then-decay.

Length reward shape comparison using triangular and band rewards.

4.6 Effect of Decay Schedule Shape

The linear schedule decreases the target in roughly three equal drops from 256 to 87 over the same training duration. The exponential schedule makes larger reductions earlier, while both schedules reach the same final target.

Table 3 reports average accuracy increasing from 57.9% with exponential decay to 60.0% with linear decay, while average token count rises from 135 to 140.
Linear decay improves MATH500 from 37.4% to 42.8% and College Math from 13.4% to 17.2%. Outputs on those datasets are slightly shorter: 198 versus 201 tokens and 154 versus 187 tokens, respectively.
The benefit varies by benchmark: GSM+ accuracy decreases from 67.6% to 66.4%, with token count increasing from 110 to 143.

Exponential versus Linear decay schedules over 600 training steps.

Decay scheduler type comparison for exponential and linear budget reduction.

5 Limitations

The authors identify short context windows, the amount of text available to the model, and training targets capped at 256 tokens as limitations for problems requiring extended reasoning. All experiments use QWEN-2.5-7B, leaving behavior at smaller and larger model sizes, and on open-ended generation tasks, untested.

The PDF does not report uncertainty estimates or results from repeated runs with different random seeds, so the stability of small differences cannot be assessed.
Average lengths of 137.1 tokens in the MATH500-trained comparison and 201 tokens in the MATH500 ablation exceed the 87-token target. The target functions as a training incentive; the reported results do not demonstrate a hard inference limit.

6 Conclusion

The conclusion presents decreasing token targets as a way to train shorter mathematical reasoning while retaining or improving accuracy. The experiments support sensitivity to budget trajectory and reward design within the tested model and datasets; broader generalization and the proposed exploration-to-compression mechanism remain unverified.

Appendix

The supplied 14-page PDF contains no appendix and ends with References. It provides no additional appendix experiments or implementation details.

Brief Thoughts

Table 1 provides the clearest engineering lesson: identical initial and final training targets can produce different accuracy and output length. A reproduction should establish dataset separation, reconcile numerical prose with plotted points, and measure output-length distributions before relying on the method for budget control.