s1: Simple test-time scaling | Summary

This article explains the key points of s1: Simple test-time scaling.

• 2025-01-31 (arXiv)
• Muennighoff, Niklas, Yang, Zitong, Shi, Weijia, Li, Xiang Lisa, Fei-Fei, Li, Hajishirzi, Hannaneh, Zettlemoyer, Luke, Liang, Percy, Candès, Emmanuel, Hashimoto, Tatsunori.
• Stanford University, University of Washington, Seattle, Allen Institute for AI, Contextual AI
• Paper

Read this article in Korean

Summary

• The paper asks how a language model can improve reasoning by spending more computation while answering a question, a practice called test-time scaling. Its s1-32B model raises accuracy on AIME24, the 30-problem 2024 American Invitational Mathematics Examination benchmark, from 50.0% without budget forcing to 56.7% with budget forcing, an intervention that controls when the model stops thinking.
• The recipe combines s1K, a curated dataset of 1,000 questions with generated reasoning traces—intermediate reasoning before an answer—and answers, with supervised fine-tuning, training the model to predict those supplied outputs. Fine-tuning Qwen2.5-32B-Instruct takes 26 minutes on 16 NVIDIA H100 GPUs; budget forcing then shortens reasoning by inserting an answer delimiter, a marker that starts the answer, or extends it by suppressing that delimiter and appending “Wait”.
• The curated 1,000 examples remain competitive with training on all 59K examples, while budget forcing achieves the highest AIME24 accuracy among the tested scaling methods. Further extension eventually produces repetitive loops or exceeds the context window, the maximum sequence the model can process. The comparisons do not establish that sequential scaling, where later computation uses earlier results, is universally superior to parallel sampling, where independent attempts are combined.

1. Introduction

The introduction seeks a simple, openly reproducible way to obtain both strong reasoning performance and accuracy gains from additional inference computation. The system uses next-token prediction, learning to predict the next element of a sequence, on examples distilled from Gemini Thinking Experimental, meaning that another model supplies the reasoning traces and answers, followed by an intervention during generation.

• The contribution has two parts: selecting a small reasoning dataset and controlling computation after training.
• The model weights, reasoning data, and code are released at https://github.com/simplescaling/s1.

Accuracy rises with average thinking-token use on all three benchmarks over the displayed settings. The finite range demonstrates tested operating points without establishing continued improvement beyond them.

2. Reasoning data curation to create s1K

The s1K construction process first collects a large pool and then selects 1,000 examples using quality, difficulty, and diversity. Quality concerns usable source material and formatting, difficulty concerns problems that require substantial reasoning, and diversity concerns coverage across subject domains.

2.1. Initial collection of 59K samples

The initial pool contains 59,029 questions from 16 sources, including 30,660 NuminaMATH problems, 4,250 OlympicArena questions, 4,238 OmniMath problems, and 2,385 AGIEval questions. Two new sources add 182 probability questions from Stanford statistics qualifying examinations as s1-prob and 23 “Hard” quantitative interview problems from PuzzledQuant as s1-teasers.

• For each question, Gemini Flash Thinking generates a reasoning trace and a response; the resulting training example contains the question and both generated components.
• The authors deduplicate the data and check overlap against MATH500, a 500-question competition mathematics benchmark, GPQA Diamond, a 198-question doctoral-level science benchmark, and AIME24 using 8-grams. These are contiguous sequences of eight elements; the PDF does not specify whether the elements are words or tokens.

2.2. Final selection of 1K samples

Removing questions affected by generation-service errors leaves 54,116 examples, and filtering formatting problems leaves 51,581. Difficulty filtering removes questions solved by either Qwen2.5-7B-Instruct or Qwen2.5-32B-Instruct, with Claude 3.5 Sonnet comparing their attempts against reference solutions, leaving 24,496 questions.

• The selection retains 384 examples from sources judged especially suitable, then repeatedly samples a domain uniformly and favors longer reasoning traces within that domain until reaching 1,000 examples.
• Domains follow the Mathematics Subject Classification, a subject taxonomy that the authors also use for science topics; the main text reports 50 domains, whereas Appendix C.1 reports 51.
• Correctness is not required for every generated solution: the final grader marks 53.6% of s1K and 63.0% of s1K-1.1 correct. The authors retain incorrect generations because their stated objective includes capturing the reasoning process.

The domain display shows the breadth of the selected examples, while the comparison places s1 near models trained on much larger reasoning datasets at similar MATH500 accuracy. The example counts exclude pretraining and teacher-data production costs.

3. Test-time scaling

The paper distinguishes sequential scaling from parallel scaling and focuses on extending or truncating a single reasoning trace. It evaluates accuracy alongside control over the computation spent.

3.1. Method

Budget forcing sets a maximum thinking budget by inserting the end-of-thinking delimiter, optionally followed by “Final Answer:”, so the model supplies its current answer. To extend reasoning, it suppresses the delimiter when the model attempts to stop and optionally appends “Wait”, encouraging the model to reconsider its reasoning.

• Conditional length-control baselines request limits in tokens, the units generated by the model; steps, reasoning segments separated by double newlines; or broad short/long thinking classes. The control comparison assigns an upper bound of 100 tokens per requested step.
• Rejection sampling repeatedly generates an output until its length fits a specified budget; the accepted outputs therefore reflect the model’s output distribution conditioned on length.
• The raspberry example shows a correction from two occurrences of “r” to three after “Wait” is appended. It demonstrates a possible outcome without estimating how often extension helps.

The model corrects its count of “r” in “raspberry” after the stopping delimiter is suppressed and “Wait” is appended. This successful example demonstrates the intervention without quantifying its reliability.

3.2. Metrics

Let A denote the evaluated compute settings and f(a) the accuracy at thinking-token count a. Control measures adherence to specified compute bounds, Scaling averages pairwise accuracy-versus-compute slopes, and Performance records the highest observed accuracy.

• Control = (1/|A|) Σ[a∈A] 𝟙(a_min ≤ a ≤ a_max). The indicator 𝟙 equals one when the bounds are satisfied and zero otherwise; the experiments usually constrain only the maximum.
• Scaling = (1/binom(|A|,2)) Σ[a,b∈A; b>a] (f(b)−f(a))/(b−a). Here binom(|A|,2) counts distinct pairs of settings; positive values indicate higher accuracy with greater thinking-token use across the tested settings.
• \(\mathrm{Performance}=\max_{a\in\mathcal A}f(a)\). This measures the best tested operating point rather than guaranteed improvement beyond the evaluated range.

4. Results

The results assess reasoning accuracy, changes with inference computation, and the number of reasoning fine-tuning examples used. The main comparison reports s1-32B at 56.7% on AIME24, 93.0% on MATH500, and 59.6% on GPQA Diamond.

4.1. Setup

Training starts from Qwen2.5-32B-Instruct and uses s1K with PyTorch FSDP, Fully Sharded Data Parallel training that distributes model training state across GPUs. Evaluation uses lm-evaluation-harness, a language-model evaluation framework, and, unless otherwise specified, temperature 0 greedy decoding, which selects the highest-scoring next token; accuracy corresponds to pass@1, success from a single generated attempt.

• The benchmarks contain 30 AIME24 problems, 500 MATH500 problems, and 198 GPQA Diamond questions in biology, chemistry, and physics. AIME diagrams are supplied as Asymptote vector-graphics source because the model cannot accept image inputs.
• The authors evaluate s1-32B, Qwen, and Gemini themselves, while many comparison scores come from the respective model reports.
• Gemini’s generation API returns recitation errors that obstruct evaluation. Its 60.0% AIME24 score comes from manually submitting all 30 questions through the web interface; MATH500 and GPQA Diamond remain unavailable.

4.2. Performance

Relative to the base model’s 26.7%, 84.0%, and 49.0% accuracy on AIME24, MATH500, and GPQA Diamond, respectively, s1-32B reaches 56.7%, 93.0%, and 59.6%. Without budget forcing, the fine-tuned model scores 50.0%, 92.6%, and 56.6%, separating the gains from training from the additional gains obtained through inference intervention.

• Against o1-preview’s reported 44.6% on AIME24 and 85.5% on MATH500, s1-32B improves by 12.1 and 7.5 percentage points, respectively. The abstract’s “up to 27%” refers to the relative AIME24 improvement; o1-preview remains ahead on GPQA Diamond at 73.3%.
• On GPQA Diamond, majority voting, choosing the most frequent answer among independent attempts, over 2, 4, 8, 16, 32, or 64 base-model samples at temperature 1 does not reach the fine-tuned model’s sequential-scaling accuracy. Because training and inference method both differ, this comparison does not isolate sequential versus parallel computation.
• Suppressing stopping 2, 4, or 6 times extends AIME24 reasoning, with accuracy eventually plateauing at six suppressions. Excessive suppression can induce repetitive loops.
• The sample-efficiency comparison counts additional reasoning examples: r1-distill uses 800K, while s1 uses 1K. It excludes pretraining and the cost of generating and filtering the initial data pool.

The AIME24 panel shows gains followed by a plateau as stopping is suppressed repeatedly. On GPQA Diamond, sequential extension of the fine-tuned model exceeds base-model majority voting in the tested range, but training and inference method both differ.

Separate rows for the base model, s1 without budget forcing, and s1-32B show improvements from both fine-tuning and inference intervention. The advantage over o1-preview is confined to mathematics; o1-preview scores substantially higher on GPQA Diamond.

5. Ablations

The ablations, experiments that vary components of the recipe, test the combined selection criteria and alternatives to budget forcing. They examine both training-example selection and control of inference computation.

5.1. Data Quantity, Diversity, and Difficulty

With a maximum of around 30,000 thinking tokens applied to every data-ablation model, s1K scores 50.0% on AIME24, compared with 36.7% for 1K-random, 26.7% for 1K-diverse, 33.3% for 1K-longest, and 53.3% for 59K-full. Training on the full pool requires 394 H100 GPU hours, versus approximately 7 H100 GPU hours for s1-32B.

• The 95% paired bootstrap confidence intervals estimate uncertainty by resampling the same evaluation questions for both models, using 10,000 resamples. On AIME24, differences relative to s1K are [−26.7%, −3.3%] for 1K-random, [−40.0%, −10.0%] for 1K-diverse, [−36.7%, 0.0%] for 1K-longest, and [−13.3%, 20.0%] for 59K-full.
• The full-pool intervals include both gains and losses; they support competitiveness of the selected subset without establishing statistical equivalence.
• Selection benefits vary by benchmark: 1K-longest reaches 59.6% on GPQA Diamond, above s1K’s 57.6% in this ablation. The combined recipe does not dominate every selection strategy on every task.

With the shared maximum of around 30,000 thinking tokens, random and diversity-only selection produce lower AIME24 accuracy than s1K, with reported difference intervals entirely below zero. The full pool has no statistically established advantage, while the longest-trace subset scores higher on GPQA Diamond.

5.2. Test-time scaling methods

Budget forcing achieves 100% Control, a Scaling score of 15, and 56.7% peak AIME24 accuracy across five evaluated settings. Token-conditional control reaches only 40% Control and a Scaling score of −24; adding budget forcing restores 100% Control but leaves its peak accuracy at 40.0%.

• Step-conditional control reaches 60% Control and 36.7% peak accuracy; forced stopping restores 100% Control. The model partially offsets fewer steps by making individual steps longer.
• Class-conditional control has a Scaling score of 25 but only 50% Control and 36.7% peak accuracy, estimated from two settings. Its higher slope accompanies lower control and peak accuracy.
• When stopping is suppressed twice, “Wait” raises AIME24 accuracy from 50.0% to 53.3%, while no string, “Alternatively”, and “Hmm” remain at 50.0%. On GPQA Diamond, all three supplied strings reach 59.6%, while suppression without a string falls to 55.1%.
• Rejection sampling attains 100% Control but a Scaling score of −35. The authors hypothesize that longer accepted traces more often contain mistakes and backtracking; the inspected examples illustrate this explanation without establishing causality.

Budget forcing combines perfect measured adherence with the highest observed AIME24 accuracy. Class-conditional prompting has a larger reported slope but lower control and accuracy, illustrating why the three metrics require joint interpretation.

The examples show the response format and successful answers to a game, a vector projection, and a physics question. The reasoning is shortened for display and the examples are selected for correctness, so they do not describe the model’s overall error distribution.

Suppressing the stopping delimiter without a continuation string reduces MATH500 and GPQA Diamond accuracy relative to no extrapolation. “Wait” is the only tested string that improves AIME24 in this two-extension comparison, while three strings tie on GPQA Diamond.

Accuracy declines as the average length of accepted rejection-sampled outputs increases. This result concerns sampling at temperature 1 with length-based acceptance and does not directly measure the effect of extending an existing reasoning trace.

6. Discussion and related work

The discussion compares the recipe with reasoning models trained using reinforcement learning, learning from reward signals, and models trained on larger collections of distilled outputs. It considers why a small fine-tuning set is effective and what limits further gains from inference computation.

6.1. Sample-efficient reasoning

The authors hypothesize that pretraining has already supplied much of the model’s reasoning ability and that a small supervised dataset elicits that ability. They relate this interpretation to the Superficial Alignment Hypothesis in LIMA, a prior study arguing that relatively few examples can align a pretrained model with desired response behavior.

• The gains after 1,000 examples demonstrate effective adaptation, but the experiments do not identify which reasoning capabilities were already present or establish the proposed activation mechanism.
• Related approaches include continued training on mathematics and science, synthetic reasoning data, and Chain-of-Thought prompting, which asks for intermediate reasoning before an answer.

6.2. Test-time scaling

Sequential scaling is limited by diminishing gains, repetitive reasoning, and the model’s context window. The paper also tests majority voting and REBASE, a tree-search method guided by a process reward model that scores intermediate reasoning steps, using 16 parallel solution trajectories.

• REBASE extends the plotted scaling trend more effectively than majority voting in this experiment, but its token counts exclude the additional computation required by the reward model.
• When the sequential model is instructed to use up to 512 steps, 12 of 30 AIME24 responses exceed the context window, producing a large accuracy drop.
• Appendix E.1 states that Figure 7 uses a step-controlled model trained on an earlier data version and an earlier evaluation codebase. This is a separate experiment from the final main-system comparisons.
• Rotating continuation strings, changing generation settings, and combining budget forcing with reinforcement learning are proposed research directions without demonstrated results in this paper.

Parallel aggregation improves accuracy beyond the displayed sequential method’s practical limit, with REBASE ahead of majority voting. The experiment uses an earlier step-controlled model, and REBASE’s token counts exclude reward-model computation, limiting direct efficiency comparisons.

Impact Statement

The Impact Statement motivates open reasoning research through potential improvements in productivity, complex decision-making, and scientific work. The release of model weights, reasoning data, and code supports its transparency objective; the statement does not empirically assess societal benefits or harms.

Appendix

• A. s1.1 regenerates reasoning traces for the same 1,000 questions using DeepSeek r1 and keeps the training procedure unchanged. Without budget forcing, s1.1 reaches 94.4% on MATH500, 60.6% on GPQA, 56.7% on AIME 2024, and 50.0% on AIME 2025; appending “Wait” twice yields 95.4%, 63.6%, 56.7%, and 50.0%, respectively. These rows distinguish the effect of changing teacher traces from the additional effect of inference intervention.
• B. Evaluation determinism reports that vLLM, the inference engine used for evaluation, can produce different answers under identical seeds and greedy decoding when batch size, continued generation, or tensor parallelism, distributing model computations across GPUs, changes. Long reasoning traces can amplify small numerical differences, so the authors generally use full precision for final evaluations. This variability warrants caution when interpreting small score differences.
• C. s1K details documents dataset composition, grading, diversity sampling, and overlap removal. C.1. s1K summary reports 1,000 questions and 4.7M tokens across 51 domains, with geometry contributing 109 questions, number theory 98, and combinatorics 75; C.2. Dataset composition for full 59K questions lists the 16 sources and reports a 3.6K average thinking length. It also states that 90 AIME questions from 2022–2023 were excluded because they were used during development. C.3. s1K grading prompt compares an attempted solution with a reference, using Claude 3.5 for selection and Claude 3.7 for correctness grading of the final 1,000 samples. C.4. s1K diversity selection presents Algorithm 1: retain designated correct source examples, including MATH examples with ThinkingLength(q) > 5600, then sample domains uniformly and examples with weights 2^(−ranks), where ranks order examples by thinking length to favor longer traces. C.5. Decontamination describes removing overlap with evaluation benchmarks using 8-grams; this check does not establish the absence of contamination in the pretrained base model or teacher.
• D. Training details separates thinking and answering with <|im_start|>think and <|im_start|>answer, trains for 5 epochs, complete passes through the dataset, with batch size 16 and 315 gradient steps, and applies loss only to reasoning traces and solutions. Training uses bfloat16, a reduced-precision numerical format, learning rate 1e−5, a 5% linear warmup lasting 16 steps, cosine decay to zero over 299 steps, and AdamW, an optimizer with decoupled weight decay, using β₁ = 0.9, β₂ = 0.95, and weight decay 1e−4. D.1. Training Ablations: Sequence length compares 4096 tokens, which truncates 74% of training samples, with 32768 tokens, which truncates none. The longer setting improves AIME24 accuracy from 30.0% to 50.0% while reducing average thinking length from 20721 to 6984 tokens; the authors attribute this to retaining more answer sections during training. D.2. Training Samples supplies complete examples from s1-prob, s1-teasers, and OlympicArena astronomy, showing generated reasoning with errors and revisions.
• E. Test-time scaling details expands the alternative control experiments. E.1. Sequential scaling ablations shows that requested token limits from 1024 through 16384 still produce roughly 7K–8K thinking tokens without forced stopping; step control similarly requires intervention and changes tokens per step, while generic short/long prompts do not reliably improve accuracy. E.2. Examples for rejection sampling ablation uses an earlier s1-32B version to contrast a correct 3981-thinking-token geometry solution with an incorrect 5923-thinking-token attempt, illustrating that greater output length alone need not improve reasoning.
• F. Version Control records the V2 → V3 (2025-03): additions of Appendix A and correctness counts in Section 2. V1 → V2 (2025-02): added citations and small writing changes. The s1.1 results belong to a later appendix update and are distinct from the original s1 main-body experiments.

Brief Thoughts

The engineering contribution is a compact recipe with separately evaluated training and inference components: selected reasoning examples improve the base model, and explicit stopping control adds further gains. The data and sequence-length ablations provide evidence for testing curation against a full-pool baseline and retaining complete training examples that include their answers. The evidence for longer reasoning is conditional: gains plateau, rejection sampling shows inverse scaling, and each changed answer represents a substantial accuracy difference on the 30-question AIME24 benchmark. The sequential-versus-parallel comparison changes the model and inference method, while REBASE’s plotted compute excludes reward-model overhead. Broader efficiency claims require matched comparisons with complete compute accounting.