Phi-4-Mini-Reasoning: Exploring the Limits of Small Reasoning Language Models in Math | Summary

Contents

Summary
1 Introduction
2 Background
3 Multi-Stage Continual Training for Reasoning
4 Synthetic CoT Data Generation
5 Experiment
6 Conclusion
Appendix
Brief Thoughts

This article explains the key points of Phi-4-Mini-Reasoning: Exploring the Limits of Small Reasoning Language Models in Math.

2025-04-30 (arXiv)
Xu, Haoran, Peng, Baolin, Awadalla, Hany, Chen, Dongdong, Chen, Yen-Chun, Gao, Mei, Kim, Young Jin, Li, Yunsheng, Ren, Liliang, Shen, Yelong, et al.
Microsoft
Paper

Read this article in Korean

Summary

Small language models have limited capacity to learn the intermediate steps needed for multi-step mathematical reasoning. The paper’s 3.8-billion-parameter Phi-4-Mini-Reasoning achieves 94.6% on MATH-500, a mathematics benchmark, under pass@1 evaluation, which measures single-answer correctness, compared with 91.4% for DeepSeek-R1-Distill-Qwen-7B and 86.9% for DeepSeek-R1-Distill-Llama-8B in the authors’ reproduced comparisons.
The four-stage recipe begins with distillation, training on a stronger model’s generated answers, through large-scale mid-training and selective supervised fine-tuning. These stages use Chain-of-Thought (CoT), explicit intermediate reasoning steps; Direct Preference Optimization (DPO), learning from preferred and rejected response pairs, then precedes reinforcement learning, which updates the model using rewards based on verified final-answer correctness.
The final model scores 57.5% on AIME 2024, a mathematics examination benchmark, and 52.0% on GPQA Diamond, a graduate-level question-answering benchmark. Successive training checkpoints improve across all three evaluations, but the experiments concern one base model and do not isolate the contribution of every reinforcement-learning modification.

Pass@1 accuracy of Phi-4-Mini-Reasoning, Phi-4-Mini, DeepSeek-R1-Distill-Llama-8B, and DeepSeek-R1-Distill-Qwen-7B on AIME24, MATH-500, and GPQA Diamond.

1 Introduction

The introduction presents intermediate reasoning and test-time scaling, allocating additional computation during answer generation, as ways to improve difficult multi-step problem solving. It asks how these benefits can be transferred to small language models whose limited capacity makes training sensitive to data and optimization choices.

The proposed sequence establishes reasoning through two distillation stages, reuses incorrect teacher responses for preference learning, and finally optimizes verified answer correctness through reinforcement learning.
The contribution is a coordinated training recipe informed by earlier work emphasizing either large-scale reinforcement learning or small, carefully selected reasoning datasets.

2 Background

Direct training on LIMO and S1K, two small reasoning datasets, reduces Phi-4-Mini’s accuracy in the reported experiments. Table 1 shows MATH-500 accuracy falling from 71.8% to 57.8% with LIMO and to 47.0% with S1K, while the complete recipe reaches 94.6%.

AIME 2024 and GPQA Diamond accuracy also declines under both direct-distillation settings.
These observations motivate extensive initial reasoning training before selective fine-tuning, although they do not establish that every small model requires the same sequence.

Pass@1 results for Phi-4-Mini, direct fine-tuning with LIMO or S1K, and Phi-4-Mini-Reasoning with the full recipe.

3 Multi-Stage Continual Training for Reasoning

The method starts from pretrained Phi-4-Mini and applies four consecutive stages: distillation as mid-training, distillation as supervised fine-tuning, rollout preference learning, and reinforcement learning with verifiable rewards. A rollout is a sampled model response containing its reasoning and answer.

3.1 Distillation as Mid-Training

Mid-training uses next-token prediction on diverse synthetic reasoning examples across domains and difficulty levels. DeepSeek-R1 generates reasoning answers, and rejection sampling, filtering generated responses by correctness, retains accepted answers.

Packing places multiple short examples in one input sequence to improve training efficiency.
The authors recommend repeatedly using extensive CoT data until validation performance saturates, but provide no quantitative data-scaling curve or saturation threshold.

3.2 Distillation as Supervised Fine-tuning

Supervised fine-tuning, training on selected question–response examples, uses a compact, representative, high-quality subset of the mid-training data. The subset spans diverse mathematics domains with difficulty exceeding the ‘college level’, and training uses non-packing mode, keeping examples separate so the model learns where to stop generating.

3.3 Rollout Preference Learning

Rollout preference learning reuses incorrect responses discarded during distillation. Questions classified by GPT-4o-mini as high-school mathematics or above supply pairs in which correct responses are preferred and incorrect responses are rejected.

The authors identify incorrect responses differing only subtly from correct counterparts as informative preference candidates.
DPO trains the model to favor correct over incorrect responses relative to a fixed reference model, using a previously collected dataset.

3.4 RL with Verifiable Reward

The final stage performs online reinforcement learning, learning from newly sampled responses, on the distilled, preference-trained model. The paper presents Proximal Policy Optimization (PPO), which clips changes in response probabilities during optimization, and Group-based Relative Policy Optimization (GRPO), which estimates each response’s advantage, its reward relative to a baseline, by comparing responses to the same question.

PPO uses Generalized Advantage Estimation (GAE), which combines discounted reward-prediction errors from a value function, a predictor of future reward.
GRPO standardizes rewards within each response group and includes a Kullback–Leibler (KL) penalty, a measure of divergence from a reference model’s response distribution.
The verifiable reward is +1 when the final answer passes verification against the ground truth and −1 otherwise; this reward does not directly score intermediate reasoning.

Pilot GRPO training encounters variable response lengths, groups with identical rewards and no useful relative reward signal, and a mismatch between training and evaluation sampling temperatures, which control sampling randomness. Correct responses to the same prompt can span approximately 12k to 20k tokens, and difficult questions can require a response batch size of 128 to obtain even one positive sample.

Prompt Optimization retains candidate prompts whose sampled responses have relatively uniform token lengths.
Reward Rebalancing through Oversampling and Filtering retains all positive responses and an equal number of randomly sampled negative responses. It also removes overly easy prompts using a group-accuracy threshold, with 50% given as an example.
Temperature Annealing linearly reduces sampling temperature from 1.0 to 0.6 over the first 50% of training steps, then holds it at 0.6.

4 Synthetic CoT Data Generation

The dataset combines public resources and in-house seed data, totaling around 10 million rollouts across 1.6 million samples. Existing reasoning annotations are reused; otherwise, mathematical questions receive approximately eight sampled responses each from DeepSeek-R1 (671B), a 671-billion-parameter teacher model.

A mathematics-verification tool checks answers, and GPT-4o-mini rechecks responses initially marked incorrect to reduce false negatives, correct answers mistakenly rejected by the tool.
Annotations record domain, difficulty, and repetitive patterns. Domains include algebra, geometry, theory, probability, and calculus; difficulty ranges from elementary school through graduate level.
Mid-training uses the full dataset subject to the correctness filtering described in Section 3.1, while subsequent stages use selected subsets. The paper does not specify the exact sizes of those subsets.

Public data resources, their listed sizes, and whether they provide reasoning annotations for construction of the training dataset.

5 Experiment

The experiments compare final benchmark accuracy, track performance after successive training stages, and examine multi-sample problem-solving coverage and reinforcement-learning stability. The evaluated base model is Phi-4-Mini with 3.8 billion parameters.

5.1 Evaluation

Evaluation covers AIME24, MATH-500, and GPQA Diamond, using temperature 0.6, top_p 0.95, and maximum sequence length 32K. The top_p setting restricts sampling to a set of tokens with cumulative probability 0.95, and reported performance averages 3 runs for each task.

The paper reports average accuracy without confidence intervals or per-run dispersion.

5.2 Baselines

Baselines include o1-mini, DeepSeek-R1-Distill-Qwen-7B, DeepSeek-R1-Distill-Llama-8B, Bespoke-Stratos-7B, OpenThinker-7B, Llama-3.2-3B-Instruct, and the original Phi-4-Mini.

Table 3 marks o1-mini, Bespoke-Stratos-7B, and OpenThinker-7B results as taken from published reports. The remaining results were reproduced by the authors, so the table combines different sources of evaluation evidence.

5.3 Training Settings

For the first two distillation stages, the paper specifies batch size 128, learning rate 1e-5, a total of 5 training epochs, and warmup ratio 0.1, the fraction of training used to increase the learning rate initially. Sequence length is 16K with packing during mid-training and 20K without packing during supervised fine-tuning.

Rollout DPO uses learning rate 5e-7, one epoch, and sequence length 16K.
Reinforcement learning uses learning rate 5e-7 and sequence length 25k.

5.4 Results

Final pass@1 scores are 57.5% on AIME, 94.6% on MATH-500, and 52.0% on GPQA Diamond, exceeding every open-source baseline listed in Table 3. Relative to DeepSeek-R1-Distill-Qwen-7B, the gains are 4.2, 3.2, and 2.5 percentage points, respectively.

Successive Phi-4-Mini checkpoints score 10.0/71.8/36.9 initially, 30.0/82.9/42.6 after mid-training, 43.3/89.3/48.3 after fine-tuning, 50.0/93.6/49.0 after Roll-Out DPO, and 57.5/94.6/52.0 after reinforcement learning, in AIME/MATH-500/GPQA Diamond order.
Against the published o1-mini scores, the final model leads on MATH-500, 94.6% versus 90.0%, but trails on AIME, 57.5% versus 63.6%, and GPQA Diamond, 52.0% versus 60.0%.

Pass@1 reasoning performance of comparison models and successive Phi-4-Mini training checkpoints on AIME, MATH-500, and GPQA Diamond.

5.5 Ablations

The ablations use pass@k, the fraction of problems with at least one verified correct answer among k sampled responses, to assess problem-solving coverage. Figure 2a shows large gains from the base model to the Roll-Out DPO checkpoint, followed by additional gains from reinforcement learning; the authors describe the latter as approximately 7 points on average.

Figure 2b compares the proposed reinforcement-learning recipe with DAPO, an existing reinforcement-learning training system, using consensus@16 accuracy on AIME 2024. The paper names this consensus-based evaluation metric but does not define its aggregation procedure; the proposed recipe remains comparatively stable while DAPO deteriorates in this setting.
The pass@k plot compares accumulated training checkpoints, so it does not isolate DPO from the preceding distillation stages. The reinforcement-learning comparison evaluates the combined recipe without separately removing prompt selection, reward balancing, or temperature annealing.

AIME 2024 pass@k curves for the base, Roll-Out DPO, and additional reinforcement-learning checkpoints, alongside consensus@16 training curves comparing the proposed recipe with DAPO.

5.6 Safety Statement

The authors state that development followed Microsoft’s responsible AI principles and that Azure AI Foundry’s Risk and Safety Evaluation framework assessed harmful content, direct jailbreak, meaning attempts to bypass safeguards, and model groundedness, whether responses are supported by evidence. They refer readers to the Phi-4-Mini-Reasoning Model Card for further information, but provide no numerical safety results in this paper.

6 Conclusion

The conclusion attributes the model’s reasoning performance to coordinated large-scale distillation, rollout preference learning, and reinforcement learning with verifiable rewards. The reported comparisons show that this 3.8-billion-parameter model exceeds the listed 7B and 8B open-source baselines on these evaluations.

Appendix

The supplied 12-page paper contains no appendix. References occupy pages 10–12.

Brief Thoughts

The clearest evidence is the improvement across all three benchmarks at successive training checkpoints, alongside the accuracy declines from direct LIMO and S1K training in Table 1. These results support the recipe for Phi-4-Mini, while the sequential design leaves the optimal stage order and necessity of each component unresolved. Reproducibility is limited by in-house seed data, unspecified subset sizes, and incomplete quantitative criteria for prompt selection and filtering. Small parameter count does not establish low total cost: the recipe uses a 671B teacher, around 10 million rollouts, and long generation sequences, without reporting training compute, inference latency, or a cost-matched comparison. The evaluation supports gains on the reported benchmarks. Additional base models and independent component ablations would be needed to test the recipe’s broader applicability.