Reinforcement Learning for Reasoning in Small LLMs: What Works and What Doesn't | Summary

Contents

Summary
1 Introduction
2 Methodology
- High-Quality Dataset Curation
- Reinforcement Learning Algorithm
3 Experiments
4 Conclusion
Appendix
Brief Thoughts

This article explains the key points of Reinforcement Learning for Reasoning in Small LLMs: What Works and What Doesn't.

2025-03-20 (arXiv)
Dang, Quy-Anh, Ngo, Chris.
VNU University of Science, Vietnam, Knovel Engineering Lab, Singapore
Paper
Github

Read this article in Korean

Summary

This paper tests whether reinforcement learning (RL), training through reward feedback, can improve mathematical reasoning in small large language models (LLMs) under limited compute. Starting from DeepSeek-R1-Distill-Qwen-1.5B, Open-RS3 reaches 46.7% on AIME24, the 2024 American Invitational Mathematics Examination, versus the externally reported o1-preview score of 44.6% under zero-shot pass@1, the proportion solved on the first attempt without example demonstrations; the authors use 7,000 training questions and estimate $42 for a 24-hour training run.
The method combines filtered mathematical questions with Group Relative Policy Optimization (GRPO), an RL algorithm that compares rewards within groups of generated answers without a separate critic model for estimating expected rewards. Three experiments vary question difficulty, response-length limits, and rewards for correctness, structure, and length; Open-RS2 reaches 80.0% on AMC23, the 2023 American Mathematics Competition, versus the base model’s reported 62.9%.
Most gains appear within 50–100 global training steps, the study’s training-update count, while continued training brings declining accuracy and mixed-language outputs. Open-RS3 averages 56.3% across five benchmarks, below DeepScaleR-1.5B-Preview’s 57.0%, supporting inexpensive improvements on selected mathematical tasks while leaving sustained optimization and generalization beyond mathematics unresolved.

1 Introduction

Reinforcement Learning for Reasoning in Small LLMs: What Works and What Doesn’t asks how small models behave under strict training constraints and whether an approach inspired by DeepSeek-R1 can improve their reasoning. The study uses a 1.5-billion-parameter model trained on 4 NVIDIA A40 graphics processing units (GPUs), each with 48 GB of memory, within a 24-hour window, and releases code and datasets at https://github.com/knoveleng/open-rs.

The contribution is an empirical study of resource-constrained training behavior, including early gains, later instability, and comparisons with larger or more extensively trained models. Figure 1 summarizes reported AIME24 performance, model sizes, and estimated training costs.

Reported AIME24 zero-shot pass@1 against model size and estimated training cost.

2 Methodology

The methodology has two components: curate mathematical questions that challenge a small model, then apply GRPO to train it from reward feedback. Rule-based rewards evaluate correctness, response structure, and length without a separately trained neural reward model.

High-Quality Dataset Curation

The authors refine s1, a general-purpose reasoning-question collection, and DeepScaleR, a mathematics-question collection, into open-s1 and open-deepscaler, producing 39,659 mathematical questions in total. They use different filtering models to diversify the selection criteria and seek to reduce excessive overlap.

For s1, the initial 59,029 questions become 31,323 after retaining solutions containing the LaTeX answer command \boxed{}. DeepSeek-R1-Distill-Qwen-1.5B removes trivial questions, leaving 21,533; Qwen2.5-7B-Instruct removes noisy or multi-part questions, leaving 18,615 open-s1 examples.
DeepScaleR initially contains 40,315 mathematical questions from AIME (1984–2023), AMC (prior to 2023), Omni-MATH, and Still. Qwen2.5-Math-7B-Instruct removes easy questions, leaving 21,044 open-deepscaler examples.
The later 7,000-question experiments use 3,000 open-s1 questions, 3,000 open-deepscaler questions, and 1,000 easier questions from raw DeepScaleR. These experiments therefore use a subset of the available questions.

Reinforcement Learning Algorithm

GRPO samples a group of answers for each question and normalizes each answer’s reward using the group mean and standard deviation. Its objective combines a clipped update, which limits changes in answer probabilities relative to the previous model, with a Kullback–Leibler (KL) divergence penalty, which discourages the trained model’s output distribution from departing from a reference model.

The accuracy reward assigns 1 for a correct final answer and 0 otherwise, requiring the answer in \boxed{} format for verification.
The cosine reward scales correctness feedback with a cosine schedule over response length: shorter correct solutions receive higher rewards, while longer incorrect solutions receive less severe penalties.
The format reward gives positive feedback when reasoning appears inside <think> and </think> tags. Together, these rules target answer correctness, output length, and response structure.

3 Experiments

Three experiments examine high-quality data alone, a mixture of easy and hard questions, and a reward that depends on response length. Evaluation tracks training behavior and compares selected checkpoints, saved model states at particular training steps, across five mathematical benchmarks.

Experimental Setup

All experiments start from DeepSeek-R1-Distill-Qwen-1.5B, a small model distilled from larger models, and skip an additional supervised fine-tuning (SFT) phase, meaning training on supplied target responses. The implementation adapts Hugging Face’s open-r1, an open-source reproduction of DeepSeek-R1, and uses 4 NVIDIA A40 GPUs, six generated answers per question, and completion limits of 4096 or 3584 tokens, the units of generated text.

The configuration specifies one epoch, a pass through the training dataset, but Experiment 1 is capped at 500 global steps within 24 hours, compared with approximately 1500 steps needed for its full dataset.
Appendix E specifies learning_rate = 1.0e-06, gradient_accumulation_steps = 4, temperature = 0.7, seed = 42, and checkpoint saving every 50 steps. These control the update size, accumulation of gradients before an update, sampling randomness, random initialization of the run, and saving frequency, respectively.

Benchmark Datasets

The evaluation uses AIME24, MATH-500, AMC23, Minerva, and OlympiadBench, containing 30, 500, 40, 272, and 675 problems, respectively. MATH-500 is a competition-mathematics subset; Minerva evaluates undergraduate quantitative reasoning, with the mathematics subset used here; OlympiadBench contains Olympiad-level mathematics and physics problems.

The paper identifies AIME24 and AMC23 through https://huggingface.co/datasets/AI-MO/aimo-validation-aime and https://huggingface.co/datasets/AI-MO/aimo-validation-amc.

Baseline Models

Baselines include Llama-3.1-70B-Instruct and o1-preview; the 7-billion-parameter models Qwen-2.5-Math-7B-Instruct, rStar-Math-7B, Eurus-2-7B-PRIME, and Qwen2.5-7B-SimpleRL; and the 1.5-billion-parameter models DeepSeek-R1-Distill-Qwen-1.5B, DeepScaleR-1.5B-Preview, and Still-3-1.5B-Preview. Table 1 combines externally reported baseline scores with the authors’ evaluations of Open-RS using lighteval, an evaluation package.

Evaluation Metric

Evaluation uses zero-shot pass@1, measuring whether the first answer is correct without example demonstrations in the prompt. Final answers must use \boxed{} format for automated evaluation.

Process and Results

Experiment 1: Impact of High-Quality Data trains on 18,615 open-s1 questions with accuracy and format rewards and a 4096-token completion limit. Within 50–100 steps, AMC23 rises from approximately 63% to 70% and MATH-500 from approximately 83% to 84%, but accuracy deteriorates around 200 steps and remains below its early peak at later checkpoints.

Figure 2 shows the benchmark decline, while Figure 3 shows noisy accuracy rewards and completion lengths that shorten before sharply increasing again.
The authors report unreadable and non-English outputs during later training. They interpret responses reaching the length limit before a final answer and subsequent language drift as evidence of difficulty with open-s1 questions and reward misalignment, where optimizing the reward fails to preserve the intended behavior.

AMC23 and MATH-500 performance across global training steps for the three experiments, with initial baselines shown as red dashed lines.

Accuracy reward and completion length across local steps in Experiment 1.

Experiment 2: Balancing Easy and Hard Problems uses the 7,000-question mixture, retains accuracy and format rewards, and lowers the completion limit to 3584 tokens. AMC23 reaches 80% and MATH-500 approximately 85% within 50–100 steps, but performance declines after 150–200 steps as KL divergence rises sharply and mixed-language outputs return.

Initial completions shorten to approximately 2800 tokens. The revised configuration improves early performance, but changing dataset composition and completion length together prevents attributing the improvement to either change alone.

KL divergence and completion length across local steps in Experiment 2.

Experiment 3: Controlling Length with Cosine Reward keeps the same 7,000 questions and 3584-token limit, replaces the accuracy reward with the cosine reward, and adds the prompt instruction “Reply in English only, do not use other languages”. At 50 steps, AMC23 reaches 72.5% and MATH-500 84.4%; reported completion lengths span approximately 1000–3500 tokens, but mixed-language content persists after 200 steps.

The authors interpret these length traces as improved control of response length. Figure 5 still shows substantial later divergence from the reference model, so the observed length regulation does not establish stable training overall.

Overall Comparison selects Open-RS1 at 100 steps, Open-RS2 at 50 steps, and Open-RS3 at 50 steps, yielding five-benchmark averages of 53.0%, 55.7%, and 56.3%. Open-RS3 leads the reported AIME24 comparison at 46.7%, while Open-RS2 achieves 80.0% on AMC23 and ties Open-RS1 at 52.4% on OlympiadBench.

DeepScaleR-1.5B-Preview retains the highest average at 57.0% and the highest MATH-500 score at 87.8%. Open-RS3 scores 26.8% on Minerva, below the listed 7B baselines with available results.
The paper estimates approximately $42 for 24 hours on 4 A40 GPUs, compared with $1088 for Eurus-2-7B-PRIME, $1633 for Qwen2.5-7B-SimpleRL, $3629 for DeepScaleR-1.5B-Preview, and $2268 for Still-3-1.5B-Preview. The $42 figure describes the reported training budget rather than a measured cost for reaching the selected 50-step checkpoint.
Tables 2 and 3 compare training data, hardware, and duration. The costs use https://www.runpod.io/pricing and externally reported configurations, so they provide resource estimates across different training procedures rather than a controlled efficiency comparison.

Zero-shot pass@1 results for general models, 7B models, 1.5B models, and selected Open-RS checkpoints across five benchmarks.

KL divergence and completion length across local steps in Experiment 3.

Reported data usage, training hardware, duration, and estimated costs for 7B comparators and Open-RS.

Reported training data and compute requirements for 1.5B reasoning models sharing the same base model.

4 Conclusion

The paper concludes that additional RL training can improve an existing small reasoning model with limited data and compute. Its experiments show early mathematical benchmark gains followed by recurring degradation during continued training, making checkpoint selection and completion limits central to interpreting the reported results.

Appendix

A Related Work covers Reasoning in Large Language Models and Reasoning with Reinforcement Learning. It discusses scratchpad prompting, which asks for intermediate computations; chain-of-thought prompting, which elicits intermediate reasoning steps; verification of generated answers; self-reflection, where models revise their reasoning; SFT; and reward-driven training. The section positions the study within efforts to reduce the resource requirements of reasoning-model training.
B Limitations & Discussion identifies four limitations: the 24-hour training window, insufficient completion lengths for extremely hard questions, language drift into unintended languages after roughly 150–200 steps, and evaluation restricted to mathematical reasoning. Its Discussion interprets the results as a trade-off among efficiency, reasoning depth, and training stability; the cosine reward regulates length but does not produce the highest AMC23 or MATH-500 score.
Future Directions proposes longer training or multi-stage length schedules, lightweight language rewards or monolingual filtering, evaluation beyond mathematics, and combinations of GRPO with Monte Carlo Tree Search, a method for exploring a tree of candidate reasoning paths. These proposals are not evaluated in the experiments.
C Datasets describes AIME24 as advanced high-school mathematics, AMC23 as foundational competition mathematics, MATH-500 as a competition-problem subset, Minerva as undergraduate quantitative reasoning with the mathematics subset used, and OlympiadBench as Olympiad-level mathematics and physics. D Baseline Models groups the comparisons into general-purpose models, mathematics-focused 7B models, and mathematics-focused 1.5B models. Its descriptions distinguish methods using search, rewards for intermediate reasoning steps, or final-answer rewards, and identify the distilled model as the baseline before the additional training studied here.
E Hyperparameter Setup lists the GRPO trainer configuration. Selected settings are bf16 = true, enabling a reduced-precision numerical format; use_vllm = true, enabling the vLLM generation engine; vllm_gpu_memory_utilization = 0.7; vllm_max_model_len = 4608; max_prompt_length = 512; max_steps = 500; num_generations = 6; and reward_weights = 1.0, 2.0 for format and accuracy or cosine rewards. The learning-rate scheduler is cosine_with_min_lr with min_lr_rate = 0.1 and warmup_ratio = 0.1, specifying a cosine schedule with a minimum learning rate and an initial warmup period.

Brief Thoughts

The clearest practical finding is the value of evaluating early checkpoints: all three experiments improve quickly, while continued training repeatedly loses accuracy. The study demonstrates low-cost additional training of an already distilled reasoning model; it does not test whether the same budget could produce comparable reasoning ability from an ordinary pretrained model.

Causal attribution remains limited because Experiment 2 changes both the dataset and completion limit, while Experiment 3 changes both the reward and language prompt. The paper reports no repeated-seed results or uncertainty intervals, and the headline AIME24 comparison uses 30 evaluation problems and externally reported baseline scores.