Tina: Tiny Reasoning Models via LoRA | Summary
22 Apr 2025 | Paper Review LoRA Reinforcement Learning With Verifiable Rewards Mathematical ReasoningContents
This article explains the key points of Tina: Tiny Reasoning Models via LoRA.
- 2025-04-22 (arXiv)
- Wang, Shangshang, Asilis, Julian, Akgül, Ömer Faruk, Bilgin, Enes Burak, Liu, Ollie, Neiswanger, Willie.
- University of Southern California
- Paper
- Github
- Project Page
Source and license. This article summarizes Tina: Tiny Reasoning Models via LoRA, arXiv:2504.15777v1 by Shangshang Wang, Julian Asilis, Ömer Faruk Akgül, Enes Burak Bilgin, Ollie Liu, and Willie Neiswanger. The arXiv record lists the paper under CC BY 4.0. The images below are crops from the cited version's PDF: Figure 1 (PDF p. 2), Figure 2 (PDF p. 3), Table 1 (PDF p. 6), Table 2 (PDF p. 7), Table 3 (PDF p. 8), Table 4 (PDF p. 9), Figure 3 (PDF p. 10), and Figure 4 (PDF p. 11).
Summary
- Tina investigates how cheaply additional training can improve a language model with 1.5 billion parameters on mathematical and scientific problem solving. Its best checkpoint, a saved model state, achieves a 50.60% average across six benchmarks versus 41.60% for Open-RS2, which updates all model parameters, with both models built on DeepSeek-R1-Distill-Qwen-1.5B and evaluated under standardized settings.
- The method combines low-rank adaptation (LoRA), which trains small matrix updates while freezing pretrained weights, with reinforcement learning (RL), which optimizes generated responses using reward signals. Training uses Group Relative Policy Optimization (GRPO), which compares rewards within groups of sampled answers without a separate network for predicting expected rewards, on two NVIDIA L40S graphics processing units (GPUs).
- The best Tina-Open-RS2 checkpoint scores 43.33% on AIME24, the 2024 American Invitational Mathematics Examination benchmark, under zero-shot pass@1 evaluation: correctness from one generated answer without task demonstrations. Its estimated $9 cost covers $8 of training to that checkpoint and $1 of evaluation; reproducing all reported experiments costs $526.
- The authors hypothesize that LoRA rapidly adapts reasoning-output structure while preserving existing knowledge, based on changes in format rewards, which score adherence to required response structure, and response lengths during training. This evidence is indirect: the experiments use one base model already trained to imitate stronger reasoning models through distillation, and the appendix documents exceptions to the proposed relationship between training transitions and peak performance.
1. Introduction
The introduction frames reasoning improvement as a resource problem: supervised fine-tuning, which learns from example responses, often uses reasoning traces, or step-by-step solutions, generated by stronger models, and obtaining those demonstrations can be costly. RL pipelines can also demand substantial computation. Tina starts from the already distilled DeepSeek-R1-Distill-Qwen-1.5B model and measures the additional gains obtainable with small parameter updates.
- The reported budget covers additional training and evaluation, excluding the creation of the pretrained and distilled starting checkpoint.
- The authors release code, training logs, evaluation scripts, and checkpoints to support reproduction.
Tina has the highest displayed six-task average, 50.60%, with a $14 training-cost estimate, compared with 48.74% and $3629 for DeepScaleR. Appendix A specifies that the resource bars describe full runs and exclude evaluation, so the $14 bar and the abstract's $9 checkpoint estimate have different accounting scopes.
2. Related Work
The related-work section reviews public reasoning-model reproductions, RL with checkable rewards, and adaptation that trains only a small subset of parameters. These strands motivate Tina’s combination of a small starting model, existing public training resources, and LoRA-based RL.
2.1. Open-Source Reasoning Replicas
Open-source reasoning replicas are public models intended to reproduce the reasoning performance of systems such as o1-preview and DeepSeek-R1. The review covers curated demonstrations, lightweight RL, response-length control, and small-model training, including STILL, Sky-T1, SimpleRL, s1, DeepScaleR, and Open-RS.
- The cited projects provide precedents and public resources, but use different training recipes.
The timeline places Tina among public reasoning-model projects released from November 2024 through April 2025. It gives historical context for the comparison choices without measuring relative performance or training efficiency.
2.2. RL with Verifiable Rewards
Verifiable rewards provide training feedback from checkable properties such as answer correctness or required response structure. The paper reviews learned reward evaluators, rule-based verification, and methods that encourage exploration before describing GRPO.
- GRPO estimates each response’s advantage, its reward relative to the sampled group, using other responses to the same question.
- Dr.GRPO modifies GRPO to address its bias toward long responses; Tina evaluates it in an ablation, an experiment that varies a selected design choice.
2.3. Low-Rank Adaptation
LoRA represents a weight update as the product of two low-rank matrices, whose small shared dimension limits the number of trainable parameters. Tina tests whether these restricted updates can produce reasoning improvements comparable to full-parameter training; the resulting adapters, or separately stored update parameters, also allow behavior changes without storing multiple complete model copies.
3. Tina: Tiny Reasoning Models via LoRA
Tina is a family of DeepSeek-R1-Distill-Qwen-1.5B models trained further with LoRA during GRPO-style RL. Its efficiency strategy combines a small starting model and restricted parameter updates with public datasets, shared training infrastructure, two-GPU execution, and a limited experimental budget.
3.1. Training Pipeline: Baselines & Datasets
All Tina models and comparison baselines share the same underlying 1.5B checkpoint. Tina variants reuse the datasets and reward implementations of STILL-3-1.5B-preview, DeepScaleR-1.5B-Preview, and Open-RS1/2/3.
- STILL uses 33k reasoning examples from competition-mathematics resources, including MATH, a mathematics problem dataset; NuminaMathCoT, a mathematical reasoning dataset; and AIME problems from 1983–2023.
- DeepScaleR uses approximately 40k problem-answer pairs from AIME, AMC, the American Mathematics Competition; OMNI-MATH, an olympiad-level mathematics dataset; and STILL. Tina mirrors its dataset and reward design.
- Open-RS variants use curated datasets derived from s1 and DeepScaleR, with Tina retaining the corresponding data splits and reward implementations.
3.2. Training Setup: Infrastructure & Budget
The implementation builds on OpenR1, an open reproduction framework for DeepSeek-R1, and runs training alongside vLLM, the response-generation engine, on two NVIDIA L40S GPUs. Most hyperparameters, the settings chosen before training, follow existing recommendations and remain fixed; reward settings vary by task, and individual factors vary in ablations.
- Reported costs assume approximately $1 USD per GPU hour, including 300 GB storage, at the pricing observed when the paper was written. Some runs used NVIDIA RTX 6000 Ada GPUs, but cost reporting uses L40S hardware as its reference.
- A training step typically completes within one minute, and evaluating one checkpoint across all six benchmarks requires approximately one L40S GPU hour.
- Table 1 distinguishes the $9 path to the selected best checkpoint from $31 for its full training run with checkpoint evaluations, $275 for all main tasks, and $526 for all experiments.
All tasks total $396 in training and $130 in evaluation, while the selected best checkpoint requires $8 and $1, respectively. The $17 evaluation estimate for all checkpoints in the best-performing task exceeds its $14 training estimate, illustrating the cost of checkpoint selection.
4. Surprising Effectiveness of Efficient RL Reasoning via LoRA
The experiments first re-evaluate baselines under common settings, then compare five Tina variants with their paired baselines, and finally vary training data, learning rate, adapter rank, and RL algorithm. Performance comparisons use the average percentage score across six mathematical and scientific reasoning benchmarks.
4.1. Experiments Stage I: Baseline Model Re-Evaluation
The authors re-evaluate all baselines with lighteval, the evaluation framework, and vLLM on two L40S GPUs to reduce differences caused by incompatible evaluation settings. All results use zero-shot pass@1 with a common generation configuration.
- The six tasks are AIME24 and AIME25, with 30 competition-mathematics problems each; AMC23, with 40 problems from the 2023 competition; MATH500, with 500 competition problems; GPQA Diamond, with 198 doctoral-level science questions; and Minerva, with 272 quantitative reasoning problems.
- Evaluation allows 32768 new tokens, the units of generated text, with temperature 0.6 controlling sampling randomness and top_p 0.95 restricting sampling to a cumulative-probability subset. The generation limit exceeds the 3584-token maximum completion length used during training.
- The starting model averages 41.18%; re-evaluated full-parameter baselines range from 41.60% for Open-RS2 to 48.74% for DeepScaleR-1.5B-Preview.
Common evaluation settings give the starting model a 41.18% average and DeepScaleR the highest baseline average, 48.74%. Different baselines lead on different tasks, so the aggregate ranking does not describe every benchmark.
4.2. Experiments Stage II: Tina Model Evaluation
The five selected Tina checkpoints average 48.16%–50.60% across the six tasks after 19%–57% of one epoch, a pass through the training data. Four outperform their corresponding full-parameter baseline averages; Tina-DeepScaleR-1.5B-Preview scores 48.38%, slightly below its baseline’s 48.74%.
- Tina-Open-RS2 achieves the highest average, 50.60%, compared with 41.60% for Open-RS2: a gain of 9.00 percentage points and more than 20% relative improvement.
- Its task scores are 43.33% on AIME24, 26.67% on AIME25, 77.50% on AMC23, 87.00% on MATH500, 36.36% on GPQA, and 32.72% on Minerva.
- The headline checkpoint is step 450, about 51% of one epoch. These comparisons use selected checkpoints, and later checkpoints fluctuate substantially.
Four of the five Tina variants exceed their paired baseline averages. Tina-Open-RS2 leads at 50.60% after 51% of an epoch, while the DeepScaleR counterpart scores slightly below its paired baseline.
4.3. Experiments Stage III: Tina Ablation Variants
Larger datasets and higher adapter ranks do not consistently improve the selected checkpoint’s average. In the algorithm comparison, Dr.GRPO reaches a similar peak score to GRPO at an earlier checkpoint.
- Across seven datasets, the 7k-example Open-RS configuration reaches 50.60%, compared with 49.26% for 93.7k-example OpenR1 and 48.47% for LIMR, the 1.39k-example dataset used in several ablations. Dataset size alone does not explain this ranking, but differing reward settings prevent attributing it solely to data quality.
- On LIMR, learning rates, which control update size, of 5e-6, 1e-6, and 5e-7 yield 47.87%, 48.47%, and 47.91% in Table 4. Adapter ranks 4, 8, 16, 32, and 64 yield 47.72%, 47.89%, 48.92%, 48.47%, and 46.95%, respectively; the rank experiments also change the adapter scaling parameter alpha.
- On Tina-Open-RS3, GRPO reaches 49.45% at 57% of an epoch, while Dr.GRPO reaches 49.53% at 17%. This comparison establishes earlier peak performance in the reported setup, without demonstrating a general algorithm ranking.
The selected checkpoints show no monotonic improvement with dataset size or adapter rank. Dr.GRPO peaks at 49.53% after 17% of an epoch, compared with GRPO's 49.45% after 57%, demonstrating earlier peak performance in this particular setup.
5. Hypothesis for Effective and Efficient LoRA: Rapid Format Adaptation
The rapid format adaptation hypothesis proposes that small updates improve how existing knowledge is expressed in rewarded reasoning structures. Its evidence combines approximate training-compute comparisons with logs showing changes in format rewards and completion lengths, or generated response lengths, that are less distinct in answer-correctness rewards.
- Figure 3 plots reasoning scores against estimated training floating-point operations, a measure of computational work. The estimates use reported hardware and training durations; the figure is not a controlled experiment varying compute alone.
- Figure 4 places the best Tina-Open-RS1 and Tina-Open-RS2 checkpoints before marked changes in format-related metrics. Its curves use an exponential moving average, which smooths observations over time, with factor 0.1.
- These observations do not directly measure knowledge preservation or establish that format learning causes the benchmark gains. Appendix E reports later peaks for OpenR1 and OpenThoughts and no comparable clear transition in LIMR variants.
Several Tina configurations combine lower estimated training computation with higher average scores than the full-parameter baselines. The points differ in datasets, rewards, and training conditions, so their arrangement does not establish that reducing compute itself improves reasoning.
The marked best Tina-Open-RS1 and Tina-Open-RS2 checkpoints precede the marked changes in format rewards and response lengths, while answer-correctness rewards show less distinct changes. These correlations motivate the format-adaptation hypothesis without directly measuring knowledge preservation.
6. Conclusion
The conclusion emphasizes reasoning improvements from LoRA-based RL at low incremental computational cost. Its explicit limitations are the use of one 1.5B base-model scale, evaluation concentrated on mathematical and scientific reasoning, and deliberately limited hyperparameter optimization.
- Transfer to coding and other domains remains untested.
- The reported experiments directly compare models built on the same 1.5B starting checkpoint; they do not establish broad equivalence to larger reasoning models.
Appendix
- A. Cost Breakdown clarifies that the resource comparisons in Figures 1 and 3 concern full epochs or predefined runs and exclude evaluation. The abstract’s $9 instead covers training to the selected best checkpoint plus evaluating it; Appendix A gives $14 for the corresponding full training run and approximately $31 including evaluations. Baseline resource estimates come from prior technical reports, so the abstract’s estimated 260x reduction combines different accounting scopes; Table 1 also lists $15 training and $32 total for the main Tina-Open-RS2 task, differing from its $14 and $31 summary row.
- B. Background behind Tina Training supplies B.1. GRPO Formulation and B.2. LoRA Formulation. GRPO normalizes rewards within each sampled group, limits changes in response-probability ratios, and applies a Kullback–Leibler divergence penalty, which discourages departure from a reference model’s output distribution. LoRA changes the forward computation to \(\hat{h}(x)=W_0x+ABx\), with frozen weights W₀ and trainable matrices A and B whose shared dimension r is much smaller than the original matrix dimensions.
- C. Additional Experimental Details, C.1. Hyperparameters gives default LoRA rank 32, update scaling parameter alpha 128, dropout 0.05, and query, key, value, dense target modules. Training uses the AdamW optimizer with momentum coefficients 0.9 and 0.999, learning rate 1e-6, a cosine learning-rate schedule with a minimum, warmup ratio 0.1, mixed bfloat16 numerical precision, gradient accumulation over 4 steps, total batch size 32, and one epoch. Maximum prompt and completion lengths are 512 and 3584, with four generated responses per question, vLLM GPU memory utilization 0.4, and vLLM maximum model length 4608; Table 6 records reward and ablation changes. C.2. Evaluation Command specifies dtype=bfloat16, data_parallel_size=2, max_model_length=32768, gpu_memory_utilization=0.5, and generation_parameters={max_new_tokens:32768,temperature:0.6,top_p:0.95}. Evaluation uses lighteval vllm with the chat template, which formats model inputs, and the OpenR1 custom task script at https://github.com/huggingface/open-r1/blob/4f5b21e21dec473af9729bce8e084deb16223ae4/src/open_r1/evaluate.py.
- D. Full Tina Model Performance Evaluation lists checkpoint results in Tables 7–21, showing substantial fluctuations: Tina-Open-RS2 averages 50.60% at step 450 and 43.05% at step 500. Some entries conflict with the main summaries: Table 16 repeats Table 12’s default LIMR results despite its 5e-6 label and disagrees with Table 4’s 47.87% peak for that learning rate. Table 18’s caption states alpha 512, while Table 6 lists 256 for rank 64.
- E. Full Tina Model Training Phase Transition extends the training-log analysis across Figures 5–12. OpenR1 and OpenThoughts achieve their best evaluated checkpoints after the marked turning point, while LIMR variants lack a comparable clear transition; the authors suggest that the short LIMR runs may explain the latter observation. These exceptions limit the evidence for using the transition as a general rule for stopping training.
Brief Thoughts
Tina’s practical result is that LoRA-based RL improves four of five paired baseline averages under a common evaluation protocol while keeping additional training costs low. Its explanation is less established: training-log correlations do not isolate format adaptation from other effects, and the appendix documents exceptions. The $9 estimate reproduces a known winning checkpoint. Finding that checkpoint through repeated evaluation has a different cost, represented more fully by the reported run-level accounting. The paper provides no repeated-seed uncertainty estimates for these comparisons. Small benchmark sets, stochastic generation, and checkpoint fluctuations leave small score differences difficult to interpret confidently.