START: Self-taught Reasoner with Tools | Summary

This article explains the key points of START: Self-taught Reasoner with Tools.

• 2025-03-06 (arXiv)
• Li, Chengpeng, Xue, Mingfeng, Zhang, Zhenru, Yang, Jiaxi, Zhang, Beichen, Wang, Xiang, Yu, Bowen, Hui, Binyuan, Lin, Junyang, Liu, Dayiheng.
• University of Science and Technology of China, Alibaba Group
• Paper

Read this article in Korean

Summary

• START: Self-taught Reasoner with Tools addresses computational errors and unreliable code checking in language models that solve problems through extended intermediate reasoning. Compared with its QwQ-32B-Preview base model, START raises single-answer accuracy from 50.0% to 66.7% on AIME24 competition mathematics and from 41.4% to 47.3% on the evaluated LiveCodeBench programming problems, using greedy decoding, which selects the most probable next token, with at most 6 Python tool calls.
• Hint-infer inserts manually written suggestions into chain-of-thought (CoT), the model’s explicit intermediate reasoning, to elicit Python execution. Hint Rejection Sampling Fine-Tuning (Hint-RFT), training on selected successful generated solutions, first teaches the tool interaction format; START-0 then generates reasoning trajectories, sequences of reasoning steps and tool interactions, for a second training stage that produces START.
• A comparison using the same training questions supports the contribution of tool-integrated reasoning (TIR), intermediate reasoning combined with external tool calls: ordinary rejection sampling fine-tuning produces smaller improvements and reduces AIME25 competition mathematics accuracy. Evidence remains limited to one base model, one external tool, and selected science, mathematics, and programming benchmarks; additional hints can also reduce START’s accuracy.

1 Introduction

The introduction asks how extended reasoning can incorporate executable computation and feedback. The authors report that direct instructions, detailed tool instructions, and demonstrations failed to elicit Python interpreter calls in their tested settings, motivating Hint-infer: inserting suggestions after words such as “Alternatively” and “Wait,” or immediately before reasoning would stop.

• START uses only a Python interpreter, which executes generated code and returns its output to the model.
• Figure 2 illustrates the intended benefit: execution exposes incorrect sample outputs, allowing the model to revise its implementation instead of relying on imagined test results.

2 Related Work

Related work connects extended CoT, distillation, training on another model’s generated reasoning, and test-time scaling, spending additional computation during inference before answering. The paper also reviews Python-assisted mathematical reasoning and Lean, a system used for mathematical proof verification, positioning START as a method for combining extended reasoning with interpreter feedback.

3 Methodology

The methodology has two training stages, illustrated in Figure 1. Hint-infer supplies successful tool-assisted trajectories for an initial model, START-0; START-0 then generates a broader collection of trajectories used to fine-tune QwQ-32B-Preview into START.

The diagram separates learning the tool interaction format from generating the final training corpus. START-0 supplies trajectories for the second stage, while the final fine-tuning starts from QwQ-32B-Preview.

The example shows how actual execution can expose errors that verbal self-reflection misses. START revises its program after mismatched outputs, illustrating a possible mechanism behind the aggregate programming gains without establishing its frequency.

3.1 Training data

Training questions come from AIME problems before 2024, the MATH and Numina-MATH mathematics datasets, Codeforces and code contests programming problems, and LiveCodeBench before July 2024. The main text describes approximately 40K mathematics and 10K programming problems and applies the decontamination procedure of Yang et al. (2024), intended to reduce overlap with evaluation data.

• Table 5 provides source counts, but its entries sum to 46,969 rather than its stated total of 49,969; the paper therefore does not establish one consistent exact training-set size.

3.2 Hint-RFT

The Hint-Library contains mathematical suggestions for computation, reflection, checking logic, exploring alternatives, and deeper reasoning, plus programming hints that include a debugging template. For mathematics, Hint-infer inserts suggestions after selected conjunctions or before the stop token, which ends generation; for programming, it requests testing before the final solution, then returns interpreter outputs to the model.

• The seed dataset, D_seed, retains questions that greedy QwQ-32B-Preview inference fails but hint-assisted inference solves, comprising 10K mathematics and 2K programming examples.
• Rule-based scoring and repetition filtering select trajectories. Programming trajectories replace Python identifiers in the hints with “Debug Code Template” and remove output placeholders before supervised fine-tuning, training on the selected trajectories, produces START-0.
• Figure 4 shows frequent conjunctions used to motivate insertion locations; it does not establish that those locations are optimal.

Mathematics hints provide short reasons to use Python, while the programming hint supplies a longer test-and-debug template. Reproducing the procedure therefore requires task-specific hint construction as well as model-generated data.

Frequent words such as “Wait” and “Alternatively” motivate where the authors interrupt generation to insert hints. The word cloud describes the base model’s text; it does not compare the effectiveness of different insertion policies.

3.3 RFT

Rejection sampling fine-tuning (RFT) uses START-0 to generate 16 samples per training question with temperature 0.6, controlling sampling randomness, and top-p 0.95, a threshold on cumulative token probability. The authors score the trajectories, remove repetitive responses, manually modify unreasonable content, and retain at most one response per question in D_START.

• The final training stage fine-tunes QwQ-32B-Preview again using D_START, described as containing 40,000 mathematics and 10,000 programming entries.
• This use of a model’s own generated trajectories is self-distillation. Manual hint construction and content modification remain part of the procedure, so the data pipeline is not fully automated.

4 Experiment

The experiments evaluate whether Python-assisted reasoning improves science question answering, mathematical problem solving, and code generation. They compare START with its base model and other language models, then examine additional training without tools and repeated hint insertion during inference.

4.1 Benchmarks

GPQA is an expert-written graduate-level science question-answering benchmark covering biology, physics, and chemistry; the paper describes 448 multiple-choice questions. Mathematical evaluation uses MATH500, a high-school mathematics benchmark, and the competition mathematics datasets AMC23, AIME24, and AIME25.

• Programming evaluation uses 112 LiveCodeBench problems from August 2024 through November 2024, spanning easy, medium, and hard difficulty.
• Training LiveCodeBench data precedes July 2024, separating that source chronologically from the selected programming evaluation period.

4.2 Baselines

General-model baselines include Qwen2.5-32B-Instruct, Qwen2.5-Coder-32B-Instruct, DeepSeek-V3-671B, Llama3.3-70B-Instruct, and GPT-4o. Reasoning-model comparisons include the o1 series, o3-mini(low), R1-Distill-Qwen-32B, s1-32B, Search-o1-32B, and QwQ-32B-Preview.

• Table 1 marks several results as taken from official model releases. Those comparisons do not establish a common inference-compute or tool-access budget across every model.

4.3 Implementation Details

Full-parameter fine-tuning, updating all model parameters, uses learning rate 7e-6, global batch size 128, a cosine schedule with 3% warm-up, maximum context length 16,384 tokens, and 3 epochs, without early-stop checkpoint selection. Training uses DeepSpeed ZeRO-3, a distributed memory optimization method, on 32 NVIDIA A100 GPUs.

• Evaluation uses greedy decoding, maximum sequence length 32,768, at most 6 tool uses, and the QwQ-32B-Preview chat template.
• The reported metric is pass@1, success with a single generated answer, using evaluation procedures from Yang et al. (2024).

4.4 Main Results

Table 1 reports START scores of 63.6% on GPQA, 94.4% on MATH500, 95.0% on AMC23, 66.7% on AIME24, 47.1% on AIME25, and 47.3% on LiveCodeBench. Improvements over QwQ-32B-Preview are respectively 5.5, 3.8, 15.0, 16.7, 7.1, and 5.9 percentage points.

• START matches Search-o1-32B on aggregate GPQA. Table 2 shows physics improving from 73.8% to 80.0%, chemistry from 41.9% to 47.3%, and biology remaining at 68.4%; the authors attribute this pattern to the usefulness of computation, but do not directly test that explanation.
• Against R1-Distill-Qwen-32B, START has higher reported GPQA, MATH500, AMC23, and AIME25 scores, but lower AIME24 and LiveCodeBench scores. Its advantage therefore depends on the task.
• Table 3 reports unchanged easy programming accuracy at 92.3%, medium accuracy increasing from 46.0% to 84.6%, and hard accuracy increasing from 10.2% to 12.2%. The authors suggest that easy questions need little debugging and hard questions remain difficult within the available debugging budget.

START improves on its base model across all six reported tasks, with its largest absolute gain on AIME24. The comparison with R1-Distill-Qwen-32B is mixed, and several competing-model results come from official releases.

START’s GPQA gains occur in physics and chemistry, while biology remains unchanged from the base model. Search-o1 performs better in biology, so their aggregate GPQA tie conceals different subject-level results.

The reported improvement is concentrated in medium programming problems: 46.0% becomes 84.6%. Easy performance is unchanged and hard performance improves only slightly, limiting claims about debugging benefits across difficulty levels.

4.5 Analysis

In “4.5.1 Long CoT vs Long TIR,” the authors generate ordinary reasoning trajectories for the same questions using 32 samples per question, temperature 0.7, and top-p 0.95, then filter incorrect and repetitive responses and retain at most one per question. The resulting QwQ-RFT reaches 53.3% on AIME24 and 42.1% on LiveCodeBench, versus START’s 66.7% and 47.3%, while its AIME25 score falls to 33.3% from the base model’s 40.0%.

• Table 4 supports a benefit from the tool-assisted training pipeline beyond merely reusing the question set. Different trajectory-generation procedures and sampling settings prevent treating it as a fully isolated intervention on tool access alone.
• In “4.5.2 Analysis of Hint-infer,” repeated hints before the stop token extend inference. Figure 5 and Table 6 show QwQ-32B-Preview improving from 50.0% to 60.0% on AIME24 and from 80.0% to 92.5% on AMC23 over three additional rounds.
• START responds unevenly: AIME25 rises from 47.1% to 60.0% and LiveCodeBench from 47.3% to 50.0%, but GPQA falls from 63.6% to 61.6% by Round 3. Table 6 also lists base-model GPQA at 58.5% in Round 0, whereas Tables 1 and 4 report 58.1%.

Ordinary rejection sampling fine-tuning on the same question set yields smaller gains than START and worsens AIME25 performance. This supports the tool-assisted pipeline, although the two pipelines also differ in trajectory generation and sampling settings.

The plots relate accuracy to average generated thinking tokens as hints extend inference. The base model benefits across the plotted mathematics tasks, while START’s changes vary by benchmark and round; the horizontal axis does not measure interpreter latency or total execution cost.

The listed source quantities total 46,969, which differs from the printed total of 49,969 and the main text’s approximate 50K description. This discrepancy limits exact reconstruction of the training corpus from the paper.

Additional hints improve several mathematics and programming scores but reduce START’s GPQA accuracy across all three added rounds. The table’s QwQ-32B-Preview GPQA Round 0 value also differs from Tables 1 and 4, so those starting values must remain distinguished.

5 Conclusion

The conclusion presents Hint-infer and Hint-RFT as a route for adding interpreter use to extended reasoning. The experiments support improved benchmark accuracy over QwQ-32B-Preview and illustrate computation, checking, and debugging behaviors, with comparative performance varying across tasks and models.

6 Limitations

The stated limitations are reliance on Python as the sole external tool, possible disruption from manually chosen hints and insertion locations, and evaluation on a limited benchmark collection. The authors also discuss possible misuse of code generation and problem-solving capabilities, without reporting an empirical safety evaluation.

• The paper does not report repeated-run uncertainty estimates or a systematic latency and execution-cost comparison. Its accuracy results do not establish the efficiency of the complete model-and-interpreter system.

Appendix

• A.1 Training set of START: Table 5 lists 890 AIME problems before 2024, 7,500 MATH problems, 28,505 Numina-MATH problems, 7,505 Codeforces problems, 2,011 code contests problems, and 558 LiveCodeBench problems before July 2024. These entries sum to 46,969; the printed total is 49,969, and the main text uses approximately 50K.
• A.2 More results about Hint-infer: Table 6 records rounds 0–3 for both models. QwQ-32B-Preview reaches 92.4% on MATH500 and 42.0% on LiveCodeBench by Round 3; START peaks at 95.6% on MATH500 in Round 2 before falling to 95.2%, showing that extra inference does not consistently improve results.
• A.3 Prompting Methods for Data annotation compares direct, detailed, and demonstration-based prompts. The authors report that QwQ-32B-Preview, DeepSeek-R1, and o1-mini did not produce the desired interpreter-call reasoning with these methods, whereas Hint-infer triggered Python code generation with 100% success; this concerns eliciting code, not solution accuracy, and the appendix does not specify the denominator. For the o1 series, the authors could inspect only whether the reasoning summary included Python invocation.
• A.4 Hint-infer for test time scaling supplies three successive mathematics and GPQA hints asking the model to check, recheck, and deepen its reasoning with Python. Programming hints use debugging templates with or without starter code, embedding example inputs and printing their outputs for comparison.
• A.5 Case Study For START includes “A.5.1 A case form AMC23,” where enumeration of 6, 10, and 15 cent coin combinations returns the largest missing value 29 and digit sum 11; “A.5.2 A case form AIME24,” where the trajectory revises code after a reported SyntaxError and obtains (17, 110); and “A.5.3 A case form AIME25,” where symbolic computation and a cross-product calculation give area 507√3 and answer 510. These are selected illustrative trajectories, rather than estimates of how frequently each behavior occurs.
• “A.5.4 A case form MATH500” uses symbolic computation, algebra performed by software, for a functional equation and returns −2, 1; “A.5.5 A case form GPQA” revisits numerical calculations and selects B after obtaining factor 4.514. “A.5.6 A case form LiveCodeBench” revises a counting implementation after outputs 2 and 6 disagree with expected values 4 and 126, then reports matching sample outputs; matching those samples alone does not prove correctness for all valid inputs.

Brief Thoughts

The strongest contribution is a concrete data-generation procedure for teaching an existing reasoning model to use executable feedback. The same-question QwQ-RFT comparison and the debugging examples support that interpretation, although the experiment combines changes in tool use, trajectory construction, and data selection.

The results support testing this approach on tasks where computation or executable checks can expose mistakes. Python-only evaluation, mixed gains from additional hints, unreported uncertainty and runtime costs, and inconsistent dataset counts leave reliability and deployment costs incompletely assessed.