rStar2-Agent: Agentic Reasoning Technical Report | Summary

This article explains the key points of rStar2-Agent: Agentic Reasoning Technical Report.

• 2025-08-28 (arXiv)
• Shang, Ning, Liu, Yifei, Zhu, Yi, Zhang, Li Lyna, Xu, Weijiang, Guan, Xinyu, Zhang, Buze, Dong, Bingcheng, Zhou, Xudong, Zhang, Bowen, et al.
• Microsoft Research
• Paper
• Github

Read this article in Korean

Summary

• rStar2-Agent addresses how to train a language model to solve difficult mathematical problems by using Python execution feedback to verify and revise intermediate reasoning. In the report’s benchmark comparison, its 14-billion-parameter model achieves 80.6% average pass@1, single-response correctness averaged over 16 samples per question, on AIME24, a competitive mathematics benchmark, versus 79.8% for DeepSeek-R1 (671B); on AIME25, it scores 69.8% versus 70.0%.
• The central method is Group Relative Policy Optimization with Resampling on Correct (GRPO-RoC), which learns from relative rewards within a response group and resamples complete reasoning and tool-interaction trajectories before updating the model. Successful trajectories with fewer tool errors and formatting violations receive higher selection probability, while unsuccessful trajectories are sampled without quality filtering. This agentic reinforcement learning trains the model through tool interaction while keeping rewards based solely on final-answer correctness.
• Training begins with non-reasoning supervised fine-tuning (SFT), learning instruction following and tool formatting from demonstrations, followed by three stages of reinforcement learning (RL), learning from correctness rewards. The reported run uses 42K curated mathematics problems, maximum response lengths of 8K→12K→12K tokens, the units of text processed by the model, and 510 RL steps on 64 AMD MI300X GPUs within one week.
• The experiments show competitive mathematics accuracy and shorter responses than several reported baselines, plus a science-reasoning gain from 42.1% to 60.9% on GPQA-Diamond, a scientific question-answering benchmark. General tool-use and alignment scores, measuring instruction-following and chat behavior, decline slightly from the SFT checkpoint. Continued RL beyond the selected 510-step checkpoint causes policy and reward collapse in the reported 14B experiments.

1 Introduction

The introduction argues that extending Chain-of-Thought (CoT), a generated sequence of intermediate reasoning steps, can leave subtle mistakes undetected when verification depends entirely on the model’s own reflection. Agentic RL lets the model interact with a Python interpreter, check intermediate claims against execution results, and revise its approach. The practical challenges are tool failures, final-answer rewards that still reward successful trajectories containing intermediate errors, and expensive, uneven multi-turn generation.

• The proposed approach combines trajectory resampling, an isolated execution service, and staged training.
• Figure 1 compares accuracy against training steps across different model sizes and training recipes; its curves do not establish an equal-compute comparison.

The overview combines benchmark scores with an AIME24 training curve. rStar2-Agent-14B reaches 80.6 in 510 RL steps, while the plotted DeepSeek-R1-Zero (671B) curve ends at 71.0 after substantially more steps. Different model sizes and recipes prevent this from serving as a controlled compute comparison.

2 Agentic Reinforcement Learning Methodology

The methodology treats a rollout, a complete generated reasoning and tool-interaction trajectory, as alternating model turns and execution feedback. The model can reason, invoke Python, inspect the returned result, and continue until it provides a final answer. Figure 2 illustrates computation followed by a separate verification call before answer submission.

Execution feedback becomes part of the next model turn. The worked example obtains (17, 110), checks divisibility in a separate Python call, and returns 110, illustrating verification of an intermediate computation before submission.

2.1 Smarter Reasoning in a Code Environment

The Python environment includes Numpy for numerical computation, Scipy for scientific analysis, and SymPy for symbolic mathematics. Each assistant turn ends at an end-of-sequence token; if a tool call is present, the service executes it and appends feedback under the user role before generation resumes. A turn without a code call terminates the rollout, and interaction also stops at the configured maximum number of turns.

• Calls use JSON, a structured data format, inside <tool_call> tags, with the function execute_python_code_with_standard_io and arguments containing code and input. Feedback appears inside <tool_response> tags.
• The service returns standard output, an IPython result from the interactive Python environment when standard output is absent, an execution error with traceback, or a timeout. Invalid call formatting also produces feedback.
• The prompt specifies <reason> blocks and a single <answer> block, with the final numeric result inside \boxed{} for extraction. Figure 3 documents this interface.

The experimental prompt defines the Python function interface and tags for reasoning, tool calls, and final answers. These conventions allow the training system to parse requests and extract numerical answers.

2.2 End-to-End Agentic Reinforcement Learning

2.2.1 Preliminary: GRPO introduces Group Relative Policy Optimization (GRPO), which updates the policy, the model’s response-generating distribution, using rewards normalized within a group of responses to the same question. Each trajectory receives reward 1 when its extracted final answer matches the reference under math_verify, a rule-based answer checker, and 0 otherwise. The implementation removes the Kullback–Leibler (KL) divergence penalty, which constrains departure from a reference policy, removes the entropy loss, an incentive to increase uncertainty in next-token predictions, and uses Clip-Higher, asymmetric clipping of the new-to-old policy probability ratio, with lower and upper thresholds of 0.2 and 0.28.

• The group-relative advantage is A_i,t = (r_i − mean({r_1, …, r_G}))/std({r_1, …, r_G}). It measures a trajectory’s reward relative to other sampled trajectories for that question.
• These modifications are intended to permit exploration of tool-assisted reasoning while avoiding instability from an explicit entropy incentive.

2.2.2 Challenges in Agentic Reinforcement Learning identifies a mismatch between final-answer correctness and intermediate execution quality. A trajectory can receive positive reward despite faulty tool calls, reinforcing behavior that spends tokens repairing errors. Figure 4 shows that under ordinary GRPO the tool-error proportion within successful trajectories plateaus at roughly 10% for Qwen3-14B and 15% for Qwen2.5-32B, whereas GRPO-RoC continues reducing it.

Both panels measure tool-call errors within correctly answered trajectories. Ordinary GRPO retains a substantial error rate after initial improvement, whereas GRPO-RoC continues reducing errors. This shows why final-answer correctness alone does not ensure clean intermediate execution.

2.2.3 GRPO-RoC: Group Relative Policy Optimization with Resampling on Correct retains the binary correctness reward but changes which trajectories enter the update. For each question, it generates 2G trajectories and selects G: unsuccessful trajectories are downsampled according to their original distribution, while successful trajectories are selected preferentially for low tool-error and formatting scores. Advantages are recomputed using the selected group, and the objective averages over its generated tokens.

• The tool score p_err is 0.5 for a trajectory without tool calls; otherwise it is the number of erroneous calls divided by all calls. This default favors successful, error-free tool use over avoiding tools.
• The formatting score p_format is 1 when no <answer> tag exists; otherwise it is min(1, (number of <answer> tags − 1)/number of turns). Multiple answer blocks therefore reduce selection preference.
• The combined score is p_total = p_err + p_format. The paper describes positive selection probability as inversely proportional to this score, but the displayed rule does not specify how zero scores are handled.

3 Large-Scale Agentic RL Infrastructure

The infrastructure builds on VERL v0.2, the report’s RL training framework, and SGLang, its inference engine. It separates model generation from CPU execution because local execution of many generated programs can overwhelm resources or stall training. A dedicated execution service addresses tool concurrency, while a dynamic rollout scheduler addresses uneven generation lengths and repeated synchronization.

The architecture separates GPU inference engines from CPU execution workers. Queues and schedulers route asynchronous calls through the environment service, keeping execution outside the main generation process while using the cluster's CPU resources.

3.1 Reliable High-Throughput Code Environment

The execution service uses CPU cores in the 64-GPU cluster, with a centralized queue and 32 send workers on the master node. Send workers group up to 64 calls into a batch, dispatching when full or after a timeout; each remaining worker node has a task scheduler and 1024 execution workers. Figure 6 reports up to 45K tool calls per training step with approximately 0.3 seconds average end-to-end latency per call, including scheduling and execution.

• Answer verification is also offloaded to this service because complex extracted answers can make rule-based verification slow enough to block generation.
• The paragraph on page 10 contains the inconsistent phrase ‘45 calls per step’; Figure 6 and the surrounding text support the stated 45K scale.
• The report provides an architecture and latency trace, but no numerical scheduler speedup or equal-hardware infrastructure ablation.

Tool-call counts reach approximately 45K per step while average dispatch-to-response latency remains around 0.3 seconds. The trace supports execution-service scalability over the displayed interval but does not measure speedup against a baseline service.

3.2 Load-Balanced Rollout Scheduler

Static assignment gives each GPU the same number of trajectories even though their turn counts and lengths differ, producing idle time while shorter requests wait for longer ones. It can also overflow the key–value (KV) cache, stored attention states used during generation, causing SGLang to evict and later recompute unfinished requests. The proposed scheduler estimates safe concurrency from available cache capacity and maximum rollout length, dispatches tool calls asynchronously, and assigns new requests when capacity becomes available.

• Figure 7 illustrates the mechanism through timelines; it does not quantify the scheduler’s throughput benefit.

The upper timeline shows idle periods from fixed request assignment and synchronized tool calls. The lower timeline illustrates asynchronous execution and reassignment as cache capacity becomes available. The diagram explains the scheduling mechanism without supplying a numerical throughput comparison.

4 Training Recipe

The main recipe starts from Qwen3-14B-Base, performs non-reasoning SFT, and then applies GRPO-RoC in three stages. Short initial responses allow an 8K training limit before later stages increase the limit and focus on harder problems. Average response length grows across these stages; the concision evidence concerns comparisons with alternative training methods and evaluated models.

4.1 Non-Reasoning Cold Start for Instruction Following

The cold-start dataset contains 165K function-call examples: 117K from ToolACE-11K, APIGen-MT-5K, and Glaive-function-calling-v2-101k, plus 48K Magicoder examples converted to JSON function calls. It also contains 30K Tulu3 instruction-following examples with responses rewritten using o4-mini and 27K chat examples from the LLaMA-Nemontron post-training dataset with conversation prompts rewritten using o4-mini. This stage teaches the interaction interface without a reasoning-focused demonstration dataset.

• Table 1 reports 63.1 on BFCL v3, the Berkeley Function-Calling Leaderboard tool-use benchmark; 83.7 on IFEval, an instruction-following benchmark; and 86.8 on Arena-Hard, the report’s chat and alignment benchmark.
• Mathematics remains weak after this stage: MATH-500, a mathematics benchmark, is 57.4 versus 62.0 for the base model, while AIME24 and AIME25 are 3.33 and 0. These results support preparation for tool interaction rather than an initial mathematics gain.

The cold-start checkpoint has reported tool-use, instruction-following, and chat scores close to or above official Qwen3-14B, while its mathematics scores remain much lower. Its MATH-500 score of 57.4 is also below the base model's 62.0, supporting the interpretation that this stage prepares the interaction interface.

4.2 RL Data Curation

RL data curation restricts training to challenging problems with correctly labeled integer answers because symbolic equivalence checks can give incorrect rewards. The candidate pool contains 17K integer-answer problems from DAPO, an open-source RL training system, 93K problems from Art of Problem Solving forums through OpenMathReasoning, and 937 Project Euler problems requiring mathematics and programming. Cleaning produces 42K problem–answer pairs.

• For OpenMathReasoning, Qwen3-32B generates 16 responses per problem; retention requires integer answers matching the original label at least twice. Unverifiable answers, complex answer formats, and incorrect answers are removed.
• Project Euler problems with excessively large answers, such as 6.5e27330467, are removed to avoid verifier timeouts. Integer-only selection narrows training to readily verified numerical answers.

4.3 Multi-Stage RL Training

RL uses a learning rate of 1e-6 and batches of 512 prompts, generating 32 trajectories per prompt and selecting 16 through Resample on Correct (RoC). Stage 1 trains for 300 steps on 42K problems with an 8K response limit; Stage 2 raises the limit to 12K and ends at cumulative step 385. Stage 3 retains the 12K limit and trains for 125 further steps on 17.3K harder problems, reaching 510 steps in total.

• Stage 1’s average training response length grows from about 1K to 4K tokens. When rewards and evaluation scores plateau and truncation stabilizes around 10%, Stage 2 increases the limit and average length rises to about 6K.
• For Stage 3, the latest policy generates eight trajectories for each original problem, and problems with all eight correct are removed. Optimizer states are reset and the reference model is updated to the latest policy; average training responses then rise to about 8K tokens.
• Table 2 compares reasoning SFT, training steps, and length limits across reported recipes. These quantities do not normalize total compute across different sampling and training configurations.

The comparison places rStar2-Agent's three stages, 510 steps, and 8K→12K→12K limits alongside reported alternatives with longer schedules or length limits. These recipe differences do not by themselves establish a normalized compute advantage.

4.3.1 Unsuccessful Attempts And Lessons reports that overlong filtering, removing truncated trajectories from learning, increased the proportion of overlong responses in these experiments. The authors instead retain truncated trajectories as unsuccessful training signals. Downweighting successful trajectories using n-gram repetition detection, checking repeated sequences of n tokens, also harmed response length and reasoning scores; the authors suggest that similar verification calls were sometimes mistaken for undesirable repetition.

• The report presents these failures as experiences from its own runs, without claiming that the alternative methods are universally ineffective.

5 Experiments

The experiments evaluate mathematics performance, transfer to science reasoning, general tool use and alignment, and the contribution of RoC. The main model is Qwen3-14B-Base with the proposed cold start; a second experiment uses Qwen2.5-32B-Instruct to compare with prior work at the 32B scale. Resource constraints limit that second model to the first two RL stages.

5.1 Setup

SFT runs for three epochs with learning rate 5e-6, batch size 128, 4% warm-up, and cosine decay; Qwen2.5-32B-Instruct receives no additional SFT. RL uses AdamW, an optimizer with decoupled weight decay, with constant learning rate 1e-6, linear warm-up over 20 rollout steps, temperature 1.0 controlling sampling randomness, and maximum interaction counts T=10 for the first two stages and T=15 for the final stage. All experiments use 64 AMD MI300X GPUs.

• Training problems sharing 8-grams, sequences of eight tokens, with competitive mathematics benchmarks are removed as the reported decontamination procedure.
• For mathematics and GPQA-Diamond, evaluation permits up to 30K tokens and 30 turns at temperature 0.6. Each question is sampled 16 times to estimate average pass@1; these scores are not best-of-16 or majority-vote accuracy.
• BFCL v3, IFEval, and Arena-Hard use their default prompts at temperature 0. Table 3 does not establish identical inference budgets or training compute for every listed model.

rStar2-Agent-14B leads the listed models on AIME24 at 80.6 and exceeds DeepSeek-R1 on MATH-500 and HMMT Feb.25. Its AIME25 score of 69.8 is below DeepSeek-R1's 70.0 and o3-mini's 77.0, making the comparison benchmark-specific.

5.2 rStar2-Agent-14B Main Results

rStar2-Agent-14B scores 97.8 on MATH-500, 80.6 on AIME24, 69.8 on AIME25, and 52.7 on HMMT25, a competitive mathematics benchmark. DeepSeek-R1 (671B) scores 97.3, 79.8, 70.0, and 44.4 respectively, so rStar2-Agent scores higher on three of these four benchmarks. OpenAI o3-mini (medium) remains higher on AIME25 and HMMT25 at 77.0 and 53.0.

• Reported stage-end AIME24/AIME25 scores progress from 72.1/64.2 after Stage 1 to 77.0/64.8 after Stage 2 and 80.6/69.8 after Stage 3. Figure 8 also shows increasing training response length.
• The narrative reports a 3.6-point AIME24 advantage over Claude-Opus-4.0 (Think), but Table 3 gives 80.6 versus 76.0, a 4.6-point difference.
• The results demonstrate the full recipe’s performance without reasoning-focused SFT; they do not separately identify the contributions of cold start, data curation, and stage transitions.

The panels show improving AIME24 and AIME25 scores across stage transitions alongside rising average training response length. Length increases from roughly 1K initially to roughly 8K late in training, so the recipe's relative concision does not imply continuously shrinking responses.

Table 4 reports average responses of 9339.7 tokens on AIME24 and 10943.4 on AIME25, versus 14246.8 and 17132.9 for DeepSeek-R1-Zero (671B). Table 5 shows GPQA-Diamond improving from 42.1 after SFT to 60.9 after mathematics RL, compared with 59.1 for DeepSeek-V3. General tool-use and alignment scores decline slightly from their SFT values: BFCL v3 changes from 63.1 to 60.8, IFEval from 83.7 to 83.4, and Arena-Hard from 86.8 to 86.6.

• QWQ-32B and Qwen3-14B also have longer reported responses on both AIME benchmarks. Qwen3-14B scores 70.4 on AIME25 versus rStar2-Agent’s 69.8, so shorter responses do not accompany higher accuracy in every comparison.
• Table 4 contains DeepSeek-R1-Zero rather than DeepSeek-R1. It therefore does not directly substantiate the abstract’s shorter-response comparison against DeepSeek-R1.
• The transfer evidence is strongest for science reasoning. The other scores support approximate retention of cold-start capabilities, with small measured declines.

rStar2-Agent-14B has the shortest listed average responses: 9339.7 tokens on AIME24 and 10943.4 on AIME25. The DeepSeek comparison is specifically DeepSeek-R1-Zero, with 14246.8 and 17132.9 tokens; the table does not measure response length against DeepSeek-R1.

5.3 Ablation Study and Discussions

On Qwen2.5-32B-Instruct, rStar2-Agent completes 700 steps across two stages and scores 94.8 on MATH-500, 69.4 on AIME24, and 57.3 on AIME25. ReTool-32B, a tool-using RL comparison method, reports 93.4, 67.0, and 49.3 after 400 steps, while DAPO-Qwen-32B reports 90.3, 50.0, and 32.1 after more than 5000 steps. Table 6 also reports higher scores than the listed CoT-only Qwen3-14B baselines.

• The comparisons combine different training recipes and budgets. The setup states that the 32B instruct model receives no additional SFT, despite the nearby narrative describing its approach as relying on non-reasoning SFT.
• The proposed benefit from adding Stage 3 to the 32B run is a forecast, not an experimental result.

At 32B scale, rStar2-Agent reports 69.4 on AIME24 and 57.3 on AIME25 after 700 steps, versus ReTool's 67.0 and 49.3 after 400 steps. The 14B rows also favor rStar2-Agent over the listed DAPO variants, although recipes and budgets differ.

The RoC ablation, a comparison that removes the proposed component, uses GRPO with Tool as its baseline: it generates 16 trajectories per question and uses all of them for updates over 300 steps with the other hyperparameters unchanged. Figure 9 shows higher AIME24 and AIME25 accuracy and shorter training responses with RoC. After the main 14B run peaks at step 510, additional training causes policy and reward collapse; increased temperature, longer response limits, more tool interactions, higher clipping, and optimizer resets do not resolve it.

• GRPO-RoC generates 32 candidates before selecting 16, whereas the baseline generates 16. This comparison tests oversampling and selection together, without isolating selection at equal generated-trajectory count.
• The DAPO reference in Figure 9 comes from previously reported results after 2000 steps, rather than a contemporaneous 300-step control.
• The authors hypothesize that collapse reflects a capacity limit inherited from pretraining. The reported failure does not establish that explanation or a general ceiling on RL.

GRPO-RoC achieves higher AIME accuracy and shorter training responses than GRPO with Tool over the displayed 300-step comparison. RoC generates 32 candidates and selects 16, while the baseline generates 16; the curves therefore test oversampling together with selection.

5.4 Analysis of Agentic Reasoning Behaviors

The behavioral analysis randomly samples 64 trajectories and highlights the top 20% of tokens by entropy. The authors identify forking tokens, phrases that initiate a change or verification in reasoning, and reflection tokens, sequences that interpret execution feedback. Figures 10 and 11 illustrate verification of a successful computation and recovery from a SymPy execution error.

• In Figure 10, the model checks whether its grid-coloring program implements the problem’s conditions and ultimately confirms 24. In Figure 11, it replaces a failing symbolic check with a remainder calculation and verifies the final result 117.
• The report observes comparatively low entropy in Python code and comments, attributing this tentatively to code pretraining. Extension of this pattern to non-coding tools remains open.
• The highlighted traces describe feedback-responsive reasoning; they do not isolate whether the highlighted tokens cause the accuracy gains.

The model reviews whether its grid-coloring program captures adjacency and corner conditions before confirming 24. Highlighted high-entropy tokens occur around checking and interpretation of feedback, illustrating verification behavior without establishing its causal contribution to accuracy.

After a SymPy GeneratorsNeeded error, the model changes its symbolic remainder check, verifies the candidate values, and returns 117. High-entropy spans appear around diagnosis and revision, illustrating recovery from execution feedback.

6 Conclusion

The conclusion attributes the results to GRPO-RoC, scalable execution infrastructure, and staged training. The supported outcomes are competitive mathematics performance from a 14B model trained with Python interaction and final-answer rewards, shorter reported responses than the models in Table 4, and transfer to GPQA-Diamond. The report states that code, datasets, and recipes are publicly released at https://github.com/microsoft/rStar and proposes broader reasoning domains and environments as future work.

Appendix

• The supplied 22-page report contains no appendix. References begin on page 19 and continue through page 22; no appendix experiments or implementation specifications are provided.

Brief Thoughts

The method separates reward correctness from trajectory selection: final answers determine reward, while execution quality influences which successful examples train the policy. Figure 4’s decreasing tool-error rates and Figure 9’s accuracy and response-length curves support the combined resampling strategy. Its demonstrated setting uses executable Python checks and integer answers that can be verified reliably. Fewer RL steps and shorter responses are reported, but oversampling adds generation work. The report provides neither an equal-compute end-to-end comparison nor a measured scheduler speedup. Integer-only training, a Python-centered environment, and incomplete 32B training limit the demonstrated scope. Science transfer is supported; improvement in general tool use and the explanation for post-510-step collapse remain unestablished.