Parallel-R1: Towards Parallel Thinking via Reinforcement Learning | Summary

Contents

Summary
1 Introduction
2 Related Work
- 2.1 Parallel Thinking
- 2.2 Improving Reasoning via RLVR
3 Learning Parallel Thinking via Reinforcement Learning
4 Experiments
5 Conclusion
Appendix
Brief Thoughts

This article explains the key points of Parallel-R1: Towards Parallel Thinking via Reinforcement Learning.

2025-09-09 (arXiv)
Zheng, Tong, Zhang, Hongming, Yu, Wenhao, Wang, Xiaoyang, Dai, Runpeng, Liu, Rui, Bao, Huiwen, Huang, Chengsong, Huang, Heng, Yu, Dong.
Tencent AI Lab Seattle, University of Maryland, College Park, University of North Carolina at Chapel Hill, City University of Hong Kong, Washington University in St. Louis
Paper
Github

Summary

Parallel-R1 addresses how to train large language models (LLMs), models that generate language, to explore multiple reasoning paths concurrently and combine their findings. Using Qwen-3-4B-Base, its Parallel-R1-Seen variant achieves 48.9% accuracy averaged across four mathematical benchmarks versus 45.1% for sequential reinforcement learning (RL), learning through reward feedback, directly on the harder training dataset: a gain of 3.8 percentage points, or approximately 8.4% relative.
The curriculum begins with supervised fine-tuning (SFT), training on example responses, to teach the parallel reasoning format on easier mathematics problems, then uses RL on harder problems. The paper compares an unchanged causal model, whose tokens can attend to preceding tokens, with a structured model that blocks attention between reasoning paths; the causal variant obtains the highest average accuracy.
During training, parallel blocks move later in the response, which the authors interpret as a shift from exploring solutions to checking a candidate answer from multiple perspectives. A separate experiment encourages parallel reasoning for 200 steps before switching to accuracy alone and reaches a peak of 25.6% on AIME25, the paper’s 2025 mathematical reasoning benchmark; the abstract reports a 42.9% relative improvement over its baseline, a separate comparison from the 48.9% versus 45.1% result. Evaluation is limited to one model scale and mathematical benchmarks.

1 Introduction

The introduction identifies three obstacles: models initially lack the parallel reasoning format, valid demonstrations are difficult to generate for hard problems, and rewards must balance answer correctness with parallel usage, the proportion of responses containing explicit parallel structure. Parallel-R1 addresses these through an easy-to-hard curriculum, different training recipes for the two attention architectures, and analysis of branching behavior during training. The stated release location is https://github.com/zhengkid/Parallel-R1.

Adaptive parallel reasoning workflow and progressive training curriculum.

The related work connects methods for exploring multiple reasoning paths with RL based on automatically checkable outcomes. The paper positions its contribution as learning when to branch and aggregate on general mathematical tasks after initial format training.

2.1 Parallel Thinking

Earlier approaches generate independent responses from the beginning, exchange intermediate reasoning at fixed intervals, or select branches through externally specified search rules. The authors argue that these approaches constrain when branching and aggregation occur. They distinguish Parallel-R1 from SFT methods that reorganize existing reasoning traces and from earlier RL demonstrations on Countdown, a synthetic task used in prior parallel reasoning work.

2.2 Improving Reasoning via RLVR

Reinforcement Learning with Verifiable Rewards (RLVR) trains models using outcome rewards that can be checked automatically, avoiding a separately trained reward model or human annotations for individual reasoning steps. The cited literature applies this approach across mathematics and other tasks, but the authors identify predominantly sequential reasoning as a remaining limitation. Parallel-R1 focuses on learning parallel behavior; it does not evaluate solutions to the broader issues of faithfulness, whether stated reasoning reflects the model’s decision process, or robustness, reliability under challenging inputs, discussed here.

3 Learning Parallel Thinking via Reinforcement Learning

The method separates learning the output format from learning adaptive parallel behavior and solving harder problems. It defines a branching-and-summary workflow, constructs initial demonstrations on easy problems, and develops different RL recipes for causal and structured attention variants.

3.1 Overview

The central design choice is to generate cold-start demonstrations, initial examples that teach the unfamiliar parallel format, on simple tasks instead of synthesizing parallel solutions for the final challenging tasks. RL subsequently develops the behavior on harder mathematics problems. The two model settings differ in whether attention masks, rules controlling which tokens can access other tokens, and position IDs, indices used to represent token positions, are modified to isolate reasoning paths.

3.2 Formulation of Parallel Thinking Behaviors

Parallel thinking alternates between Exploration, which temporarily suspends the main reasoning chain and generates N independent trajectories, or reasoning paths, and Summary, which aggregates their findings before resuming the chain. The control tags are <Parallel>…</Parallel>, <Path>…</Path>, and <Summary>…</Summary>. At inference, predicting <Parallel> pauses autoregressive generation, the usual process of generating tokens successively, and triggers concurrent path generation, followed by aggregation and continued generation; this cycle can repeat.

3.3 The Simple and Scalable Data Pipeline for Parallel Thinking

DeepSeek-R1-0528-Qwen-3-8B generates parallel reasoning demonstrations through detailed zero-shot prompts, instructions supplied without worked examples. Under identical prompts and sampling settings, Table 1 reports valid parallel formatting for 83.7% of 7,472 GSM8K examples, an easier mathematics dataset, versus 0.0% of 17,916 DAPO examples, the harder mathematics training dataset used here. The Key Finding 1 box instead gives 83.6%, an internal discrepancy.

The resulting Parallel-GSM8K dataset teaches the initial format rather than directly supplying demonstrations for the final target tasks.
Algorithm 1, Parallel Thinking Format Check, filters traces by checking matched and properly nested opening and closing tags; this check does not establish mathematical correctness or semantic independence between paths.

Parallel formatting rates for demonstrations generated on GSM8K and DAPO.

3.4 Eliciting Parallel Thinking via Reinforcement Learning in Causal Models

Parallel-R1-Seen retains the original architecture and uses Group Relative Policy Optimization (GRPO), an RL algorithm that compares each sampled response’s reward with rewards in the same group. Its rollout, a response generated for training, alternates between the main reasoning chain, parallel paths, and their summary.

3.4.1 Reinforcement Learning Algorithms: GRPO computes each response’s advantage, its normalized reward relative to the group’s mean and standard deviation. The objective clips the ratio of response probabilities under the updated and previous policies to limit update size and penalizes divergence from a reference policy.
3.4.2 The Training Recipe and Reward Modeling: Cold-start SFT uses Parallel-GSM8K demonstrations generated by DeepSeek-R1-0528-Qwen-3-8B, retaining only its non-thinking output, described as the final short chain-of-thought, or explicit reasoning trace.
RL on Easy Math uses the same GSM8K questions and specifies R_final = R_⟨Parallel⟩ × R_acc. The stated operational rule gives +1 only when the response contains at least one parallel thinking unit and its final answer is correct; all other responses receive −1. RL on General Math then uses DAPO with answer accuracy as the sole reward.

3.5 Eliciting Parallel Thinking via Reinforcement Learning in Structure Models

Parallel-R1-Unseen changes attention to isolate paths explicitly, addressing the potential for representations from one path to affect another in the Seen model. Its training recipe omits the easy-math RL stage because applying the causal variant’s recipe to the structured variant performs poorly.

3.5.1 Structured Attention Mechanism: Path-window masking restricts each token within a <Path> block to its own path and shared context. The text describes multiverse position encodings as assigning disjoint position indices to paths, although Figure 2 visibly repeats indices across paths. The shared <Summary> block retains access to the paths for aggregation.
3.5.2 The Training Recipe and Reward Modeling: S1 optimizes accuracy alone. S2 alternates rewards within windows of W=10 steps, using accuracy-only rewards for 80% of steps and a tiered reward for the remaining 20%.
The S2 tiered reward gives +1.2 for a correct answer with at least one parallel unit, +1.0 for a correct answer without a parallel unit, and −1.0 otherwise.

4 Experiments

The experiments compare format training, sequential RL baselines, and causal and structured Parallel-R1 variants. They evaluate mathematical accuracy, parallel usage, the effects of removing or adding training stages, reward schedules, and changes in branching position.

4.1 Experimental Setups

All experiments use Qwen-3-4B-Base and evaluate AIME’24 and AIME’25, mathematical reasoning benchmarks from the corresponding years, AMC’23, another mathematical reasoning benchmark, and MATH, a mathematical problem-solving dataset. Sampling temperature, the parameter controlling generation randomness, is T=1.0: MATH uses one response per question, while the other benchmarks use 16 independent responses and report mean@16, average response accuracy, and pass@16, success when at least one sampled response is correct.

The code is adapted from VERL, the RL training framework used by the authors, primarily following its official recipe without hyperparameter tuning.
Cold-start SFT uses batch size 128, learning rate 1e-5, weight decay 0.01, warm-up ratio 0.1, and a cosine learning-rate schedule, with 58 and 230 gradient updates for the Seen and Unseen variants, respectively.
Optional GSM8K RL runs for five epochs with batch size 1024, 5 rollouts, learning rate 1e-6, and 35 gradient updates. DAPO RL uses 300 gradient updates, batch size 512, 8 rollouts, and learning rate 1e-6; both RL stages omit warm-up and learning-rate scheduling.

4.2 Main Results

Table 2 compares direct GRPO training on DAPO with a sequential baseline that first receives GSM8K RL and then DAPO RL. Parallel-R1-Seen achieves 19.2%, 19.4%, 70.5%, and 86.7% on AIME25, AIME24, AMC23, and MATH, respectively, averaging 48.9% against 45.1% for direct GRPO and 45.3% for the two-stage sequential baseline.

Parallel-SFT-Seen and Parallel-SFT-Unseen average 36.0% and 31.7%, despite both showing a parallel usage rate of 95.6%; frequent parallel formatting alone does not deliver the strongest accuracy.
Parallel-R1-Unseen S1 averages 47.1% with 13.6% parallel usage, while S2 averages 46.8% with 63.0% usage. S2 has the highest AIME25 pass@16 of 42.2% and AMC23 pass@16 of 91.5% among the listed methods, while Seen has the highest average response accuracy.
The narrative cites a base-model average of 4.6%, but Table 2 gives 6.6%; the table value agrees with the displayed benchmark scores.

Accuracy, pass@16, and parallel usage across base, SFT, sequential RL, and Parallel-R1 configurations.

4.3 Analysis

The ablations, comparisons that remove or add individual components, show that effective training stages and rewards depend on the attention architecture. They also distinguish maintaining parallel formatting from improving answer accuracy.

4.3.1 Ablation on Training Approach: Removing GSM8K RL from Seen lowers average accuracy from 48.9% to 46.6%, a 2.3 percentage-point reduction. Adding it to Unseen S1 lowers the average from 47.1% to 38.5%, an 8.6 percentage-point reduction; poor transfer of the structured mask is the authors’ hypothesis, rather than an independently established mechanism.
4.3.2 Ablation on Parallel Thinking Prompt: Removing the prompt lowers Seen’s average from 48.9% to 47.1%, although AIME25 improves from 19.2% to 20.4%. The prompt’s measured benefit is therefore an average effect across benchmarks.
4.3.3 Ablation Studies on Reward Modeling: How to Effectively Stimulate Parallel Thinking: Accuracy-only rewards produce 13.6% parallel usage, direct parallel rewards 80.3%, and alternating rewards 63.0%. Direct parallel rewards reduce AMC23 accuracy to 59.4% versus 69.7% for accuracy-only training. Alternating rewards improve AIME25 and MATH over accuracy-only training but reduce AIME24 and AMC23.

Training-stage and parallel-prompt ablations.

Reward-schedule trade-offs for Parallel-R1-Unseen.

4.4 Evolution of Parallel Thinking Behavior During RL Training

The relative position of a <Parallel> block is its starting token index divided by the total response length. Figure 3 shows its average position increasing during RL, and the authors interpret this as a shift from early computational exploration to later verification after a candidate solution has been found. Figures 5 and 6 illustrate this interpretation on the same base-number divisibility problem, but these selected examples do not measure how often each behavior occurs.

Relative starting position of parallel blocks during RL training.

4.5 Extra Bonus: Parallel Thinking as a Mid-Training Exploration Strategy for RL Training

The separate scaffold experiment, which uses parallel reasoning as temporary support during training, applies Parallel-R1-Unseen S2-style alternating accuracy and parallel rewards for steps 0–200, then switches to accuracy-only GRPO. Figure 4 shows parallel usage falling toward zero after the switch while AIME25 mean@16 accuracy reaches a peak of 25.6%. This supports the hypothesis that an earlier parallel exploration phase can improve subsequent mostly sequential reasoning; the abstract’s reported 42.9% baseline improvement belongs to this experiment, and the PDF does not specify the exact baseline score or checkpoint used to calculate it.

AIME25 accuracy and parallel usage during exploration followed by accuracy-only training.

5 Conclusion

The conclusion emphasizes separating format acquisition, adaptive behavior, and mathematical reasoning through a progressive curriculum. The experiments show improved average accuracy over the evaluated sequential RL baselines and higher accuracy in the scaffold experiment even after explicit parallel usage largely disappears. The authors propose that temporary parallel exploration helps training discover a more effective policy, the model’s response-generating behavior.

Early-stage parallel exploration on a base-number divisibility problem.

Late-stage parallel verification after deriving candidate bases.

Appendix

A Prompts provides the Baseline Prompt, which requests step-by-step reasoning and a final result after “Final Answer:”, and the Parallel Thinking Prompt. The latter requests at least two distinct independent <Path> blocks per <Parallel> block, no ordering information or cross-references between paths, a following <Summary> block, adaptive repetition, and a final line beginning “Final Answer:”. Algorithm 1, Parallel Thinking Format Check, uses a stack to reject unmatched, incorrectly nested, or unclosed tags.
B Case Studies compares early and late responses to finding the sum of integer bases b > 9 for which 17_b divides 97_b. Both final responses identify b=21 and b=49 and return 70, but the early response branches before completing the solution, while the late response branches after deriving the candidate bases. The early second path contains an erroneous expansion and produces no valid bases; the late first path incorrectly states that digit validity requires b > 7 despite the presence of digit 9, although the given b > 9 condition already guarantees validity.

Brief Thoughts

The training-stage ablation gives a concrete design lesson: the same easy-math RL stage helps Seen and harms Unseen, so the tested variants require different curricula. The scaffold experiment suggests that explicit parallel reasoning can help training even when branching later disappears, but its peak result is not accompanied by repeated training runs or uncertainty estimates. The PDF also provides no inference-latency measurements or compute-matched comparison, leaving efficiency and generalization beyond Qwen-3-4B-Base and mathematics unresolved.