On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models | Summary

This article explains the key points of On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models.

• 2025-12-08 (arXiv)
• Zhang, Charlie, Neubig, Graham, Yue, Xiang.
• Carnegie Mellon University, Language Technologies Institute
• Paper
• Github

Read this article in Korean

Summary

• The paper asks when reinforcement learning (RL), training with rewards, helps language models solve reasoning problems beyond those learned during pre-training. In controlled synthetic arithmetic experiments, RL improves pass@128, success with at least one verified solution among 128 samples, over the base model on harder problems when training targets its edge of competence, tasks it can sometimes solve; already mastered or excessively difficult training tasks produce little improvement on this metric.
• The experiments train 100M-parameter Qwen2.5-style models, models using the decoder-only architecture specified in the paper, from scratch on 10B synthetic tokens and independently vary arithmetic complexity and narrative context. Process-verified evaluation requires every reference reasoning step and the final answer to be correct, making the reported gains stricter than final-answer accuracy alone.
• Contextual generalization, transferring reasoning across different narratives, depends on sufficient pre-training exposure to elementary tasks in the target context. When that exposure is restricted to two-operation examples, 0% and 0.1% shares fail to support complex transfer, while 1% and 10% enable substantial performance after the same RL training.
• Mid-training, additional supervised token prediction between pre-training and RL, changes which tasks benefit from the training budget: more supervision favors nearby tasks, while a smaller supervised stage followed by heavier RL favors harder unseen tasks. The introductory overview reports a +10.8% average pass@128 advantage over RL alone on OOD-hard tasks, problems substantially outside the pre-training operation range; appendix experiments show that the preferred allocation also depends on total compute.
• Process supervision, rewarding correctness of intermediate reasoning steps, improves verified accuracy and reduces structural errors in the tested reward comparisons. On 15–20-operation problems, a reward comprising 0.2 outcome reward and 0.8 process reward increases pass@1, verified success with one sampled solution, from 5.6% to 9.9% and pass@128 from 23.2% to 27.7% relative to outcome-only RL.

The three panels summarize how high-sample gains depend on RL task difficulty, complex contextual transfer depends on elementary pre-training exposure, and a hybrid allocation improves difficult-task performance under the reported fixed budget. The +10.8% comparison is an average over OOD-hard difficulties; it does not imply equal gains at every difficulty.

1 Introduction

The paper attributes conflicting claims about RL’s reasoning benefits partly to opaque pre-training corpora: researchers cannot reliably determine which reasoning primitives, elementary skills composed into longer solutions, a base model already possesses. It controls the data distributions of all three training stages and examines both deeper compositions and transfer across surface contexts.

• The design combines explicit arithmetic operations, observable reasoning traces, and systematic distribution changes to isolate the effects of each training stage.
• The introduction identifies four interacting factors: remaining room for improvement, RL task difficulty, pre-training coverage of relevant primitives, and supervision of intermediate steps.

2 Preliminaries

The preliminaries establish a synthetic generator, two generalization axes, and an evaluation protocol that checks reasoning processes as well as answers. These components separate changes in problem complexity from changes in narrative context while retaining the reference computation for verification.

2.1 Controllable Synthetic Reasoning Dataset

The dataset builds on GSM-Infinite, a framework for generating arithmetic word problems with controlled reasoning complexity. Each instance derives from a directed acyclic graph (DAG), a dependency graph without directed cycles, whose nodes represent quantities and whose edges encode the dependencies needed to compute them.

• The paper defines operation complexity as \(\operatorname{op}(\mathcal{G})=|\mathcal{E}|\), where E is the graph’s edge set; op denotes the operation count used to vary task difficulty.
• Contextual templates such as animals–zoo and teachers–school render graph variables and dependencies into natural language. Numeric assignments, graph structure, and linguistic context are controlled separately.
• Disjoint data splits and reference graphs support contamination control and verification of intermediate steps; exact deduplication details appear in A.1.5.

Separating graph structure from contextual rendering enables changes in complexity and narrative while retaining the reference computation. The evaluation panel shows why final-answer correctness alone is insufficient: incorrect dependencies or arithmetic steps cause a generated solution to fail verification.

2.2 Task Setup

Extrapolative (Depth) Generalization measures correctness on operation chains longer than those encountered during training. Contextual (Breadth) Generalization measures whether reasoning transfers to different narrative templates with equivalent underlying computational structure.

• The depth experiments vary operation ranges across training phases; the context experiments vary exposure to frequent and long-tailed templates, templates underrepresented in the training mixture.
• A.2 clarifies that training graphs are sampled randomly rather than deliberately paired across templates, so equivalent structures need not appear in every training context.

2.3 Evaluation Protocol.

The evaluator parses each generated solution into a predicted dependency graph and final answer. For each reference node, it checks the node’s presence, dependencies, and numeric value; process accuracy is the average of these step-level correctness indicators.

• A solution counts as fully correct only when process accuracy equals 1 and its final answer matches the reference answer. All reported pass@k results, success with at least one verified solution among k samples, use this criterion.
• The protocol rejects solutions with missing or incorrect reference steps even when the final answer is correct. Verification concerns the parsed solution trace and reference graph.

2.4 Training Setup.

The full synthetic corpus contains 30B tokens, with disjoint splits for pre-training, mid-training, and post-training. The default base model is a decoder-only 100M-parameter Qwen2.5-style model trained from scratch on 10B tokens, a 100× token-to-parameter ratio, covering op=2–10 across contextual templates.

• Pre-training establishes elementary arithmetic and compositional skills while leaving deeper tasks incompletely solved; accuracy within its operation range approaches saturation at pass@128.
• Mid-training retains the pre-training objective but narrows the distribution toward tasks on which competence is emerging. It is introduced in Section 5.
• Post-training uses GRPO, the RL optimization algorithm selected for these experiments, on curated difficulty ranges and context mixtures. Supervised Fine-tuning (SFT), training on labeled task examples, is discussed as another post-training strategy but is not the focus of the experiments.

3 When Does Post-Training Incentivize Reasoning Beyond the Base Model?

Section 3 fixes pre-training and varies RL data difficulty. In-Distribution (ID), within the pre-training range, comprises op=2–10; out-of-distribution (OOD), outside that range, is divided into OOD-edge at op=11–14 with nonzero base-model pass@128 and OOD-hard at op=15–20 with near-zero base accuracy.

• The main text describes approximately 200K RL samples drawn separately from op=7–10, op=9–12, op=11–14, or op=17–20. A.9 records 204.8k samples and lists op=8–10 for the easiest recipe, which differs from the main text.
• Observation 1: ID pass@1 improves, but ID pass@128 does not improve across the tested RL recipes. Training on op=11–14 improves both OOD-edge and OOD-hard pass@128; training directly on op=17–20 leaves hard-task performance near zero.

Takeaway 1 ties verified high-sample gains to two conditions: pre-training must leave room for improvement, and RL examples must remain partially solvable. Discussion 1 uses this dependence on task coverage and difficulty to reconcile studies reporting capability refinement with studies reporting extrapolative gains.

• Practical Guidance 1 recommends selecting examples that fail at pass@1 but succeed at larger pass@k, avoiding both redundant easy tasks and tasks with effectively absent rewards.
• The authors propose periodically reassessing this pool as competence improves; this adaptive curriculum is guidance rather than an evaluated procedure in Section 3.

The ID curves converge near the same high pass@128 despite improved pass@1 after RL. Training on op=11–14 produces the strongest displayed gains in both nearby and harder ranges, whereas training directly on op=17–20 leaves hard-task accuracy near zero, supporting the partially solvable-data condition.

4 How Does Pre-training Exposure Shape Post-Training Generalization?

Section 4 pre-trains on 10B tokens containing context A problems at op=2–20 and context B problems restricted to op=2. It varies context B’s pre-training share across 0%, 0.1%, 1%, and 10%, then applies the same approximately 200K-sample RL mixture of 50% context A and 50% context B at op=2–20.

• Observation 2: With 0% or 0.1% context B exposure, transfer fails on complex context B problems. With 1% or 10%, RL supports substantial verified accuracy even at op=20.
• Takeaway 2 interprets elementary target-context exposure as a prerequisite that RL can reinforce. The observed ≥1% condition applies to this synthetic setup; the experiments do not establish a general threshold.

Discussion 2 examines topological similarity, similarity of dependency-graph structure, between correct generated context B solutions and reference context A graphs. Simpler solutions often resemble existing context A patterns, whereas harder solutions show lower similarity when context B exposure is sufficient.

• The authors interpret lower similarity as evidence of more novel compositions. The analysis concerns generated structures within the synthetic arithmetic framework.
• Practical Guidance 2 recommends covering basic long-tail knowledge, rules, and skills during pre-training so that RL has usable primitives to compose.

With the RL mixture fixed at equal shares of contexts A and B, varying elementary B exposure during pre-training sharply changes complex-task performance. The 0% and 0.1% conditions fail at longer solutions, while the 1% and 10% conditions sustain substantial pass@128 through 20 operations.

5 How Does Mid-Training Interact with Post-Training?

Section 5 compares supervised mid-training and RL using approximate floating-point-operation equivalence, an estimate of arithmetic compute expressed in training-token units. Its RL cost is T_RL ≈ (5/3)NrL_total, with N RL samples, rollout multiplicity r=6, meaning six generated solutions per sample, and total sequence length L_total=2048; an RL budget fraction β assigns T_mid=(1−β)T and T_RL=βT.

• The main text specifies Full mid-training on 1B supervised tokens, Full RL with 100 updates at batch size 1024, and Light-RL (β=0.2), Medium-RL (β=0.5), and Heavy-RL (β=0.8), using the same base model and op=11–14 adaptation range. Under its stated cost formula, the Full RL configuration corresponds to approximately 2.10B equivalent tokens, so the two stated endpoint budgets do not match.
• Observation 3: Full mid-training and Light-RL favor OOD-edge performance, with Light-RL strongest at pass@1 in Figure 6. Heavy-RL improves OOD-hard pass@1 and pass@128 relative to the other displayed allocations.
• Figure 1 reports an average +10.8% OOD-hard pass@128 advantage for mid-training plus RL over full RL under the same reported budget.

At sufficient B exposure, correct harder solutions occupy lower graph-similarity ranges than many simpler solutions. This supports the authors’ interpretation of different compositions within the generator, but the distributions alone do not establish new general-purpose reasoning capabilities.

Takeaway 3 and Practical Guidance 3 recommend allocating compute according to the desired generalization regime: more mid-training with light RL for nearby-task reliability, and a smaller mid-training stage with heavier RL for complex extrapolation. Discussion 3 connects these results to cited work on mid-training as a distributional bridge, supervision that brings the training data closer to subsequent RL tasks.

• The comparison estimates compute from model passes and token lengths; it does not report hardware-specific wall-clock equivalence.
• A.10 qualifies the recommendation: at larger budgets, full RL can achieve the highest OOD-hard pass@128, so a hybrid allocation is not uniformly optimal.

The allocation ranking changes between nearby and hard evaluation ranges: Light-RL is strongest for nearby-task pass@1, while Heavy-RL retains higher performance on several harder unseen operation counts. All displayed strategies approach zero at the most difficult counts, limiting the extent of the observed extrapolation.

6 Mitigating Reward Hacking via Process Supervision in Outcome Rewards

Section 6 addresses reward hacking, obtaining correct final answers through shortcuts or invalid reasoning. It trains on op=11–14 with the composite reward R=αR_out+(1−α)R_pv, where R_out is binary final-answer correctness, R_pv measures intermediate-step accuracy, and α controls their weighting; it also tests a strict reward that grants R_out only when R_pv=1.

• Observation 4: On op=15–20, the 0.2 outcome + 0.8 process mixture adds 4.3 percentage points to pass@1 and 4.5 points to pass@128 over outcome-only RL. The strict reward adds 4.1 and 2.6 points, respectively.
• On op=2–14, process-aware variants add 4.6–5.2 points to pass@1, while pass@128 changes range from −0.1 to +0.3 points. Benefits therefore differ by metric and difficulty.

Takeaway 4 links process-aware rewards to more faithful reasoning. Discussion 4 supports this interpretation with structural error analysis: process-aware training reduces dependency mismatches, incorrect sets of prerequisite quantities, and missing nodes more than outcome-only training in the displayed comparisons.

• At op=15–20, the base model has 92.3% dependency mismatches; the 0.2 outcome + 0.8 process mixture reduces this by 10.5 percentage points, compared with 1.9 points for outcome-only RL.
• Practical Guidance 4 recommends combining sparse outcome feedback with dense intermediate-step feedback when process supervision is sufficiently accurate.

The 0.2 outcome + 0.8 process reward raises hard-task pass@1 from 5.6% to 9.9% and pass@128 from 23.2% to 27.7%. Pass@128 on op=2–14 remains approximately unchanged, so the figure supports improvements that depend on the metric and evaluation range.

7 Related Work

Related Work contrasts studies in which RL’s advantage diminishes at large pass@k with studies showing gains on synthetic tasks and unseen compositions. The paper positions its contribution as an experimental account of how pre-training primitives and RL data difficulty can produce these different outcomes.

• It also builds on controlled language-model experiments and synthetic arithmetic generators, extending control across pre-training, optional mid-training, and RL.

Process-aware variants reduce dependency mismatches and missing nodes more than outcome-only RL in the displayed comparisons. For op=15–20, the 0.2 outcome + 0.8 process mixture adds 9.8 percentage points to the displayed correct category relative to the base model and reduces dependency mismatches by 10.5 points, providing structural evidence alongside the pass@k gains.

8 Conclusion

The conclusion argues that reasoning improvement through post-training depends on primitives established during pre-training. The controlled experiments support conditional guidance for data curricula, reward design, and compute allocation within the tested synthetic arithmetic setting.

Appendix

• A.1 Data Generation Framework, including A.1.1 Graph-Level Formalism and A.1.2 Abstract and Instance Parameters, defines graphs with elementary addition, subtraction, multiplication, and division, separating symbolic dependencies from numeric values and linguistic entities. A.1.3 Contextual Rendering maps graphs to [question], [solution], and [answer] triples; A.1.4 Generation Pipeline and Structural Knobs describes structure sampling, instantiation, rendering, and forward generation or reverse generation, solving for a designated unknown quantity. A.1.5 Deduplication and Canonicalization removes exact normalized rendered-triple hash duplicates within and across splits, preventing identical problem–solution triples from appearing in training and evaluation.
• A.2 Task Setup formally defines extrapolation beyond the maximum operation count in the union of pre-, mid-, and post-training ranges. It also specifies contextual transfer across templates and clarifies that training does not deliberately align graphs across contexts. Consequently, the main-body OOD-edge label refers to difficulty relative to pre-training; after adaptation on op=11–14, op=15–20 tests extrapolation beyond all operation ranges seen in those training stages.
• A.3 Training Setup contains A.3.1 Model Architecture: 12 layers, hidden size 768, intermediate size 3,072, 12 attention heads, and 2 key-value heads for the 100M-parameter model. A.3.2 Tokenizer and Input Representation trains a byte-pair encoding (BPE) tokenizer, which builds a vocabulary by merging recurring token pairs, with 2,200 tokens and a maximum sequence length of 2,048. A.3.3 Hyperparameters specifies a 512K-token pre-training batch, learning rate 2×10^−4, weight decay 0.1, minimum learning rate 3×10^−5, 5% warmup, and one epoch; mid-training uses learning rate 1×10^−4 and 15% warmup.
• A.3.3 further specifies RL batches of 1,024 examples, maximum prompt and response lengths of 1,024 tokens each, two epochs, actor learning rate 1×10^−6, and reference-policy regularization, a penalty for departing from the reference model, with coefficient 10^−3. RL sampling uses temperature 1.0; evaluation uses temperature 0.7, top-p=1.0, top-k=−1, and at most 1,024 new tokens, with the latter sampling settings applying no probability-mass or token-count truncation. A.3.4 Performance Ladder shows near-saturated op=2–10 pass@128, partial op=11–14 competence, and zero op=15–20 competence during base-model pre-training.
• A.4 Process-Verified Evaluation deterministically parses “Define … as …” steps, extracts dependencies from referenced variables, and evaluates each step’s last computable arithmetic expression, falling back to its last numeric literal when necessary. Missing reference nodes, incorrect parent sets, or incorrect values receive zero step credit. Extra predicted nodes do not affect process accuracy; full correctness requires all reference nodes to pass and the extracted answer to match.
• A.5 Training Dynamics for § 3 reports negative log-likelihood (NLL), a loss measuring how poorly the model predicts reference text, and RL reward trajectories. Training on op=11–14 yields the largest displayed NLL reductions on op=15–20; op=9–12 and op=11–14 rewards improve, while easy-range rewards largely plateau and op=17–20 rewards remain zero. These dynamics are consistent with the accuracy comparisons favoring partially solvable training examples.
• A.6 Detailed Analysis of Post-Training Effects on Contextual Generalization distinguishes two pre-training conditions. In A.6.1 When Reasoning Primitives are Shared During Pre-Training, 99.9% context A and 0.1% context B both cover op=2–20; Observation 5 and Takeaway 5 report transfer to B even with 0% B during RL, with further improvements as its RL share rises. In A.6.2 When Only Atomic Primitives are Exposed During Pre-Training, B occupies 1% of pre-training and is restricted to op=2; Observation 6 and Takeaway 6 show little transfer with 0–1% B during RL, substantial improvement at 10%, and stronger results at 50–100%, while A remains comparatively stable.
• A.6.3 Training Dynamics for § A.6.2 shows largely stagnant rewards at 0–1% target-context RL exposure and increasing rewards at 10–100%. Observation 6 states that introducing around 10% B raises pass@128 by over +76 points, but Figure 13’s 10% curve does not support that magnitude; its largest displayed gains occur at 50–100% B. The figure supports the qualitative dependence on target-context RL exposure, with a discrepancy in the numerical attribution.
• A.7 Detailed Analysis of Pre-Training Effects on Extrapolative Generalization varies harder-example pre-training exposure while keeping RL at op=11–14. Observation 7 and Takeaway 7 interpret partially established compositional skills as a foundation for deeper extrapolation; the displayed op=15–20 pass@128 gains are +5.9, +10.4, +22.1, +18.0, and +25.8 points at hard-data shares of 0.1%, 5%, 20%, 33.3%, and 50%, respectively. These plotted values are nonmonotonic and contradict the prose’s attribution of the largest boost to 20%.
• A.7.1 Training Dynamics for § A.7 plots increasing rewards for hard-data shares from 0.1% through 50%; its caption additionally discusses 0% and 100%, which are absent from the displayed panels. A.8 Training Dynamics for § 4 shows little sustained reward progress with 0.1% elementary context B exposure but increasing rewards with 1% and 10%. The displayed trajectories complement the accuracy comparisons; they do not verify the caption’s unplotted conditions.
• A.9 Post-Training and Pre-Training Data Recipe identifies context A as animals–zoo, B as teachers–school, and C as movie-festival, and tabulates 10B-token pre-training budgets and 204.8k-sample RL budgets. Section 3’s easiest RL range is op=8–10 in the table versus op=7–10 in the main text, and A.7’s prose describes hard examples as op=7–10 while several table recipes use op=8–10. Table 2 also includes a 0.1% op=8–20 recipe and omits an explicit 33.3% recipe corresponding to Figure 15, leaving parts of the plotted ablation insufficiently specified for reproduction.
• A.10 Mid-/Post-Training Mixing with Different Computation Budget and A.10.1 Compute Budget of Mid-Training and RL Equivalence derive supervised cost C_train≈6PT and RL cost C_RL=(8+2γ)PNrL_total, where P counts non-embedding parameters and γ toggles the reference-model pass. With γ=1, the equivalent supervised-token cost is T_RL=(5/3)NrL_total. The appendix adaptation recipe uses 20% op=2–10 and 80% op=11–14 for both stages, and Table 3 lists total budgets of 1.05, 2.10, 4.20, 8.40, 12.58, 16.78, and 20.00B equivalent tokens.
• Observation 8 and Takeaway 8 describe task- and budget-dependent optima: hybrid training often supports pass@1, while Heavy-RL is favored for OOD-hard pass@128 at 4.2B equivalent tokens and Full RL is favored at 8.4B and above. Figure 18 contains exceptions to the accompanying universal pass@1 claims: at 4.2B, full mid-training has the highest displayed OOD-edge pass@1, and at 20.0B, Full RL has the highest displayed OOD-hard pass@1. The results support complementary strengths whose preferred allocation depends on the task, metric, and budget.

Brief Thoughts

The main contribution is the joint control of pre-training coverage, adaptation difficulty, and reasoning verification. Saturated ID pass@128 alongside improved harder-task pass@128 provides evidence that RL gains depend on the training regime, while intermediate-step checks reduce the chance of counting correct-answer shortcuts as valid solutions.

The evidence comes from one 100M-parameter architecture trained on synthetic arithmetic with a small template inventory, so the 1% exposure threshold and preferred compute splits require validation on broader tasks and larger models. Failure to find a correct base-model solution in 128 samples establishes a finite-sampling performance gap; it does not prove that the solution had zero probability before RL. The parser verifies generated traces against reference graphs without directly observing internal computation.

Several appendix claims exceed their plotted support, operation ranges differ between prose and recipe tables, and Section 5’s stated endpoint configurations do not have equal costs under its own formula. These discrepancies limit precise reproduction and allocation recommendations. The main comparisons still motivate three hypotheses to test: select partially solvable RL tasks, establish target-context primitives before expecting transfer, and compare process-aware rewards with outcome-only rewards.