Demystifying Long Chain-of-Thought Reasoning in LLMs | Summary

This article explains the key points of Demystifying Long Chain-of-Thought Reasoning in LLMs.

• 2025-02-05 (arXiv)
• Yeo, Edward, Tong, Yuxuan, Niu, Morry, Neubig, Graham, Yue, Xiang.
• IN.AI, Tsinghua University, Carnegie Mellon University
• Paper
• Github

Read this article in Korean

Summary

• Training large language models (LLMs) to produce long chains-of-thought (CoTs), extended reasoning traces that explore alternatives and correct errors, requires choices that keep learning stable and improve answer accuracy. On Qwen2.5-Math-7B, supervised fine-tuning (SFT), training on example responses, followed by reinforcement learning (RL), optimizing responses using rewards, achieves 49.7% average accuracy across four benchmarks versus 41.0% for direct RL from the base model under the paper’s different training and decoding setups.
• Long CoT data distilled from an existing reasoning model, using its generated responses as training targets, supports stronger scaling and subsequent RL gains than the tested short CoT data or constructed action sequences. For Llama-3.1-8B with approximately 200k SFT samples, distilled patterns followed by RL reach 59.4% on MATH-500, a competition-mathematics test subset, and 34.6% on MMLU-Pro-1k, a 1,000-question general reasoning test subset, compared with 52.4% and 19.2% for constructed patterns.
• A reward based only on answer correctness can cause responses to exhaust the context window, the available token budget, and accuracy to collapse. Cosine Reward, a correctness reward shaped by response length, stabilizes the tested training runs; penalizing repeated token sequences further reduces response length and improves several benchmark scores.
• Noisy web-derived supervision improves out-of-distribution (OOD) performance, performance outside the main mathematical training distribution, when combined with suitable data mixtures or filtering for verifiable short answers. Direct RL also improves accuracy without consistently increasing reflection keywords or exceeding the base model’s original response length, so these measurements alone do not establish newly emerging reasoning abilities.

1. Introduction

The paper studies how SFT affects long CoT learning, how rewards control length and stability, how to scale verifiable supervision, and whether apparent reasoning behaviors originate during RL or already exist in base models. Longer reasoning can support exploring alternative paths, returning to earlier steps, and correcting errors, but the training conditions that reliably produce these behaviors remain unclear.

2. Problem Formulation

The research aim is to identify training choices that improve and stabilize long CoT reasoning through comparisons and ablations, experiments that vary selected components. The formulation separates the generated reasoning trace from the final answer and considers both supervised initialization and reward-based optimization.

2.1. Notation

For query x and output sequence y, a model with parameters θ defines the conditional token distribution πθ(yt | x, y1:t−1), and \(\operatorname{CoT}(y)\subseteq y\) denotes the reasoning tokens. Long CoT refers to both extended length and behaviors such as branching, exploring multiple solution paths; backtracking, returning to earlier points after a failed path; and error validation and correction, detecting and repairing intermediate mistakes.

2.2. Supervised Fine-Tuning (SFT)

SFT initializes the policy, the model’s distribution over generated tokens, using a dataset DSFT = {(xi, yi)}i=1N, where each target response can contain a normal or long reasoning trace. It teaches response patterns that RL can subsequently optimize.

2.3. Reinforcement Learning (RL)

RL optionally follows SFT and uses an outcome-based reward, a reward determined by the final answer rather than annotated intermediate reasoning steps. The default optimizer is Proximal Policy Optimization (PPO), a policy optimization algorithm, with a rule-based verifier that compares predicted answers directly with reference answers.

2.4. Training Setup

The main base models are the general-purpose Llama-3.1-8B and mathematics-specialized Qwen2.5-Math-7B. By default, both SFT and RL use the 7,500 MATH training prompts; SFT responses are obtained through rejection sampling, generating candidates and retaining those with correct final answers. Training uses OpenRLHF, a framework for language-model training, while later experiments introduce WebInstruct, a more diverse collection of web-extracted questions and solutions with less reliable supervision.

2.5. Evaluation Setup

Evaluation uses MATH-500 for in-domain mathematics, AIME 2024, the American Invitational Mathematics Examination’s 30 problems, for out-of-domain competition mathematics, TheoremQA for theorem-based reasoning across mathematics and related technical subjects, and MMLU-Pro-1k for general reasoning. Default generation uses temperature 0.7, which controls sampling randomness, top-p 0.95, which restricts sampling to tokens covering that cumulative probability, and a maximum output length of 16,384 tokens. Direct RL from the base model uses a separate greedy-decoding setup, selecting the highest-probability next token, described in Section 6.

The original and downsampled source-frequency bars closely align, supporting the intended distribution preservation of MMLU-Pro-1k. This checks subset composition and does not establish equal evaluation uncertainty to the full benchmark.

3. Impact of SFT on Long CoT

The SFT experiments compare long and short reasoning traces as standalone training targets and initializations for RL. A separate comparison tests manually sequenced reasoning actions against traces distilled from an existing long CoT model.

3.1. SFT Scaling

For Llama-3.1-8B, long traces come from QwQ-32B-Preview and short traces from Qwen2.5-Math-72B-Instruct. Rejection sampling uses N ∈ {32, 64, 128, 192, 256} candidates per prompt for long CoT and N ∈ {32, 64, 128, 256} for short CoT, increasing the available SFT tokens.

• Figure 1 shows continued gains from long CoT training and weaker scaling from short CoT training. Section 3.1 reports over 70% MATH-500 accuracy and no plateau at 3.5B tokens, although the plotted SFT-only curve remains below 70%; the SFT+RL curve exceeds it.
• The comparison changes the teacher model as well as trace length, so it does not isolate length as the sole cause.

3.2. SFT Initialization for RL

The authors initialize PPO from the preceding SFT checkpoints and make four passes through the MATH prompt set, sampling four responses per prompt with Cosine Reward and a repetition penalty. Figure 1 shows that RL typically adds over 3 percentage points on MATH-500 after long CoT SFT, while short CoT initializations receive little improvement.

Across four benchmarks, the long CoT curves generally improve more with additional SFT tokens and gain more from subsequent RL. Different teacher models supply the long and short traces, so the curves do not isolate response length; the plotted SFT-only MATH-500 curve also remains below the section’s stated over-70% result.

3.3. Sources of Long CoT SFT Data

Action Prompting constructs traces from clarify, decompose, solution_step, reflection, and answer actions, using a short CoT model to sequence actions and o1-mini-0912 to generate reflection steps. The alternative distills traces from QwQ-32B-Preview; both approaches use MATH prompts, rejection sampling, approximately 200k SFT samples, and the same Llama-3.1-8B base model.

• Table 1 favors distilled patterns: after SFT, MATH-500 accuracy is 54.1% versus 48.2% for constructed traces, and MMLU-Pro-1k accuracy is 32.0% versus 18.1%.
• After RL, distilled traces reach 59.4%, 4.0%, 25.2%, and 34.6% across MATH-500, AIME 2024, TheoremQA, and MMLU-Pro-1k. Constructed traces reach 52.4%, 2.7%, 21.0%, and 19.2%, showing smaller and less consistent RL gains.

Distilled patterns outperform constructed action sequences before and after RL. The MMLU-Pro-1k gap, 34.6% versus 19.2% after RL, is particularly large; constructed traces nevertheless improve on MATH-500, so their RL gains are limited rather than absent.

4. Impact of Reward Design on Long CoT

The reward-design experiments examine how answer correctness, response length, repetition, context limits, and temporal credit assignment, how later rewards influence earlier generated tokens, interact. They test whether reward shaping can maintain accurate answers while controlling response length.

4.1. CoT Length Stability

With long CoT SFT initialization and a 16K context window, both Llama-3.1-8B and Qwen2.5-Math-7B become unstable under Classic Reward, a reward of 1 for correct answers. Figure 2 shows increasing context-limit violations and MATH-500 accuracy falling near zero; Llama-3.1-8B exhibits greater length fluctuations.

• The context-exceedance rate levels off below 1. The authors interpret this as implicit pressure against excessive length, potentially arising without a dedicated penalty through normalization of rewards or advantages, the estimated benefit of actions relative to their expected value.

Both initialized models develop excessive response lengths under Classic Reward, with increasing context-limit violations and MATH-500 accuracy falling near zero. Separating terminated responses from all responses reveals why statistics restricted to completed outputs can obscure the failure.

4.2. Active Scaling of CoT Length

Cosine Reward gives correct responses higher rewards than incorrect ones, favors shorter correct responses, and penalizes short incorrect responses more than longer incorrect responses. It supplies one terminal reward based on correctness and generation length, with a separate penalty when generation reaches the maximum length.

• Figures 3 and 4 show the reward’s length dependence and its stabilizing effect on Llama-3.1-8B: Classic Reward collapses, while Cosine Reward maintains steadier accuracy and response length.
• Figure 5 shows downstream improvements and additional benefits on several benchmarks from combining Cosine Reward with a repetition penalty.

Classic Reward does not distinguish response lengths. The plotted Cosine Reward favors shorter correct responses while making short incorrect responses more costly, introducing length-dependent incentives without intermediate reasoning annotations.

4.3. Cosine Reward Hyperparameters

The authors vary rewards for correct and incorrect answers at zero and maximum generation lengths. Figure 9 shows that rewarding longer correct responses drives rapid length growth and impairs stopping within the context window; raising correct-answer rewards while keeping wrong-answer penalties fixed produces shorter traces in the tested configurations.

• The interpretation as trained risk aversion remains a hypothesis: reward ratios may affect how confident the model must be before ending its reasoning with an answer.

Reward A increases the correct-answer reward with length and produces rapid response growth. Rewards B and C favor shorter correct answers and keep lengths lower, demonstrating that reward endpoints affect stopping behavior.

4.4. Context Window Size

The context-window ablation compares 4K, 8K, and 16K windows using the same starting Llama-3.1-8B model, MATH prompts, Cosine Reward, repetition penalty, and number of training samples. Figure 6 shows that 8K improves over 4K and generally performs better than 16K under this fixed sample budget.

• The authors suggest that a larger window may require more training samples to use effectively. The experiment does not determine how much additional training would make 16K preferable.

At the same training sample count, the 8192-token setup generally outperforms 4096 tokens and often outperforms 16384 tokens. A larger context window therefore does not provide uniformly higher accuracy under this sample budget.

4.5. Length Reward Hacking

With prolonged training under length-shaped rewards, the model sometimes lengthens difficult responses by repeating text, an instance of reward hacking, exploiting the reward without improving the intended behavior. The authors also observe fewer occurrences of “alternatively,” their indirect measure of branching.

• An N-gram repetition penalty, a penalty for recurring sequences of N tokens, is applied directly to repeated tokens. Discounting the penalty, reducing its influence on more distant earlier tokens, helps localize its learning signal.
• Figures 5 and 12 show improved scores on several downstream tasks and shorter responses. The authors report that lower training accuracy can increase pressure toward repetition under Cosine Reward.

Cosine Reward supports continued downstream improvement beyond the Classic Reward run. Adding the repetition penalty generally improves MATH-500, AIME 2024, and TheoremQA, while the final MMLU-Pro-1k result is slightly below the cosine-only curve.

The count of “alternatively,” decreases during cosine-only training. Together with observed repetition, this motivates the authors’ concern about reward exploitation, although the keyword is an indirect measure of branching.

The response-length curve with repetition penalties generally lies below the cosine-only curve. This supports reduced generation length under the tested penalty, without by itself measuring the quality of the remaining reasoning.

4.6. Optimal Discount Factors

The authors modify generalized advantage estimation (GAE), the calculation of reward-based learning signals for generated actions, so each reward type has its own discount factor. With \(\lambda=1\), the estimator is \(\hat A_t=\sum_{l=0}^{L}\sum_{m=1}^{M}\gamma_m^l r_{m,t+l}-V(s_t)\), where γm discounts reward type m over future steps and V(st) is the estimated value of the current state.

• Table 5 favors preserving long-range credit for correctness while discounting repetition penalties: correctness discount 1.000 and repetition discount 0.999 achieve 58.0% on MATH-500, 26.0% on TheoremQA, and 36.5% on MMLU-Pro-1k.
• Lowering both discounts to 0.99 reduces those scores to 47.9%, 15.6%, and 25.5%. More frequent “alternatively” under a lower correctness discount accompanies worse performance, so the keyword count alone does not establish better exploration.

Lower correctness discount factors produce more occurrences of the branching keyword. Combined with weaker accuracy in Table 5, the pattern shows that more frequent switching between stated approaches can accompany worse performance.

Correctness discount 1.000 with repetition discount 0.999 gives the strongest results on three benchmarks, while 1.000 for both gives the best AIME 2024 result. Lower correctness discounts substantially reduce performance, supporting separate treatment of the reward types in this experiment.

5. Scaling up Verifiable Reward

To reduce dependence on scarce human-annotated reference answers, the authors investigate WebInstruct’s web-extracted reasoning question–answer pairs. They create WebInstruct-462k using MinHash, an algorithm used here for deduplication, and test this noisier source in SFT and RL.

5.1. SFT with Noisy Verifiable Data

SFT compares MATH-only data, WebInstruct-only data, and an approximately 50% MATH plus 50% WebInstruct mixture, with responses distilled from QwQ-32B-Preview. MATH responses are filtered using reference answers, while WebInstruct contributes one unfiltered teacher response per prompt; subsequent RL uses MATH.

• Table 2 shows that the mixture gives the best average accuracy: 30.8% after SFT and 32.1% after SFT+RL, versus 27.9% and 30.8% for MATH only. WebInstruct-only training scores below MATH-only training on the average, so adding noisy data is not uniformly beneficial.
• The mixture increases SFT MMLU-Pro-1k accuracy from 32.0% to 41.7%, while MATH-500 changes from 54.1% to 53.6%. The prose prints “510% absolute”; the table supports a 9.7 percentage-point increase for this comparison.

The MATH and WebInstruct mixture yields the highest four-benchmark average after SFT and after RL. WebInstruct-only training improves general reasoning scores but lowers mathematical scores and the average relative to MATH-only training.

5.2. Scaling up RL with Noisy Verifiable Data

The RL comparison crosses filtered versus unfiltered WebInstruct prompts with rule-based versus model-based verification, using another language model to judge answer correctness. Short answers are extracted using Llama-3.1-8B-Instruct, and two QwQ-32B-Preview responses per prompt are checked against them, yielding approximately 189k responses across 115k retained prompts. Each RL condition samples 30k prompts from its corresponding pool and generates four responses per prompt.

• Table 3 shows that filtered prompts with a rule-based verifier achieve 48.6% on MATH-500, 28.1% on TheoremQA, and 41.4% on MMLU-Pro-1k. Filtered model-based verification reaches the highest AIME 2024 score among the WebInstruct RL conditions, 3.8%, and ties at 41.4% on MMLU-Pro-1k.
• Relative to the MATH-only SFT+RL baseline, filtered rule-based training improves TheoremQA by 2.9 percentage points and MMLU-Pro-1k by 6.8 points, but lowers MATH-500 from 59.4% to 48.6%. Unfiltered rule-based verification performs worse than unfiltered model-based verification on all four benchmarks.

Filtering improves rule-based verification on all four benchmarks relative to its unfiltered counterpart. Among the WebInstruct RL conditions, filtered rule-based verification leads on MATH-500 and TheoremQA and ties on MMLU-Pro-1k; filtered model-based verification leads on AIME 2024.

6. Exploration on RL from the Base Model

The direct-RL experiments ask whether accuracy gains coincide with newly appearing reflection patterns or growth beyond the base model’s response length. They also compare direct RL against long CoT SFT initialization and examine whether similar reasoning patterns occur in web data.

6.1. Nuances in Analysis Based on Emergent Behaviors

Qwen2.5-Math-7B is trained with PPO and a rule-based verifier on approximately 8k MATH level 3–5 questions. Evaluation uses zero-shot prompting, providing no demonstration examples, greedy decoding at temperature 0, and a maximum output length of 4096 tokens; the training context length is also 4096 tokens.

• Figure 7 tracks “wait”, “recheck”, “alternatively”, “retry”, and “however” as indirect measures of reflection. Accuracy improves substantially, but these keywords do not consistently become more frequent; “recheck” already occurs in the base model.
• The evidence shows that accuracy gains need not coincide with these surface markers. It does not establish the absence of all forms of self-validation.

Accuracy rises during direct RL without a consistent increase in the selected reflection keywords. These lexical markers remain sparse or fluctuate independently of task improvement, limiting their use as evidence of newly emerging reflection.

6.2. Nuances in Analysis Based on Length Scaling

Figure 8 shows an initial response-length decrease followed by recovery that remains below the base model’s original length. Outputs containing `python are classified as coding, but their code is not executed; the model generates the code’s purported results.

• Both coding and natural-language outputs initially shorten, and natural-language outputs are initially longer. This challenges the explanation that the length drop simply reflects a transition from code to prose.
• Later length recovery coincides with decreasing Kullback–Leibler (KL) divergence, a measure of how the trained policy differs from the base model. The authors hypothesize that the KL penalty encourages a return toward the base model’s behavior; the experiment does not isolate the penalty’s causal effect.

Response lengths initially fall and then partly recover, remaining below the starting base-model length. Both code-containing and natural-language responses shorten, while later recovery accompanies falling KL divergence, motivating but not proving the proposed regularization explanation.

6.3. Potential Reasons Why Emergent Behavior is Not Observed with Qwen2.5-Math-7B

The authors propose two possible explanations for not reproducing DeepSeek-R1-like emergent behavior in Qwen2.5-Math-7B: limited capacity at 7B parameters and possible overexposure to MATH-like short instruction data during continued pre-training and annealing, the final phase of pre-training. Both remain hypotheses because the study does not directly vary model capacity or pre-training exposure.

6.4. Comparison between RL from the Base Model and RL from Long CoT SFT

For Qwen2.5-Math-7B, the long CoT route uses the 7.5k MATH training prompts, 32 QwQ-32B-Preview candidates per prompt for rejection sampling, and PPO with Cosine Reward, repetition penalty, and eight responses per prompt over eight passes through the data. The authors multiply the rotary position embedding (RoPE) parameter θ, a parameter of the model’s positional representation, by 10 to adapt its original 4096-token context to longer training.

• Table 4 reports average accuracy of 21.2% for the base model, 41.0% for direct RL, 47.3% for long CoT SFT, and 49.7% for SFT+RL. The latter reaches 85.9% on MATH-500, 26.9% on AIME 2024, 45.4% on TheoremQA, and 40.6% on MMLU-Pro-1k.
• SFT+RL exceeds direct RL by 8.7 percentage points on the reported average. Its increase over SFT is 2.4 points using the table’s rounded averages, although the prose states 2.6%.
• Training settings and decoding differ: direct RL is evaluated greedily, while the long CoT route uses default temperature sampling. The result compares complete setups and does not isolate initialization; Classic Reward training after long CoT SFT is omitted because it collapses.

Long CoT SFT exceeds direct RL on the reported average, and SFT+RL raises it to 49.7%. Training and decoding differ between routes, so the table compares complete configurations; its rounded averages imply a 2.4-point improvement over SFT, whereas the prose states 2.6%.

6.5. Long CoT Patterns in Pre-training Data

The authors use Perplexity.ai to locate webpages containing alternative solution methods and explicit verification, then search OpenWebMath, a collection of mathematical web text, using MinHash and phrases generated by GPT-4o. Retrieved forum discussions contain branching, backtracking, uncertainty, and corrections resembling long CoT traces.

• These examples motivate the hypothesis that RL recombines skills acquired during pre-training. They do not establish that particular documents trained the evaluated models or caused their reasoning behavior; the authors also acknowledge that forums are common in OpenWebMath.

7. Discussions and Future Work

The discussion identifies unresolved questions about model scale, RL infrastructure, optimizer tuning, verification, and the origins of reasoning behavior. The proposed directions extend beyond what the reported experiments establish.

7.1. Scaling up Model Size

The authors suggest that larger base models may more readily develop the behaviors absent in their direct-RL experiment. The study provides no model-size scaling experiment that confirms size as the limiting factor.

7.2. RL Infrastructure Is Still in Its Infancy

An attempted expansion to 32B models was abandoned because the required number of graphics processing units was too large. The authors identify duplicated model parameters across training and inference systems, synchronous PPO workloads, and variable response lengths that leave slow requests delaying a batch as infrastructure constraints. They supply no quantitative systems benchmark.

7.3. REINFORCE Is More Tricky to Tune than PPO

REINFORCE++, a policy-gradient alternative tested for faster RL, exhibits lower training accuracy and greater instability than PPO in the authors’ setup. Figure 13 documents unstable accuracy and length, but the authors avoid a general algorithm comparison because the REINFORCE++ configuration may be insufficiently tuned.

The run exhibits substantial length fluctuations and a late accuracy decline. It documents instability in the tested REINFORCE++ configuration, whose potentially untuned settings prevent a general comparison with PPO.

7.4. Scaling up Verification

Rule-based verification with prompt filtering performs well in the reported experiments, but extending rules and curated prompt sets across domains remains labor-intensive. The authors propose investigating silver supervision, targets without fully reliable human verification, and self-supervised approaches that derive supervision without manually annotated targets to scale verification.

7.5. Latent Capabilities in Base Models

The paper frames reasoning behaviors as potentially latent in base models and identifies human forum dialogue as one possible source. It calls for analyses that trace behaviors to training data, since the presented textual similarities do not resolve their origins.

Impact Statement

The impact statement describes potential benefits from stronger reasoning and problem solving and emphasizes reliable data preparation, stable training, and verifiable reference answers. It recommends careful reward and training design to limit unintended consequences, without reporting a separate empirical safety evaluation.

Appendix

• A. Related Work situates the study among CoT prompting, inference-time scaling, increasing computation while generating an answer, and outcome-reward RL systems such as DeepSeek-R1 and Kimi K1.5.
• B. Figures and Tables supplements the main results with reward-shape ablations, reflection-keyword counts, response-length trajectories, optimizer instability, and discount-factor comparisons.
• C. Algorithms and Formulas gives C.1. Cosine Reward Formula as CosFn(t, T, ηmin, ηmax) = ηmin + ½(ηmax − ηmin)(1 + cos(tπ/T)). C.2. N-gram Repetition Penalty records previously observed N-grams and assigns penalty P to tokens in subsequent occurrences.
• D. Extracts illustrates rapid switching between tentative approaches at correctness discount \(\gamma_c=0.99\).
• E. Experimental Setup and E.1. Evaluation Setup specify 30 AIME 2024 problems, 800 TheoremQA samples covering 350+ theorems, and a 1,000-question MMLU-Pro-1k subset whose source distribution closely matches the original in Figure 14. Accuracy is averaged over at least 4 random seeds, with 16 responses per AIME prompt to reduce sampling variance. Inference uses vLLM, an inference library, and SymEval, an answer grader supporting mathematical objects such as matrices and functions; some RL experiments used an earlier grader version that can cause performance differences. E.2. Details about Distillation uses temperature 1.0, top-p 0.95, and an 8192-token output limit for QwQ-32B-Preview, versus temperature 0.7, top-p 0.95, and 4096 tokens for Qwen2.5-Math-72B-Instruct. E.3. Details abour SFT Setup reports batch size 256, context length 128K, learning rate 5e-6, 2 epochs, and micro batch size 1, using OpenRLHF with Flash Attention 2, an efficient attention implementation, and ZeRO, a memory optimization method, through DeepSpeed. E.4. Details about RL Setup uses OpenRLHF and its hyperparameter naming conventions. E.5. Experiment Hyperparameters distinguishes passes through the prompt set from optimization epochs and defines Samples as responses per prompt; BS denotes rollout_batch_size and train_batch_size, which generally share a value. E.5.1–E.5.4 document SFT initialization, Classic Reward instability, active length control, and reward-endpoint ablations; E.5.5–E.5.7 specify context limits, repetition penalties, and separate discount factors. Common settings include batch size 512, actor learning rate 5e-7, KL coefficient 0.01, and, for many 16K runs, 2048 prompt tokens plus 14336 generation tokens; the repetition penalty commonly uses N = 40 and P = −0.05. E.5.8 compares noisy-data verifiers using 30k sampled prompts and 4 responses per prompt. E.5.9 documents direct base-model RL with correct, wrong, and missing-answer rewards of +1, −0.5, and −1, a 1024-token prompt limit, and a 3072-token generation limit, alongside long CoT initialization setups; E.5.10 reports an early-stopped REINFORCE++ run with learning rate 5e-7 and clipping value 0.1. E.6. Implementation of the Model-Based Verifier gives the verifier the last 20 response lines and the reference solution; it names Qwen2.5-7B-Instruct, whereas Section 5.2 names Qwen2.5-Math-7B-Instruct, leaving an implementation inconsistency in the PDF. E.7. Implementation of Short-Form Answer Extraction uses Llama-3.1-8B-Instruct with temperature 0, a 512-token limit, and extraction from \boxed{…}; E.8. Action Prompting Framework, E.8.1. CONTROL FLOW, and E.8.2. ACTION PROMPTING TEMPLATES describe a state machine that appends generated thoughts, lets the model select actions and branches, and terminates after answer, with o1-mini generating reflection.
• F. Long CoT Patterns in Pre-training Data presents F.1. Snapshot of webpages with verification and alternative arithmetic methods, then F.2. OpenWebMath, F.2.1. QUERIES, and F.2.2. MATCHES with generated search phrases and examples from MC Stan, Physics Forums, and StackExchange. Code is available at https://github.com/eddycmu/demystify-long-cot and the grader at https://github.com/tongyx361/symeval.

Brief Thoughts

The clearest engineering evidence connects reward design, context-limit failures, and downstream accuracy: Figures 2–5 show why rewarding correct answers alone can fail after long CoT initialization. Table 3 also shows that filtering for answers the verifier can check improves several generalization scores, with lower mathematical accuracy than the MATH-only baseline.

The evidence for particular training configurations is stronger than the evidence about the origins of reasoning. Keyword counts, correlated length and KL trajectories, different teacher models and decoding conditions, and qualitative web matches leave the causal explanations unresolved.