Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs | Summary

This article explains the key points of Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs.

• 2025-03-03 (arXiv), COLM 2025
• Gandhi, Kanishk, Chakravarthy, Ayush, Singh, Anikait, Lile, Nathan, Goodman, Noah D..
• Stanford University, SynthLabs
• Paper
• Github

Read this article in Korean

Summary

• The paper investigates why similarly sized language models improve differently under reinforcement learning (RL), which rewards generated solutions on verifiable problems. Under identical training on Countdown, an arithmetic puzzle requiring input numbers to be combined into a target, Qwen-2.5-3B reaches approximately 60% accuracy versus Llama-3.2-3B’s 30% after 250 training steps.
• The authors study verification, checking intermediate results; backtracking, abandoning unsuccessful approaches; subgoal setting, proposing intermediate objectives; and backward chaining, reasoning backward from a desired outcome. Priming Llama by fine-tuning it on examples containing these behaviors enables subsequent RL performance that matches or exceeds unprimed Qwen, while empty or placeholder-filled reasoning controls provide little comparable benefit.
• Incorrect solutions containing the target behaviors support subsequent improvement comparable to the standard all-strategies priming condition, whose demonstrations are not uniformly correct. Continued pretraining on behavior-enriched OpenWebMath, a mathematical web-text dataset, also brings Llama’s learning trajectory close to Qwen’s, whereas a behavior-minimized control with the same 8.3 million-token budget improves less.
• The interventions support a causal contribution of the supplied reasoning behaviors within this experimental setting, particularly backtracking and verification. Generalization is uneven: behavioral changes transfer to other question-answering tasks, but the two continued-pretraining Llama variants both achieve approximately 12% on GPQA, a graduate-level question-answering benchmark, despite different behavior frequencies.

1 Introduction

Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs asks which properties of an initial language model enable productive use of test-time compute, the computation spent generating a solution at inference. The authors hypothesize that initially available search-like behaviors help RL discover useful extended reasoning, and test this through synthetic behavioral priming and targeted continued pretraining.

• The investigation moves from observing different learning trajectories to modifying the initial models, providing intervention evidence beyond the comparison between model families.
• Code and prompts are available at https://github.com/kanishkg/cognitive-behaviors.

The overview connects the initial Qwen–Llama learning gap with differences in observable reasoning and shows two interventions that reduce that gap. Its behavioral panels motivate the hypothesis, while the priming and continued-pretraining panels provide intervention evidence.

2 Related Work

The paper distinguishes external search, which evaluates multiple generated candidates; in-context search, which explores alternatives sequentially within generated text; and RL approaches that reward successful reasoning trajectories. Its contribution concerns why the initial model supports or limits subsequent RL improvement, complementing earlier work on search procedures, self-correction examples, and long structured reasoning demonstrations.

• External search can repeat previously explored solutions when candidates do not share their search history.
• Training on explicit search or correction traces can introduce useful behaviors, but requires deliberate construction of the training data.

3 Identifying and Engineering Self-Improving Behavior

The experiments compare two base models, define observable behavioral measurements, intervene on their initial behavior distributions, and test a broader mathematical-data intervention. This sequence examines whether examples of particular reasoning behaviors prepare a model for later reward-driven improvement.

3.1 Initial Investigation: A tale of two models

Countdown provides a bounded arithmetic search problem with relatively limited dependence on mathematical knowledge. For example, the inputs 25, 30, 3, and 4 can reach the target 32 through (30 − 25 + 3) × 4.

• The authors train Qwen-2.5-3B and Llama-3.2-3B using Proximal Policy Optimization (PPO), an RL optimization algorithm, through the VERL training library and TinyZero implementation for 250 steps, sampling 4 generated solution trajectories per prompt.
• Qwen’s accuracy and response length increase around step 30, and its final accuracy is approximately 60%, compared with Llama’s 30%. PPO was selected for stability across hyperparameter settings; the paper describes similar performance across alternative algorithms only anecdotally.
• Later Qwen outputs sometimes replace explicit verbal checking with sequential attempts at different solutions, showing that measured explicit behavior can change while task performance remains high.

3.2 A Framework for Analyzing Cognitive Behaviors

The framework counts explicit occurrences of four behaviors in generated reasoning: revisions that replace previous approaches, checks against solution criteria, proposed intermediate objectives, and paths constructed backward from the target. GPT-4o-mini serves as the classifier that identifies and counts these patterns.

• The four categories are selected because they are identifiable in text and describe search-like reasoning beyond a single uninterrupted sequence of steps.
• These measurements concern visible output patterns and do not directly establish the model’s internal computational process. The paper’s examples of implicit checking illustrate this measurement limit.

3.3 The Role of Initial Behaviors in Self-Improvement

Before RL, Qwen exhibits more of all four measured behaviors than either Llama-3.2-3B or Llama-3.1-70B. During RL, its gains coincide especially with verification and backtracking, while the smaller Llama shows little evidence of either.

• Mean baseline verification counts are 0.62 for Qwen, 0.10 for Llama-3.2-3B, and 0.31 for Llama-3.1-70B; corresponding backtracking counts are 0.65, 0.20, and 0.21.
• The larger Llama increases some behavior frequencies, but its backtracking count remains close to the smaller Llama’s. Model size therefore does not explain this behavioral difference by itself.
• These observations motivate the intervention experiments; the baseline correlations alone cannot establish that the behaviors cause better learning.

3.4 Intervening on initial behaviors

The authors prime the models through supervised fine-tuning (SFT), training on supplied reasoning examples before RL. Claude-3.5-Sonnet generates five behavioral datasets: all strategies, backtracking only, backtracking with verification, backtracking with subgoal setting, and backtracking with backward chaining.

• All five behavioral conditions include backtracking. Both model families subsequently improve substantially, and primed Llama matches or exceeds unprimed Qwen.
• With all strategies supplied, RL retains verification and backtracking while reducing subgoal setting and backward chaining. The latter behaviors persist more when each is paired only with backtracking.
• The design supports the usefulness of these backtracking-containing interventions, but does not independently test verification-only, subgoal-only, or backward-chaining-only priming.

Qwen’s baseline backtracking count is 0.65, compared with 0.20 for Llama-3.2-3B and 0.21 for Llama-3.1-70B. The larger Llama raises verification and subgoal-setting frequencies more than backtracking, indicating that greater parameter count does not uniformly increase the measured behaviors.

Two controls replace chain-of-thought (CoT), the generated intermediate reasoning text, with either an empty sequence, <think></think>, or length-matched placeholder tokens. The paper reports performance around 30–35% for these controls, comparable to base Llama, whereas behavioral priming produces substantially larger gains.

• The length-matched control weakens an explanation based solely on allocating more training tokens to a reasoning field.
• An additional all-strategies dataset contains only incorrect solutions while preserving the desired behaviors. Its subsequent learning curve closely matches the standard all-strategies condition, which also contains imperfect demonstrations, showing that correct priming answers are not required for the observed gains.
• The incorrect-solution finding concerns priming before RL with verifiable rewards; it does not show that correctness is unnecessary throughout training.

Empty reasoning and length-matched placeholder priming remain well below all-strategies priming in subsequent Countdown performance. The placeholder control supplies reasoning-field length without supplying the behavioral content, helping distinguish these factors.

The standard all-strategies and incorrect-answer-only priming curves nearly overlap in both later performance and response length. Behavioral demonstrations can therefore support subsequent improvement even when their final answers are wrong; the later RL stage still uses correctness-based rewards, and the standard priming set itself contains imperfect solutions.

To investigate whether ordinary mathematical text supplies these behaviors, the authors use Qwen-2.5-32B to classify 200,000 randomly sampled documents from OpenWebMath and FineMath, mathematical pretraining corpora. Verification and backtracking occur infrequently, while subgoal setting is the most common measured behavior.

• This analysis motivates enriching mathematical pretraining data with selected behaviors, although it does not inspect or reconstruct the actual pretraining distributions of the compared base models.

3.5 Selectively amplifying behaviors in pretraining data

The authors construct behavior-enriched and behavior-minimized OpenWebMath datasets, each containing 8.3 million tokens. Qwen-2.5-32B selects documents and rewrites them into a structured question-thought-answer format, enabling a comparison with matched token budgets and a common output structure.

• After continued pretraining and RL, behavior-enriched Llama reaches performance comparable to Qwen and shows sustained verification and backtracking. The behavior-minimized model improves less.
• The intervention extends beyond Countdown demonstrations, supporting the usefulness of broader mathematical behavioral data for subsequent Countdown learning.
• Appendix E specifies additional behavioral instructions during rewriting of the enriched set. The experiment therefore evaluates a combined selection-and-rewriting intervention.

Behavior-enriched continued pretraining brings Llama’s Countdown trajectory close to Qwen’s, while the behavior-minimized control remains lower. The enriched model also develops sustained verification and backtracking, linking the performance difference to the intended behavioral change under matched 8.3 million-token dataset budgets.

4 Discussion

The authors interpret the intervention results as evidence that initial reasoning behaviors causally support later RL improvement. They emphasize that the behaviors favored during training depend on the task, with verification and backtracking especially useful for Countdown.

• The four categories are explicitly non-exhaustive; the discussion identifies analogies and awareness of existing knowledge as possible additional behaviors.
• The principal controlled evidence comes from two approximately 3B base models and one arithmetic environment. Broader claims about coding, game play, or creative writing remain proposed extensions.
• Transferred behavior frequencies and transferred answer accuracy require separate evaluation, as the appendix’s GPQA results demonstrate.

Appendix

• A Data Generation: The paper describes a 50-50 split between “3-digit and 4-digit Countdown problems,” with randomly sampled starting numbers and targets. Randomization varies instances while retaining the experimental difficulty parameters.
• B Priming: Each behavioral dataset uses 1,200 unique Countdown games with reasoning trajectories generated by claude-3-5-sonnet-20241022, divided into 1,000 training and 200 validation examples. SFT uses a 2048-token context window, training/validation batch sizes of 64/64, the AdamW optimizer, a peak learning rate of 1e-5, linear warmup over 5% of total steps, cosine annealing, and 5 epochs. The generated datasets contain imperfect solutions: the all-strategies dataset scores 0.43, and the backtracking-with-verification dataset scores 0.44.
• C Reinforcement Learning: PPO uses training/validation batch sizes of 256/1312, a context budget of 256 prompt + 1024 response tokens, actor/critic learning rates of 1e-6/1e-5, a Kullback–Leibler (KL) coefficient of 0.001 for the policy-divergence penalty, a mini-batch size of 128, 4 rollouts, and rollout temperature 1.0. The reward structure assigns 1.0 to a correct answer, 0.0 to an incorrect answer, and a correct-format bonus of 0.1.
• D Metrics: Evaluated model outputs use temperature 1.0 and a maximum of 1024 tokens. GPT-4o-mini receives four separate behavior-counting questions per trajectory, with examples defining each category; classifier generation uses 512 tokens and temperature 0. The reported metrics average distinct occurrences of each behavior.
• E Pretraining Data Interventions.: Qwen2.5-32B classifies mathematical documents at temperature 0, with a generation length of 1024 tokens per behavior. Documents are separated by behavioral presence and rewritten using XML, a markup format that separates question, thinking, and answer components, with first-person language in the thinking sections. Rewriting explicitly requests the target behaviors for the enriched dataset and omits those instructions for the behavior-minimized control.
• F Interrater Reliability for Classifying Behaviors: On 100 randomly sampled reasoning trajectories, the authors compare GPT-4o-mini, Claude, and two humans using the Intraclass Correlation Coefficient (ICC3), a measure of agreement between raters. GPT-4o-mini’s agreement with humans is 0.66/0.65 for verification, 0.67/0.67 for backtracking, and 0.70/0.77 for subgoal setting. Backward chaining has lower agreement with humans, 0.55/0.40, so conclusions involving that category warrant greater measurement caution.
• G Test-time Scaling.: All-strategies-primed Llama checkpoints from 0 to 200 PPO steps are evaluated with maximum response budgets of 128, 256, 512, and 1024 tokens. Larger budgets improve performance, with diminishing returns after 512 tokens; later checkpoints achieve approximately 0.4 with 128 tokens, and the best checkpoints reach around 0.7 with 1024 tokens. These results show improvements in both performance under a fixed budget and the ability to benefit from additional inference tokens.
• H Transfer of Behaviors to Other Domains.: On 200 GPQA questions, behavior-enriched Llama shows more of all four behaviors than its control, including approximately 6.5 versus 0.7 subgoal-setting instances per question, but both score approximately 12%. A separate comparison evaluates Countdown-trained Qwen against base Qwen on 200 questions each from GPQA and MATH, a mathematical problem benchmark: MATH accuracy rises from 38% to 50% with the authors’ prompt format. The paper additionally reports 44% with four examples in the prompt versus 50% with no examples using Qwen’s prompt format; these different prompting conditions limit direct attribution of that second comparison to training alone.
• I Results for Instruction-tuned Models: Models previously trained to follow user instructions exhibit verification and backtracking during subsequent Countdown RL and generate longer responses than their base counterparts. Both instruction-tuned families reach higher task scores in the plotted comparison. The authors suggest that instruction-tuning data may supply useful cognitive behaviors, but do not directly analyze those datasets to establish their contents.

Brief Thoughts

Behavioral priming, length controls, and incorrect-answer priming together provide evidence that the content of initial reasoning demonstrations affects subsequent reward-driven learning. For an ML engineer, the results support evaluating behavioral coverage when selecting initialization data, alongside answer accuracy and token count. The evidence supports the tested interventions more strongly than a universal requirement for all four behaviors: every synthetic behavioral condition includes backtracking, and the continued-pretraining intervention combines selection with rewriting. Behavior counts are insufficient success criteria on their own. The GPQA accuracy parity and lower backward-chaining rater agreement show why task outcomes and measurement reliability must accompany them.