BOLT: Bootstrap Long Chain-of-Thought in Language Models without Distillation | Summary

This article explains the key points of BOLT: Bootstrap Long Chain-of-Thought in Language Models without Distillation.

• 2025-02-06 (arXiv)
• Pang, Bo, Dong, Hanze, Xu, Jiacheng, Savarese, Silvio, Zhou, Yingbo, Xiong, Caiming.
• Salesforce AI Research
• Paper

Read this article in Korean

Summary

• Bootstrapping LongCoT (BOLT) addresses how to train language models to generate long chain-of-thought (LongCoT), an extended reasoning sequence before a final answer, without distilling reasoning from an existing LongCoT model’s outputs. Compared with the original Meta-Llama-3.1-8B-Instruct model, BOLT raises Arena-Hard-SC, a challenging-query evaluation controlling for response length and markdown style, from 18.3 to 44.1 and MATH500, a competition mathematics evaluation, from 52.9 to 60.0.
• The method prompts Llama-3.1-70B-Instruct through in-context learning, conditioning generation on demonstrations included in the prompt, using a pool of 10 examples. It filters the generated responses, performs supervised finetuning (SFT), training on selected example responses, and applies online Direct Preference Optimization (DPO), training on preferred and rejected responses sampled from the current model. Improvements appear across 7B, 8B, and 70B models, where B denotes billion parameters, and five benchmarks covering general assistance, logic, and mathematics.
• The experiments support the complete pipeline but do not isolate the benefit of longer reasoning from additional training or inference computation. An outcome reward model (ORM), a model scoring the final answer given its query, supplies quality feedback without directly assessing reasoning steps; online training also rewards compliance with the response format. The appendix includes extended responses with unsupported calculations and conclusions.

1. Introduction

The introduction distinguishes LongCoT models, which generate analysis, plans, alternative approaches, reflection, and backtracking, from ShortCoT models, ordinary models with shorter and simpler reasoning. BOLT seeks to develop these behaviors without knowledge distillation from an existing LongCoT model, meaning training on that model’s generated reasoning, while extending evaluation beyond mathematics and coding.

2. Related Work

The related work connects LongCoT training to reinforcement learning, learning from rewards assigned to generated responses, and to methods that select or iteratively improve model outputs. The authors position BOLT against approaches requiring established LongCoT teachers and discuss concurrent DeepSeek work using a 671B-parameter model. This discussion motivates experiments at smaller model sizes but supplies no controlled performance comparison with DeepSeek.

3. BOLT

BOLT combines LongCoT Bootstrapping, LongCoT Supervised Finetuning, and LongCoT Online Training. It represents each example as a query x, generated internal thoughts z, and an external solution y, the user-facing answer. The authors report that earlier prompt-engineering and multi-agent attempts using an actor to generate responses and a judge to evaluate them were unstable or unreliable, but provide no quantitative comparison for those preliminary approaches.

The data flow shows two uses of outcome scoring: selecting bootstrapped examples before supervised training and constructing preference pairs during online training. The scorer assesses the external solution in both cases; supervised training then teaches the selected reasoning sequences without a separate assessment of their correctness.

3.1.1. LongCoT with in-Context Learning

The initial LongCoT behavior comes from prompting a sufficiently capable ShortCoT model with demonstrations. The authors collect 10 examples containing a query, internal thoughts, and an external solution, illustrating problem analysis, planning, branching, and reflection. Figure 2 shows the intended separation between exploratory reasoning and the final response.

The example separates an extended calculation and a checking passage from a shorter external solution. It shows the intended response structure, but does not measure how often checking occurs or corrects errors across generated data.

3.1.2. Query Mixture Curation

Query curation retains first-turn queries from public instruction datasets, scores each query with seven binary criteria, and keeps queries scoring at least 5. The criteria are specificity, domain knowledge, complexity, problem-solving, creativity, technical accuracy, and real-world application; topic tagging and subsampling then adjust the topic distribution. Figure 3 shows that coding and mathematics still dominate, which the authors attribute to task difficulty and the composition of available source datasets.

Computer Science and Coding and Mathematics occupy the largest shares after topic subsampling. The mixture includes other topics but remains concentrated in these two domains.

3.1.3. Response Generation

For each query, Llama-3.1-70B-Instruct generates 8 candidate responses using 3 examples drawn from the pool of 10 demonstrations. Figure 4 shows the prompt template requesting internal thoughts followed by an external solution. The authors report that Llama-3.1-8B-Instruct cannot reliably produce the requested LongCoT responses under this prompting procedure, making a sufficiently capable bootstrapping model a practical requirement.

The template combines demonstration responses with explicit delimiters for the new query, internal thoughts, and external solution. It shows how an existing instruction model is prompted to adopt the desired output format before BOLT training.

3.1.4. Response Filtering

Response filtering first removes outputs that violate the demonstrated format. An ORM then scores external solutions, removes candidates below the 30th percentile of the overall score distribution, and selects the highest-scoring remaining response for each query. Because the authors lack a reward model for internal thoughts, this process selects for answer quality without directly checking the reasoning sequence.

3.2. LongCoT Supervised Finetuning

The filtered query–thought–solution examples train a ShortCoT model to generate internal thoughts before its external response. The target model can differ from the bootstrapping model, allowing the generated dataset to train multiple architectures and sizes. The resulting model initializes online training.

3.3. LongCoT Online Training

The online-training objective maximizes expected reward while penalizing departure from the SFT model through Kullback–Leibler divergence, a measure of the difference between response distributions. Rewards include external-solution scores and a rule-based reward for following the required format. The selected implementation uses DPO with the highest- and lowest-scoring responses among 8 current-model samples as the preferred and rejected pair.

4. Experiments

The experiments compare BOLT with its starting models across model sizes and task domains. They then examine the training trajectory and vary the reward model, starting model, and online learning algorithm to assess how these choices affect the reported scores.

4.1. Experiment Setup

The principal starting models are Mistral-7B-Instruct-v0.3, Meta-Llama-3.1-8B-Instruct, and Meta-Llama-3.1-70B-Instruct; a separate ablation, an experiment varying a pipeline component, starts from Meta-Llama-3.1-8B-base. Evaluations use MT-Bench for multi-turn assistance across eight domains, Arena-Hard-SC and WildBench for challenging real-user queries, ZebraLogic for logic-grid puzzles, and MATH500 for competition mathematics. ZebraLogic reports cell-level accuracy, the proportion of correct grid entries, rather than the percentage of entirely solved puzzles.

SFT uses 220k examples for 4 epochs, complete passes through the dataset, with maximum sequence length 16,384, batch size 128, learning rate 2e-5, a cosine learning-rate schedule, warm-up ratio 0.1, and the AdamW optimizer. Online training uses sampling temperature 1.0, controlling generation randomness, top-p 1.0, retaining the full probability mass for sampling, 8 samples per query, ArmoRM-Llama3-8B as the ORM, DPO regularization coefficient β = 0.1, learning rate 5e-7, a cosine schedule, warm-up ratio 0.1, batch size 128, AdamW, and 3 iterations of 2 epochs with 33k hard prompts per iteration. Reported SFT times are about 6 hours on one 8xH100 accelerator node for 7B/8B and about 5 hours on eight 8xH100 nodes for 70B; online training takes about 14 hours for 7B/8B on one 8xH100 node and about 20 hours for 70B on eight 8xH100 nodes.

4.2. Main Results

Figure 5 reports improvements over each original instruction model on all five benchmarks. Mistral-7B rises from 15.2 to 28.2 on Arena-Hard-SC and from 13.2 to 37.2 on MATH500; Llama-3.1-8B rises from 18.3 to 44.1 and from 52.9 to 60.0, respectively. Llama-3.1-70B rises from 51.8 to 72.2 on Arena-Hard-SC, from 27.1 to 48.1 on ZebraLogic, and from 66.1 to 74.2 on MATH500, while its WildBench gain is smaller, from 48.0 to 50.6.

Every model–benchmark pair improves after BOLT, with substantial variation in gain: Llama-3.1-70B gains 20.4 points on Arena-Hard-SC and 2.6 on WildBench. MT-Bench uses a separate vertical axis from the other evaluations, and ZebraLogic measures cell-level accuracy, so bar heights across benchmarks are not directly comparable.

4.3. Performance Trajectory Over Training

Figure 6 follows Llama-3.1-8B from its initial checkpoint through SFT and three DPO iterations. SFT produces a large initial improvement, and each subsequent DPO iteration improves the plotted Arena-Hard and WildBench results. The trajectory shows gains after both training stages, although the figure labels its first metric Arena-Hard while the main results use Arena-Hard-SC.

The curves show a substantial improvement after SFT followed by further gains through DPO-1, DPO-2, and DPO-3. The figure supports incremental gains during online preference training, but provides no uncertainty intervals and labels its upper metric Arena-Hard.

4.4. Ablation on Reward Models

Replacing ArmoRM-Llama3-8B with Skywork-Reward-Llama-3.1-8B raises Arena-Hard-SC from 44.1 to 51.6 and WildBench from 43.0 to 49.2. Table 1 also reports greater response lengths: 674 to 890 on Arena-Hard-SC and 3354.51 to 4588.15 on WildBench, without specifying the length unit. The authors select ArmoRM-Llama3-8B for the main experiments to favor shorter responses and reduce potential length bias.

Using Skywork-Reward-Llama-3.1-8B produces higher evaluation scores alongside longer responses on both benchmarks. This comparison explains the authors’ choice of ArmoRM-Llama3-8B for shorter responses, while leaving the contribution of response length to the score difference unresolved.

4.5. Ablation on Initial Models

Starting from Meta-Llama-3.1-8B-base yields BOLT scores of 41.3 on Arena-Hard-SC and 39.79 on WildBench, exceeding the unmodified instruction model’s 18.3 and 32.08. Starting BOLT from the instruction model performs better still, reaching 44.1 and 42.96. The target model therefore need not begin as an instruction model, while the pipeline continues to use the 70B instruction model for data generation.

BOLT applied to the base model exceeds the original instruction model on both reported benchmarks, while BOLT applied to the instruction model remains strongest. The comparison supports flexibility in target initialization; training data still comes from an instruction model.

4.6. Ablation on Online Training Algorithms

DPO achieves 44.1 on Arena-Hard-SC and 42.96 on WildBench, compared with 38.3/37.07 for REINFORCE, a method updating generation probabilities from response rewards, 39.7/38.60 for RLOO, a REINFORCE variant using the other sampled responses as a reward baseline, and 37.4/35.14 for Proximal Policy Optimization (PPO), a method constraining policy updates during reward optimization. The authors attribute DPO’s advantage to selecting reward-score extremes rather than training on all sampled responses, which they argue reduces noisy feedback. Table 3 establishes the algorithm ranking in this setting but does not independently test that proposed explanation.

DPO leads all three alternatives on both evaluations, with RLOO the strongest alternative at 39.7 on Arena-Hard-SC and 38.60 on WildBench. These measurements support choosing DPO among the tested implementations, while reduced reward noise remains the authors’ explanation rather than an independently isolated result.

5. Conclusion

The conclusion presents BOLT as evidence that 10 seed demonstrations, data generation from an ordinary instruction model, SFT, and online preference training can develop LongCoT behavior across 7B, 8B, and 70B models. The demonstrated contribution is a training recipe that avoids an established LongCoT teacher while improving the reported downstream evaluations. The paper states an intention to release training data, recipes, and models; the supplied PDF does not establish that those releases have occurred.

Impact Statement

The Impact Statement describes the goal as advancing machine learning and acknowledges potential societal consequences. The authors identify no specific consequence requiring discussion and provide no dedicated safety evaluation or mitigation analysis.

Appendix

• A. Additional Experiment Details lists bootstrapping query sources: ShareGPT, Evol-Instruct, SlimOrca, MathInstruct, Magicoder-Evol-Instruct, GPT4-LLM, OrcaMath, GPTeacher, and UltraInteract. Online-training queries come from UltraFeedback, HelpSteer, OpenOrca, UltraInteract, Capybara, and DIBT-10K, using a selection procedure similar to Section 3.1.2. The DIBT-10K source is https://huggingface.co/datasets/DIBT/10k_prompts_ranked.
• B. Qualitative Examples presents 10 extended responses covering retirement-account advice, entailment, whether one statement follows from another, app reviews, campaign planning, essay writing, code optimization, a programming problem, an infinite series, question generation, and phishing classification, identifying deceptive emails. Example 7 revises explicit subsequence enumeration into a direct bitwise OR computation, combining the set bits of the input integers; Example 8 repeatedly changes its series expressions and ends with 0.084 without a valid supporting derivation. These examples illustrate the response format and revision behavior, while showing that extended reasoning can retain errors.

Brief Thoughts

The strongest evidence is the improvement across three model sizes and five evaluations, supported by further gains along the training trajectory. The generator comparison also identifies a practical constraint: the 8B model fails to generate reliable LongCoT responses through the tested prompting procedure but benefits from training on data produced by the 70B model.

The results leave the causal role and reliability of the generated reasoning unresolved. The paper reports no comparison matching additional training and inference computation while removing LongCoT, and its reward process does not directly assess internal reasoning correctness; Example 8 illustrates the resulting evaluation gap. The main-result figures and ablation tables provide no error bars or repeated-run variability, limiting assessment of uncertainty around individual gains.