Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs | Summary

This article explains the key points of Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs.

• 2024-12-30 (arXiv)
• Chen, Xingyu, Xu, Jiahao, Liang, Tian, He, Zhiwei, Pang, Jianhui, Yu, Dian, Song, Linfeng, Liu, Qiuzhi, Zhou, Mengfei, Zhang, Zhuosheng, et al.
• Tencent AI Lab, Shanghai Jiao Tong University
• Paper

Read this article in Korean

Summary

• Large language models (LLMs) can overthink: they generate additional solutions that add little correctness or reasoning diversity. Compared with the original QwQ-32B-Preview, training on shortened versions of its own responses reduces average output on MATH500, a high-school competition mathematics benchmark, from 2407.9 to 1330.7 tokens, about 44.7%, while accuracy changes from 93.0% to 92.8% in Table 4.
• The analysis divides chain-of-thought (CoT), an explicit sequence of reasoning steps, into portions that contain an answer and measures their contributions to correctness and strategy diversity. QwQ-32B-Preview and DeepSeek-R1 frequently repeat solutions, particularly on easier mathematics problems, where a smaller fraction of generated tokens is needed to first reach a correct answer.
• The preferred intervention combines Simple Preference Optimization (SimPO), training that favors preferred responses over alternatives, with First-Correct Solutions plus Reflection (FCS+Reflection), which retains a response through its second correct solution. This self-training approach uses the model’s own generated responses and substantially reduces output length, although accuracy on AIME24, the paper’s 2024 mathematics competition test, falls from 46.7% to 43.3%. Several accuracy-preservation claims in the prose therefore require qualification against the numerical results.

1 Introduction

The introduction asks whether scaling test-time compute, allocating additional computation during inference, uses resources efficiently. Its motivating example asks for 2+3: the paper reports that o1-like models, which use extended reasoning and repeated checks, consume 1,953% more tokens on average than conventional models to reach the same answer. The contributions are an analysis of redundant reasoning, two efficiency metrics, and self-training that shortens responses while retaining reflection.

The arithmetic example shows that extended-reasoning models can generate hundreds of tokens for an answer conventional models give briefly. The MATH500 panel illustrates shorter generation at similar accuracy; Table 4 supplies the exact numerical comparison used here.

2 Observing Overthinking Issues

The observational study examines whether additional solutions improve accuracy or introduce different reasoning strategies. Repeated correct answers can increase response length without introducing the first correct answer or a new approach. Figure 2 illustrates this pattern with 13 solutions in one QwQ-32B-Preview response to 2+3.

QwQ-32B-Preview's 901-token response contains 13 solutions despite reaching the correct answer in its first 39-token solution. The example makes repeated answer generation visible and supplies the worked example for both efficiency metrics.

2.1 Solution Distribution of o1-Like Models

The experiments analyze QwQ-32B-Preview and DeepSeek-R1 on ASDIV, an English mathematics word-problem corpus with 2,305 instances and grade levels 1–6; GSM8K, a grade-school mathematics benchmark with 1,319 test problems; and MATH500, which spans seven subjects and difficulty levels 1–5. A solution is a portion of a response that explicitly contains an answer, and Llama-3.3-70B separates responses into these portions.

• Across the test sets, 2–4 solution rounds account for 76%–80% of QwQ-32B-Preview responses and 59%–63% of DeepSeek-R1 responses.
• Average solution counts on ASDIV versus MATH500 are 3.5 versus 3.2 for QwQ-32B-Preview and 4.5 versus 4.3 for DeepSeek-R1.
• Within MATH500, levels 1–2 average 3.7 and 4.6 rounds for the two models, compared with 3.0 and 3.9 at levels 4–5. Token counts nevertheless increase with difficulty, so fewer solution rounds do not imply shorter responses.

Most responses contain multiple answer-containing solutions. DeepSeek-R1 has a larger tail of responses with many solutions, while both models average slightly more solutions on ASDIV than on MATH500.

2.2 Efficiency on Accuracy Improvements

The first correctness distribution records where the first correct solution appears, conditional on a response containing at least one correct solution. Although the prose states that the initial round is correct in more than 92% of cases, Figure 5 shows individual model–dataset values from 89% to 97%; on ASDIV, QwQ-32B-Preview’s first round averages 287 tokens, or 38.7% of its response. Outcome efficiency is \(\xi_O=\frac{1}{N}\sum_{i=1}^{N}\sigma_i\frac{\hat{T}_i}{T_i}\), averaging the response fraction needed to first reach a correct answer and assigning zero to responses with no correct solution.

• N is the number of test instances, T_i is the total response length, and σ_i equals 1 when any solution in the response is correct and 0 otherwise.
• T̂_i counts tokens needed to first arrive at the correct answer when σ_i = 1; otherwise it equals T_i, whose contribution is canceled by σ_i = 0.
• For the 901-token response in Figure 2, the first correct solution takes 39 tokens, giving 39/901 = 4.3% outcome efficiency. By definition, this metric gives no additional credit to verification after a correct answer has appeared.

Among responses containing a correct solution, the first correct solution usually appears at position 1, with displayed values of 89%–97% across the four conditions. Later solutions seldom introduce the first correct answer, although this observation alone does not test the causal value of verification.

2.3 Efficiency on Diverse Thinking

The distinctness ratio measures how often the solution at a given position introduces a reasoning strategy absent from earlier solutions in the same response. GPT-4o clusters solutions by strategy; responses lacking the evaluated solution position are excluded from that position’s denominator. Later solutions often repeat earlier strategies, and the second solution frequently checks the first using the same approach.

• Process efficiency is \(\xi_P=\frac{1}{N}\sum_{i=1}^{N}\frac{D_i}{T_i}\), where D_i sums the tokens in solutions that first introduce a strategy. Correctness is deliberately excluded, so a distinct incorrect solution can contribute positively.
• In Figure 2, the 13 solutions form seven strategy clusters: basic arithmetic, decomposing addition, counting objects, reverse checking, algebra, alternative numeral systems, and context-dependent interpretations.
• The first representatives of those clusters yield 35.8% process efficiency. Figure 6 shows 24%–34% distinctness for solutions at positions ≥4, compared with 33%–47% at position 3.

The first solution is distinct by definition, but later positions often reuse an earlier strategy. Position 2 is less distinct than position 3 in three of the four conditions, consistent with the authors' observation that an immediate second solution often repeats the first as a check.

2.4 Empirical Efficiency Results

Table 1 compares the reasoning models with Llama-3.3-70B-Instruct and Qwen2.5-Math-72B-Instruct. On MATH500, QwQ-32B-Preview and DeepSeek-R1 achieve 93.0% and 96.4% accuracy using 2407.9 and 2704.3 tokens, compared with 86.8% and 593.1 tokens for Qwen2.5-Math-72B-Instruct. Their outcome efficiencies are 52.3% and 51.0%, so the higher accuracy accompanies substantially longer generation.

• On ASDIV, QwQ-32B-Preview and DeepSeek-R1 achieve 96.9% and 97.1% accuracy, but outcome efficiency is only 41.9% and 45.9%. Qwen2.5-Math-72B-Instruct achieves 96.3% accuracy with 213.0 tokens, compared with 741.8 and 845.0 tokens for the reasoning models.
• The conventional models produce one solution per response, making process efficiency 100% and outcome efficiency equal to accuracy under these definitions.
• At MATH500 level 1, outcome efficiency is 46.4% for QwQ-32B-Preview and 42.4% for DeepSeek-R1; at level 5 it rises to 57.5% and 57.1%.

The reasoning models achieve substantially higher MATH500 accuracy than the conventional baselines while producing much longer responses. On ASDIV, the accuracy differences are small but length differences remain large, illustrating the paper's concern about token use on easier problems.

Outcome efficiency rises with MATH500 difficulty for both models, starting below 50% at level 1. Process efficiency varies less consistently, reflecting its separate emphasis on strategy diversity.

3 Mitigating Overthinking Issues

The mitigation study uses PRM12K as the source of training problems and generates responses with QwQ-32B-Preview for self-training. QwQ-32B-Preview is the experimental platform because it is available for post-training, additional training after the base model has been built. Response selection and preference learning teach the model to generate shorter responses.

3.1 Length Preference Optimization

For each training instance, the authors generate 10 responses at temperature 1.0, the sampling setting controlling output randomness, and discard responses that fail to produce a correct answer. The shortest retained responses average 1051.3 tokens and have higher outcome and process efficiency than greedy responses, which choose the most likely next token at each step; the longest retained responses average 2258.7 tokens. Preference methods pair a short positive response with the longest sampled negative response, which preliminary experiments found more effective than using the greedy response as the negative.

• Supervised Fine-Tuning (SFT) learns to reproduce positive examples by minimizing cross-entropy, a loss penalizing low probability on the example’s tokens. It uses the shortest sampled response without a negative pair.
• Direct Preference Optimization (DPO) learns to favor preferred responses over alternatives. Reasoning Preference Optimization (RPO) adds a negative log-likelihood term, a penalty for assigning low probability to the preferred response, to help preserve that response’s probability during training.
• SimPO is the fourth training method evaluated. These experiments use automatically selected length preferences rather than newly collected human preferences.

3.2 Streamlining Responses to Enhance Efficiency

First-Correct Solutions (FCS) retains the response prefix through the first correct solution; FCS+Reflection extends it through a second correct solution to preserve additional checking. Greedily Diverse Solutions (GDS) expands the retained prefix using a heuristic that favors new reasoning strategies. For each strategy, the shortest resulting response is selected from the 10 samples, so different strategies can select different original responses.

• FCS training examples average 1.1 solutions and 681.0 tokens, with 99.5% outcome efficiency and 99.1% process efficiency.
• FCS+Reflection averages 1.9 solutions and 878.7 tokens, with 78.4% outcome efficiency and 82.4% process efficiency.
• GDS averages 1.6 solutions and 856.8 tokens, with 86.8% outcome efficiency and 94.2% process efficiency. These are training-example statistics, not the trained model’s test results.

Truncating through the first correct solution yields the shortest training examples and the highest efficiency scores. Retaining reflection adds tokens and lowers these scores, motivating the subsequent test of whether it preserves more accuracy after training.

3.3 Experimental Results

On MATH500, SimPO trained on shortest responses reduces average length from 2407.9 to 1871.8 tokens, the reported 22.3% reduction, while accuracy changes from 93.0% to 92.4%. Using FCS+Reflection examples produces 1330.7 tokens at 92.8% accuracy, with outcome efficiency rising from 52.3% to 80.0% and process efficiency from 71.2% to 89.5%. FCS alone is shorter at 1016.0 tokens but reaches 91.0% accuracy, showing a tradeoff between aggressive shortening and retaining reflection.

• Among methods using shortest responses, SFT reaches 93.2% accuracy with 2359.5 tokens, DPO reaches 94.0% with 1929.5, and RPO reaches 91.6% with 2015.7. SimPO is selected for its stronger length reduction, although DPO has higher accuracy.
• With SimPO and FCS+Reflection, ASDIV changes from 96.9% to 96.8% accuracy while tokens fall from 741.8 to 381.6; GSM8K improves from 94.8% to 96.0% while tokens fall from 772.8 to 416.6.
• On GPQA, another challenging reasoning test used in the paper, accuracy changes from 59.6% to 59.1% and tokens from 3228.4 to 2085.7. On AIME24, accuracy changes from 46.7% to 43.3% and tokens from 9480.9 to 5154.5.

FCS+Reflection with SimPO substantially reduces generation while keeping MATH500 accuracy close to the base model. The table also limits the accuracy-preservation claim: GPQA declines slightly and AIME24 falls from 46.7% to 43.3%, despite large token reductions.

4 Related Work

The related-work discussion separates methods for increasing inference computation from methods for improving its efficiency. It covers expanding the search over candidate answers, extending reasoning within responses, and controlling how much computation a task receives. The paper positions response selection and self-training within this last problem.

4.1 Scaling Test-Time Compute

Search-based scaling includes self-consistency, generating multiple responses and selecting an answer by majority vote, alongside candidate selection and structured search over possible reasoning paths. A second family extends reasoning through CoT, debate between models, self-correction, self-critique, planning, and reasoning in latent representations, internal model states instead of explicit text. This distinction motivates examining the usefulness of additional reasoning within a response.

4.2 Efficient Thinking

The efficient-thinking discussion covers compressing long reasoning, terminating difficult attempts, prompting with token budgets, allocating computation by estimated difficulty, and stopping generation early. The authors aim to learn efficient generation through post-training without requiring a user-specified token budget. Their experiments evaluate response selection and preference training; a learned allocator that dynamically adjusts computation to each question remains outside the demonstrated results.

5 Conclusion

The conclusion identifies redundant solution generation as a resource-allocation problem and presents outcome and process efficiency as complementary diagnostics. Self-training substantially shortens QwQ-32B-Preview’s responses while retaining much of its benchmark accuracy, with the test-specific changes reported in Table 4. Adaptive computation based on problem complexity and refinement of the efficiency metrics are proposed as future work.

Appendix

• A A.1 Case Overview for Deepseek-R1-Preview provides a second qualitative example: a 936-token response to “What is the answer of 2 plus 3?” contains 11 solutions, although its first solution already obtains 5. Figure 8 shows repeated checking through fingers, a number line, reordered addition, and object-counting examples. This single response illustrates redundancy without estimating its prevalence. A.2 Prompts for Clustering Solutions supplies the GPT-4o clustering criteria used to assess reasoning diversity. Solutions may be distinguished by their method, intermediate steps, assumptions, generality, or substantial differences in complexity, even when they produce the same answer. The prompt requests clusters containing solution identifiers and reasons, so process efficiency depends on this model-based judgment of strategy differences.

Brief Thoughts

The clearest evidence comes from comparing response-selection strategies on the same base model: FCS produces the shortest outputs, while retaining another correct solution recovers some accuracy. The efficiency metrics describe token use but do not directly measure latency or hardware cost. Outcome efficiency also cannot establish that verification after an initially correct answer is causally unnecessary. The mitigation experiments cover one base model, and the supplied paper does not report repeated-run uncertainty, statistical significance, training compute, or measured inference latency. The evidence supports reduced output length more directly than a quantified deployment speedup. The introduction claims a 48.6% MATH500 token reduction, whereas Table 4’s 2407.9-to-1330.7 comparison gives about 44.7%. The prose also states that FCS+Reflection exceeds FCS by 1.4%, while the table gives 92.8% versus 91.0%, a 1.8 percentage-point difference. The prose describes maintained performance on challenging tests, but Table 4 shows a 3.4 percentage-point AIME24 accuracy decrease. Accuracy preservation therefore needs to be assessed for each test rather than assumed across difficulty levels.