Search-o1: Agentic Search-Enhanced Large Reasoning Models | Summary

This article explains the key points of Search-o1: Agentic Search-Enhanced Large Reasoning Models.

• 2025-01-09 (arXiv)
• Li, Xiaoxi, Dong, Guanting, Jin, Jiajie, Zhang, Yuyao, Zhou, Yujia, Zhu, Yutao, Zhang, Peitian, Dou, Zhicheng.
• Renmin University of China, Tsinghua University
• Paper

Read this article in Korean

Summary

• Search-o1 addresses missing knowledge that can derail large reasoning models, language models that generate extended stepwise reasoning before answering. Using QwQ-32B-Preview, it raises Pass@1, the success rate for a single generated answer, on GPQA diamond, a 198-question graduate-level science question-answering benchmark, from 58.1 without retrieval to 63.6 with search and document refinement.
• The framework combines agentic retrieval-augmented generation (RAG), where the model decides when and what external information to retrieve, with Reason-in-Documents, a separate generation process that analyzes retrieved pages against the current query and previous reasoning. Only the refined information returns to the main reasoning sequence; the paper describes an inference workflow without reporting additional model training.
• The clearest gains concern science questions and multi-hop question answering, which requires combining information across several reasoning steps: Search-o1 improves average exact match, the rate of answers matching the reference, on four multi-hop datasets by a reported 29.6% relative to standard RAG with the same model. Benefits vary by task: LiveCodeBench, a programming benchmark, remains at 33.0, matching direct reasoning, while appendix examples expose incorrect refined information and redundant searches.

1 Introduction

The introduction argues that longer reasoning sequences create more opportunities for missing facts to affect subsequent steps. Retrieving documents once for the original question may miss knowledge needed later, while inserting lengthy pages during reasoning can introduce irrelevant material and interrupt the sequence.

• Figure 1 measures expressions of uncertainty in QwQ-32B-Preview outputs on GPQA diamond. Compared with direct reasoning, Search-o1 reduces average occurrences of “perhaps” from 30.4 to 15.8 and “alternatively” from 27.1 to 11.9.
• These counts describe generated language; they do not independently establish factual correctness or whether expressed uncertainty tracks actual errors.
• The proposed response combines repeated, model-directed retrieval with a separate process that selects information relevant to the current reasoning step.

Search-o1 reduces “perhaps” and “alternatively” compared with direct reasoning and standard RAG, but “wait” remains slightly above direct reasoning at 8.2 versus 7.8. The chart measures use of uncertainty expressions rather than factual reliability.

2 Related Work

The related work connects improving reasoning through additional computation during inference with supplying external knowledge through RAG. Earlier retrieval research addresses query processing, document compression, noise removal, refinement, and retrieval decisions; Search-o1 combines on-demand retrieval with the extended reasoning of an existing model.

3 Methodology

Search-o1 repeats a cycle: generate reasoning, issue a search query when needed, analyze retrieved documents separately, insert refined information, and resume reasoning. The following subsections formalize the generated sequences and describe single-question and batched execution.

3.1 Problem Formulation

The task takes an instruction I, a question q, and dynamically retrieved documents D, and produces a reasoning sequence R and final answer a. Equation (1) separates token-by-token reasoning conditioned on earlier reasoning and retrieved documents from answer generation conditioned on the completed reasoning sequence.

• Retrieval remains available during reasoning, allowing later steps to use information unavailable when generation began.
• The formulation describes generation probabilities; it supplies neither a new training objective nor a guarantee of correct reasoning.

3.2 Overview of the Search-o1 Framework

The framework compares direct reasoning, agentic retrieval with retrieved material inserted into the reasoning sequence, and Search-o1 with Reason-in-Documents. In Figure 2’s chemistry example, the three paths produce carbon counts of 10, 14, and 11 respectively, with 11 marked correct.

• Direct reasoning can continue from an incorrect factual assumption.
• Agentic retrieval supplies external information but can return lengthy or redundant material.
• Search-o1 conditions document refinement on the current query, retrieved pages, and preceding reasoning, then returns focused information to the main sequence.

Retrieved documents pass through Reason-in-Documents before information returns to the main reasoning sequence. The chemistry example illustrates the intended benefit, with the correct count of 11 appearing only in the Search-o1 path, but it is not an aggregate evaluation.

3.3 Agentic Retrieval-Augmented Generation Mechanism

The model requests retrieval by generating a query between <|begin_search_query|> and <|end_search_query|>. Detection of the closing marker pauses generation, extracts the query, and invokes Search to retrieve the highest-ranked documents; search results use <|begin_search_result|> and <|end_search_result|>.

• Each query depends on the instruction, original question, and accumulated reasoning, including earlier queries and results.
• Retrieval can recur within one answer, allowing different reasoning steps to request different facts.
• Search decisions come from model generation; the paper defines no separate numerical uncertainty threshold.

3.4 Knowledge Refinement via Reason-in-Documents

Reason-in-Documents uses the original reasoning model in a separate generation process to analyze retrieved pages. It first generates an intermediate document-analysis sequence, then produces refined information conditioned on that analysis, the current query, and preceding reasoning.

• Only the refined output is inserted into the main reasoning sequence, limiting the retrieved material carried forward.
• Equations (4)–(6) distinguish document analysis, refined-information generation, and continued reasoning using previously refined information.
• The module aims to select accurate, relevant information, but the appendix shows that its output can contain incorrect conclusions.

3.5 Search-o1 Inference Process

For a single question, generation alternates between reasoning and search-triggered refinement until the answer is complete. Algorithm 1 maintains unfinished and finished sequence sets so that generation and refinement can operate in batches.

• Generation proceeds until an end-of-sequence token, which marks completion, or <|end_search_query|>.
• Sequences requesting retrieval receive refined information and resume; completed sequences move into the finished set.
• The text describes parallel processing to improve throughput, but the experiments do not quantify throughput, latency, or the additional generation cost of refinement.

4 Experiments

The experiments test whether search and refinement improve challenging reasoning and open-domain question answering (QA), where questions draw on general knowledge. Comparisons include direct reasoning, retrieval once before reasoning, and model-directed retrieval during reasoning.

4.1 Tasks and Datasets

The challenging-reasoning evaluation includes GPQA diamond with 198 questions, MATH500 with 500 mathematics questions, AMC2023 with 40 mathematics competition questions, AIME2024 with 30 mathematics competition questions, and 112 LiveCodeBench programming problems from August to November 2024. A separate comparison with human experts uses the 546-question GPQA extended set.

• The science questions cover physics, chemistry, and biology; LiveCodeBench contains easy, medium, and hard programming problems.
• Single-hop QA, requiring information from one knowledge point, uses Natural Questions (NQ), drawn from Google queries with Wikipedia answers, and TriviaQA, drawn from trivia websites and competitions.
• Multi-hop evaluation uses HotpotQA, requiring reasoning across Wikipedia paragraphs; 2WikiMultihopQA (2WIKI), providing explicit reasoning paths; MuSiQue, containing questions requiring 2–4 hops; and Bamboogle, containing compositional questions that Google answers incorrectly.

4.2 Baselines

Direct-reasoning baselines include Qwen2.5-32B-Instruct, Qwen2.5-Coder-32B-Instruct, QwQ-32B-Preview, Qwen2.5-72B-Instruct, and Llama3.3-70B-Instruct. Reference results for DeepSeek-R1-Lite-Preview, GPT-4o, and o1-preview come from their official releases rather than the authors’ implementations.

• Standard RAG retrieves the top-10 documents for the original question and supplies them before reasoning.
• RAG Agent (RAgent), the model-directed retrieval baseline, initially returns top-10 search snippets and lets the model request full pages from selected URLs.
• RAgent and Search-o1 differ in how retrieved information reaches the reasoning model, including snippet selection and separate document refinement. Their comparison therefore does not isolate refinement under identical document presentation.

4.3 Implementation Details

Search-o1 uses QwQ-32B-Preview with a maximum of 32,768 generated tokens, temperature 0.7, top_p 0.8, top_k 20, and repetition penalty 1.05. Retrieval uses the Bing Web Search API with region US-EN and top-k set to 10, while the Jina Reader API fetches page contents; experiments run on eight NVIDIA A800-80GB graphics processing units (GPUs).

• Temperature controls sampling randomness; top_p and top_k restrict candidate tokens by cumulative probability and count, respectively; the repetition penalty discourages repeated tokens. These generation settings are stated to apply across models.
• Baselines without specialized reasoning training receive Chain-of-Thought (CoT) prompting, an instruction to reason step by step before answering.
• All retrieval-based methods fall back to the direct-reasoning result when they fail to provide a final answer, so the reported scores include this fallback.

4.4 Results on Challenging Reasoning Tasks

Table 1 shows Search-o1 improving GPQA diamond overall Pass@1 from 58.1 for direct QwQ-32B reasoning and 61.6 for RAgent-QwQ-32B to 63.6. Its MATH500 score is 86.4 versus 83.2 and 85.0 respectively, while AMC2023 and AIME2024 match RAgent at 85.0 and 56.7.

• LiveCodeBench overall is 33.0 for Search-o1, 33.0 for direct QwQ-32B, 24.1 for standard RAG, and 26.8 for RAgent.
• Section 4.4 reports average improvements of 4.7% over RAgent and 3.1% over direct QwQ-32B. These labels appear reversed: calculating relative gains from the means of the five displayed benchmark scores gives approximately 3.05% over RAgent and 4.7% over direct reasoning.
• The table supports task-dependent improvements, including no overall coding gain over direct QwQ-32B reasoning.

With the same QwQ-32B model, GPQA overall increases from 58.1 for direct reasoning to 63.6 for Search-o1. Mathematics results improve or tie the agentic baseline, while LiveCodeBench overall remains equal to direct reasoning at 33.0.

Figure 3 varies the number of retrieved documents over 1, 2, 3, 5, and 10 using QwQ-32B-Preview. Overall GPQA performance follows a nondecreasing trend across these settings, and even the one-document setting exceeds direct reasoning and standard RAG with ten documents.

• Individual science subjects fluctuate: physics peaks at five documents, while biology drops after its two-document result before recovering at ten.
• The aggregate trend does not establish monotonic subject-level gains or an efficiency advantage after accounting for document-processing cost.

Overall GPQA performance follows a nondecreasing trend as the tested document count increases, and the one-document setting already exceeds standard RAG with ten documents. Subject-level fluctuations, including the physics peak at five documents, qualify this aggregate pattern.

On GPQA extended, Search-o1 scores 57.9 overall, compared with 51.8 for direct QwQ-32B and 54.6 for standard RAG. Its physics and biology scores, 68.7 and 69.5, exceed the corresponding specialist scores of 57.9 and 68.9, while its chemistry score of 40.7 is substantially below chemists’ 72.6.

• The displayed overall human scores are 39.9 for physicists, 48.9 for chemists, and 37.2 for biologists across the full subject mixture.
• These comparisons apply to this benchmark and subject breakdown; they do not establish general superiority to human experts.

Search-o1 has the highest displayed overall score at 57.9 and exceeds the displayed physics and biology specialist scores in their respective subjects. Chemistry remains a pronounced weakness: 40.7 for Search-o1 versus 72.6 for chemists.

4.5 Results on Open-Domain QA Tasks

Table 3 reports exact match (EM) and F1, an answer-overlap measure. On the four multi-hop datasets, Search-o1 achieves a reported average EM improvement of 29.6% relative to RAG-QwQ-32B and 5.3% relative to RAgent-QwQ-32B.

• Search-o1 reaches EM scores of 45.2 on HotpotQA, 58.0 on 2WIKI, 16.6 on MuSiQue, and 56.0 on Bamboogle.
• RAgent-QwQ-32B scores slightly higher on 2WIKI EM at 58.4, while standard RAG-QwQ-32B scores higher on TriviaQA EM at 65.6 versus Search-o1’s 63.4.
• For QwQ-32B, average single-hop EM changes only slightly, from 47.6 for standard RAG to 47.8 for RAgent. The larger multi-hop gains support the paper’s emphasis on tasks requiring several information-gathering steps.

Search-o1 improves average multi-hop exact match over both retrieval baselines. Exceptions include higher 2WIKI exact match for RAgent-QwQ-32B and higher TriviaQA scores for standard RAG, showing that repeated search and refinement have task-dependent value.

5 Conclusion

The conclusion presents dynamic retrieval and Reason-in-Documents as complementary ways to address missing knowledge during extended reasoning. Benchmark results support improvements in science and multi-hop QA, with smaller or absent gains on some mathematics and coding comparisons. The case studies in Tables 4–6 illustrate correct final answers alongside erroneous refined information, incomplete justification, and redundant searches.

The trace ends with the labeled answer B, or 11 carbon atoms, but its first refined search result incorrectly predicts that the final product should contain 10. The example shows recovery in the final answer while exposing an error within refinement.

The model retrieves a tangent multiple-angle recurrence and later reaches the labeled coefficient value −1. The displayed continuation generalizes from smaller odd values, so the correct final answer does not establish that retrieval was necessary or that the shown justification is complete.

The trace finds evidence that the bands formed in different cities but continues querying, receives conflicting statements about Echosmith's location, and exhausts the search limit before correctly answering “no.” It illustrates redundant retrieval and inconsistent intermediate information.

Appendix

• A Instruction Templates includes A.1 Instructions for Search-o1, comprising Instruction for Search-o1 and Instruction for Reason-in-Documents. The search template defines query and result markers and the placeholder {MAX_SEARCH_LIMIT}; the refinement template supplies {prev_reasoning}, {search_query}, and {document}, and specifies output beginning with “Final Information,” including “No helpful information found.” when appropriate. A.2 Instructions for Standard RAG and its Instruction for Standard RAG box supply the question and documents. A.3 Instructions for RAG Agent and its Instruction for RAG Agent box additionally define <|begin_url|>, <|end_url|>, <|begin_full_page|>, and <|end_full_page|>, with limits expressed as {MAX_SEARCH_LIMIT} and {MAX_URL_FETCH}. A.4 Task-Specific Instructions contains A.4.1 Open-Domain QA Tasks Instruction, A.4.2 Math Tasks Instruction, A.4.3 Multi-choice Tasks Instruction, and A.4.4 Code Tasks Instruction, each with a corresponding instruction box. Open-domain and mathematics answers use \boxed{YOUR_ANSWER}, multiple-choice answers use \boxed{YOUR_CHOICE} with A, B, C, or D, and code uses a Python code block. A.5 Additional Notes states that the templates are supplied as user prompts and that non-reasoning models additionally receive “You should think step by step to solve it.”
• B Case Study presents GPQA, AMC2023, and HotpotQA examples in Tables 4–6. The GPQA trace reaches the labeled answer B, corresponding to 11 carbon atoms, despite an earlier refined result predicting 10; the AMC2023 trace retrieves a tangent recurrence, a formula relating successive angle multiples, and reaches the labeled answer −1. The HotpotQA trace correctly answers “no” but encounters conflicting location information, continues searching after finding different cities, and eventually reaches the search limit.

  Brief Thoughts

Separating document analysis from the main reasoning sequence is the paper’s clearest design contribution. Improvements over RAgent on GPQA and average multi-hop EM support the complete workflow, although differences in document presentation and the absence of comparisons with matched total computation limit attribution to refinement alone.

Reliability and efficiency remain incompletely measured: fewer uncertainty words do not verify reasoning correctness, the GPQA case contains an erroneous refined conclusion, and the HotpotQA case shows redundant searches. Evaluation centers on one Search-o1 model, includes competition sets of only 40 and 30 questions, and reports neither repeated-run variability nor end-to-end cost measurements.