Atom-Searcher: Enhancing Agentic Deep Research via Fine-Grained Atomic Thought Reward | Summary

Contents

Summary
1 Introduction
2 Atom-Searcher
3 Experiments
4 Related Work
- 4.1 Prompt and SFT-based Agentic Deep Research
- 4.2 RL-based Agentic Deep Research
5 Conclusion
Appendix
Brief Thoughts

This article explains the key points of Atom-Searcher: Enhancing Agentic Deep Research via Fine-Grained Atomic Thought Reward.

2025-08-18 (arXiv)
Deng, Yong, Wang, Guoqing, Ying, Zhenzhe, Wu, Xiaofeng, Lin, Jinzhen, Xiong, Wenwen, Dai, Yuqin, Yang, Shuo, Zhang, Zhanwei, Wang, Qiwen, et al.
Ant Group
Paper
Github

Read this article in Korean

Summary

Training a language model to reason, search the web, and synthesize information is difficult when final-answer feedback gives little guidance about useful intermediate steps. Using Qwen2.5-7B-Instruct as its underlying policy model, Atom-Searcher achieves the highest answer F1, a word-overlap measure of answer correctness, on five of seven sampled question-answering benchmarks and improves mean F1 by approximately 8.5% over DeepResearcher on the four benchmarks designated in-domain, meaning within the evaluation’s training-domain grouping.
The method introduces Atomic Thoughts, tagged functional reasoning units, and Atomic Thought Reward (ATR), feedback obtained by scoring those units with a Reasoning Reward Model (RRM), a reasoning model used as an evaluator. Supervised fine-tuning (SFT), training on example reasoning and tool-use sequences, initializes the policy before reinforcement learning (RL), optimization using rewards, combines ATR with final-answer feedback and linearly reduces ATR’s weight.
Adding RRM feedback without Atomic Thoughts produces mixed results, while the full method improves every benchmark over that ablation. Atom-Searcher also generates longer responses and makes more tool calls than DeepResearcher, but the experiments do not compare accuracy at equal inference cost or vary a controlled computation budget.

Benchmark comparison of Atom-Searcher and selected search baselines across in-domain and out-of-domain tasks.

1 Introduction

The introduction motivates agentic deep research, autonomous reasoning with on-demand search and iterative information synthesis, through limitations of static model knowledge and Retrieval-Augmented Generation (RAG), generation supplemented with retrieved external information. Multi-hop questions, which require several connected reasoning steps, demand adaptable search paths.

Outcome-only rewards can penalize useful intermediate reasoning when the final answer is incorrect or reinforce poor intermediate steps when the answer is correct. The authors describe this mismatch as gradient conflict, conflicting optimization signals for intermediate reasoning and final outcomes, and identify sparse feedback as a second training limitation.
Atomic Thoughts such as <Reflection> and <Verification> provide intermediate evaluation units. Their rewards receive greater weight early in training, with final-answer correctness gaining weight as training advances.

Atomic Thought decomposition within a multi-turn reasoning and search trajectory.

2 Atom-Searcher

Atom-Searcher has two training phases. The first constructs an Atomic Thought instruction dataset and uses SFT to teach structured reasoning; the second scores generated Atomic Thoughts, combines their scores with rule-based outcome rewards, and optimizes the initialized policy with RL.

Two-phase training pipeline for Atom-Searcher with hybrid rewards and loss masking.

2.1 Preliminary

An Atomic Thought is defined as a minimal, functionally coherent reasoning unit whose interactions with other units form a complete reasoning trajectory, the sequence of reasoning, tool calls, retrieved information, and answer. The model learns task-specific decompositions rather than following a fixed inventory of manually prescribed units.

Each turn begins with reasoning inside <think> and </think>, containing tagged units such as <Reflection> and </Reflection>. The policy can issue a JSON request to web_search inside <tool_call> and </tool_call>; retrieved content is appended inside <tool_response> and </tool_response>, and the trajectory eventually produces an answer.
The task is represented as a finite-horizon Markov Decision Process (MDP), a sequential decision model that terminates after completion or an interaction limit. Its state contains retrieved information and previous actions, and its actions are generating an Atomic Thought, invoking search, and answering.
Search transitions concatenate the previous state, the action, and retrieved information; other transitions concatenate the state and action. Group Relative Policy Optimization (GRPO), an algorithm that compares rewards within groups of sampled trajectories, optimizes the policy.

2.2 Incentivizing LLMs to Generate Atomic Thoughts

The Atomic Thought dataset contains 1,000 annotated examples. Construction starts with 10 seed system-prompt templates, each containing two examples with 3 to 10 common Atomic Thoughts, including <plan> and <reflection>.

A teacher model expands the seeds into approximately 1,000 system prompts with different combinations of Atomic Thoughts. These are paired with questions and callable tools such as web_search to produce 1,000 prompts.
Qwen2.5-72B generates complete reasoning trajectories, with majority voting used during sampling to select higher-quality examples. SFT on these trajectories initializes the policy’s knowledge of Atomic Thought structures.

2.3 Reward Modeling

The RRM receives a scoring prompt and a generated trajectory, assigns scores to individual Atomic Thoughts, and aggregates them into a trajectory-level ATR. The paper leaves the aggregation function unspecified: it may be a simple average or a more elaborate weighting strategy.

The schedule is α = 0.5 × (1 − T/T_MAX), where T is the current training step and T_MAX is the maximum number of training steps. It reduces ATR’s weight from 0.5 to 0 over the training schedule.
For correctly formatted trajectories, the reward is \(R=\alpha R_{atom}+(1-\alpha)R_{f1}\); incorrectly formatted trajectories receive −1. The final-answer reward is \(R_{f1}=\frac{2\times IN}{PN+RN}\), where PN and RN are predicted and reference answer word counts, and IN is their intersection word count.
The authors motivate the decay by arguing that useful partial reasoning needs stronger intermediate feedback early in training, whereas continued process feedback could add noise after reasoning becomes better aligned with answer correctness.

2.4 RL Training Framework

GRPO converts each trajectory’s hybrid reward into an advantage, its normalized reward relative to the sampled group, by subtracting the group mean and dividing by its standard deviation. The objective clips the ratio between current and previous policy probabilities to limit updates and applies a Kullback–Leibler divergence penalty, a measure of deviation from the reference policy.

Loss masking excludes retrieved passages from optimization because they come from the environment rather than the policy. The main-text description includes model reasoning and search-query tokens in the loss; Figure 3 also marks answer tokens as subject to loss.
A sliding-window mechanism monitors abrupt reductions in policy entropy, a measure of uncertainty in token predictions. When the reduction exceeds a threshold, training raises the sampling temperature, which increases randomness in generation, and resamples the current batch.

3 Experiments

The experiments evaluate answer F1 across seven open-domain question-answering benchmarks, which require answers without a supplied reference passage. They compare prompt-based and trained search systems and examine component ablations, comparisons that remove parts of the method, and inference behavior.

3.1 Implementation Details

The policy backbone is Qwen2.5-7B-Instruct, training uses the verl framework, and the default RRM is Qwen3-30B-A3B. Each training step samples 32 prompts and generates 16 rollouts, sampled trajectories, per prompt, producing a mini-batch of 512 trajectories for one backpropagation step.

Each rollout allows up to 10 tool calls followed by a final-answer step. Qwen2.5-72B is the teacher used to construct the SFT trajectories.

3.2 Benchmarks

In-domain evaluation uses the development sets of NQ, Natural Questions; TQ, TriviaQA; HotpotQA; and 2Wiki. Out-of-domain evaluation, intended to test transfer beyond the training distribution, uses MuSiQue, Bamboogle, and PopQA, which differ in question format and information distribution.

The authors randomly sample 512 development examples from each of NQ, TQ, HotpotQA, 2Wiki, MuSiQue, and PopQA. Bamboogle evaluation uses all 125 development examples.

3.3 Baselines

The prompt-based baselines include Chain-of-Thought (CoT), explicit intermediate reasoning without external context; CoT+RAG, reasoning with retrieved context; Search-o1, multi-step search using retrieved snippets; and Search-o1-Web, real-time search and webpage browsing.

Search-r1-base and Search-r1-Instruct learn question answering with a Wikipedia retriever and differ in initialization from a base or instruction-tuned model. R1-Searcher uses two stages of outcome-driven RL to learn external search.
DeepResearcher learns an end-to-end research policy in real web environments and is the comparison used for the paper’s reported average gains. Its open-web setting differs from Search-r1’s local Wikipedia retrieval setting.

3.4 Main Result

Table 1 reports Atom-Searcher F1 scores of 44.0 on NQ, 81.8 on TQ, 57.3 on HotpotQA, 66.9 on 2Wiki, 27.6 on MuSiQue, 70.7 on Bamboogle, and 50.3 on PopQA. It leads five benchmarks, ranks second on NQ behind Search-r1-base at 45.4, and ranks second on Bamboogle behind DeepResearcher at 71.0.

3.4.1 Atom-Searcher outperforms baselines on in-domain benchmarks: the paper reports relative gains over the second-best scores of 4.3% on TQ, 2.5% on HotpotQA, and 12.1% on 2Wiki. Its reported 8.5% gain over DeepResearcher across the four in-domain datasets agrees, after rounding, with the relative increase in mean F1. Search-r1-base uses local retrieval with direct access to the relevant Wikipedia corpus, whereas Atom-Searcher searches the open web.
3.4.2 Atom-Searcher demonstrates optimal out-of-domain generalization: the reported relative gains over the second-best scores are 1.8% on MuSiQue and 3.7% on PopQA, while Bamboogle is 0.4% below the best result. The reported average gain of 2.5% over DeepResearcher is not reproduced by Table 1: the relative increase in mean F1 is approximately 1.4%, and the mean of per-dataset relative gains is approximately 1.7%.
Table 2 compares DeepResearcher as Base, Base with RRM feedback as +RRM, and the full Atom-Searcher. The component results are examined in 3.6 Ablation Study.

F1 comparison of prompt-based and training-based methods on seven question-answering benchmarks.

Component ablation comparing Base, +RRM, and the full Atom-Searcher.

3.5 Atom-Searcher Effectively Scales Computation at Test Time

Table 3 reports average response lengths of 565 tokens for Atom-Searcher and 176 for DeepResearcher, average single-think lengths of 143 and 55 tokens, and average tool-call counts of 2.65 and 2.13. These correspond to approximately 3.2 times the response tokens, 2.6 times the think tokens, and 1.24 times the tool calls; the authors report no additional incentive for longer output.

The paper calls this test-time scaling, using more computation during inference. The measurements establish greater inference expenditure, but do not vary a computation budget or compare accuracy at equal cost.

Average response tokens, think tokens, and tool calls for Atom-Searcher and DeepResearcher.

3.6 Ablation Study

Adding RRM feedback to Base without Atomic Thoughts gives mixed changes: NQ rises from 39.6 to 40.1 and 2Wiki from 59.7 to 60.0, while MuSiQue falls from 27.1 to 25.7 and Bamboogle from 71.0 to 70.5. The full Atom-Searcher improves six of seven datasets over Base, with Bamboogle slightly lower at 70.7, and improves every dataset over +RRM.

The authors speculate that Atomic Thoughts provide supervision anchors, explicit functional units that help the evaluator assign intermediate feedback. The ablation supports the combined structured reasoning procedure but does not directly verify this mechanism.
The paper reports improvements over +RRM of 6.1% in-domain and 2.5% out-of-domain. The out-of-domain figure agrees with the relative increase in mean F1 after rounding; the in-domain figure does not, with Table 2 giving approximately 7.9% by that calculation or 8.2% as the mean of per-dataset relative gains.
The ablation does not separately isolate SFT initialization, the decaying reward schedule, or entropy regulation.

3.7 Case Study

The aircraft-engine case study contrasts DeepResearcher’s single search and answer GE F110-GE-129, marked incorrect, with Atom-Searcher’s tagged observations, hypothesis testing, risk analysis, and action planning. Atom-Searcher searches and then browses webpages before answering Pratt & Whitney F100, marked correct.

The token-frequency comparison lists <observation>, <action>, hypothesis, risk, and <risk_analysis> as Atom-Searcher’s five most frequent tokens, versus I, search, need, find, and from for DeepResearcher.
The case and word clouds illustrate structured output and different vocabulary. They do not establish human-like cognition or whether the displayed reasoning faithfully represents the computation that produced the answer.

Aircraft-engine question answered by DeepResearcher and Atom-Searcher, with contrasting reasoning and tool-use traces.

Response token-frequency word clouds for Atom-Searcher and DeepResearcher during testing.

The related-work discussion separates prompt and SFT-based systems from systems trained with RL. It positions Atom-Searcher’s contribution around the granularity and structure of training feedback for search policies.

4.1 Prompt and SFT-based Agentic Deep Research

The paper discusses OpenResearcher, AirRAG, IterDRAG, Plan*RAG, Search-o1, and Open Deep Search as systems using designed workflows for external knowledge acquisition. The authors argue that fixed prompts and interaction patterns restrict adaptability and motivate learned search strategies.

CoRAG is described as using Monte Carlo Tree Search (MCTS), a sampling-based tree search procedure, to select document blocks under budget constraints. The authors identify computational overhead and dependence on supervised signals as limitations.

4.2 RL-based Agentic Deep Research

ReSearch, Search-R1, R1-Searcher, DeepResearcher, WebRL, WebThinker, ZeroSearch, and WebAgent-RL extend outcome-supervised learning to tool-using research agents. Atom-Searcher supplements final-answer feedback with scores tied to explicit functional reasoning units to guide search behavior.

5 Conclusion

The conclusion combines Atomic Thought decomposition, RRM-derived ATR, and a reward schedule that shifts toward final-answer correctness. The experiments show improved benchmark scores and longer inference trajectories, while reduced gradient conflict and accelerated convergence remain explanations without direct measurements in the reported experiments.

Appendix

A Training Details and A.1 Sliding-Window-based Entropy Regulation Mechanism describe SWERM, the Sliding-Window-based dynamic Entropy Regulation Mechanism. It averages token-level policy entropy over the latest k training steps, measures the drop between consecutive window averages, and raises sampling temperature and resamples the current batch when that drop exceeds the threshold τ. Numerical settings for k, τ, and the temperature increase are not provided.
B Prompts Employed in Atom-Searcher accompanies evaluator prompt examples. Figure 6 asks the RRM to interpret each Atomic Thought, define an action-specific scoring rubric, and assign scores from −3 to 5. Figure 7 evaluates the broader thought process through Search Strategy Intelligence, Logical Reasoning Quality, and Answer Accuracy, each from 0 to 5, using the process record, generated answer, and reference answer.

Brief Thoughts

The clearest component evidence is the contrast between +RRM’s mixed changes and the full method’s gains, particularly 2Wiki’s increase from 59.7 to 66.9 over Base. This supports the combined use of structured reasoning and its training procedure, while leaving their individual contributions unresolved. The inference tradeoff is measurable: average response length increases from 176 to 565 tokens and tool calls from 2.13 to 2.65. Latency, monetary cost, and accuracy under a matched computation budget are not reported. Evaluation uses one policy backbone and sampled question-answering development sets, without reported uncertainty intervals or repeated-run variability. The evidence therefore supports this setting more directly than claims about long-form research quality, convergence speed, or human-like cognition. The source identifies the release as arXiv:2508.12800v3, dated 29 Aug 2025, and provides code at https://github.com/antgroup/Research-Venus.

RRM prompt for evaluating individual Atomic Thoughts.