InteractComp: Evaluating Search Agents With Ambiguous Queries | Summary

This article explains the key points of InteractComp: Evaluating Search Agents With Ambiguous Queries.

• 2025-10-28 (arXiv), Proceedings of the 43rd International Conference on Machine Learning, Seoul, South Korea. PMLR 306, 2026.
• Deng, Mingyi, Huang, Lijun, Fan, Yani, Kong, Fanqi, Zhang, Jiayi, Ren, Fashen, Bai, Jinyi, Yang, Fuzhen, Miao, Dayi, Yu, Zhaoyang, et al.
• DeepWisdom, The Hong Kong University of Science and Technology (Guangzhou), Peking University, Renmin University of China, University of California, Los Angeles, AgentUniverse, McGill University, Yale University
• Paper
• Github

Read this article in Korean

Summary

• InteractComp tests whether search agents, language models that gather web information through actions, can ask users for missing details before answering ambiguous requests. Across 17 models, the highest accuracy with search and clarification available is GPT-5’s 13.73%, compared with 67.88% for GPT-5 and a benchmark maximum of 71.50% for OpenAI o3 when complete disambiguating context is supplied with search access.
• The benchmark contains 210 human-curated questions across 9 domains, including 139 English and 71 Chinese instances. Its target-distractor construction pairs a lesser-known correct entity with a popular alternative, exposes their shared attributes, and reserves distinguishing attributes for a simulated user who answers clarification questions.
• Increasing GPT-5’s action budget from 5 to 20 rounds, each allowing one action, raises accuracy from 14.00% to 20.00% on a 50-question subset, while compulsory clarification reaches 40.0%. Free-form user responses raise GPT-5 to 25.71% but leave a substantial gap to complete-context performance, supporting failures both in deciding when to ask and in identifying what to ask.

1. Introduction

Existing search evaluations often provide complete queries, leaving unresolved whether agents can discover missing user intent during retrieval. InteractComp embeds clarification into the search process and uses short, verifiable answers of 1–2 words to assess whether agents obtain the distinguishing information needed to identify the intended entity.

• Figure 1 compares historical performance on BrowseComp, a benchmark for complete search queries, with InteractComp; Figure 2 illustrates alternating search and clarification.
• The authors propose the benchmark as a resource for Reinforcement Learning from Verifiable Rewards (RLVR), training with binary correctness signals from verifiable outcomes. The paper evaluates agents and does not demonstrate such training.

The paired historical plots show rising accuracy on complete-query BrowseComp alongside a much flatter InteractComp trajectory. The comparison motivates evaluating ambiguity resolution separately from conventional search performance, without establishing the cause of the divergence.

The sport example illustrates how hockey-related search results can leave several candidates plausible. Clarification about whether the projectile is a puck or a ball changes the illustrated candidate beliefs before a final answer; the displayed percentages illustrate the workflow rather than report benchmark measurements.

2. Related Work

The paper distinguishes search benchmarks such as BrowseComp and GAIA, an evaluation of general assistants using tools and multimedia, from interaction evaluations such as IN3, which tests implicit user-intention understanding, and Tau-Bench, which tests tool-agent-user interaction. AmbigQA, a dataset of disambiguated question-answer pairs, and information-seeking clarification research address related forms of ambiguity. InteractComp requires agents to choose among searching, asking, and answering within a search task whose final answer can be verified.

3. The INTERACTCOMP Benchmark

The benchmark follows “Easy to verify, Interact to disambiguate”: answers become identifiable once distinguishing context is available, while the initial question admits multiple plausible interpretations. Humans construct the final instances, with search tools supporting research and language models used for validation rather than direct generation of the final benchmark.

3.1. Task Overview

Each instance contains an ambiguous question, hidden context, a correct answer, and a distractor. Agents see only the question and can execute search, interact, or answer; the simulated user sees the context and returns “yes”, “no”, or “I don’t know”.

• The restricted response channel tests specific attribute hypotheses and prevents clarification from becoming unrestricted information retrieval.
• The agent must elicit relevant context details through interaction.

3.2. Data construction and Verification

3.2.1. CONSTRUCTION PROCESS selects a lesser-known target and a popular distractor, separates shared from distinctive attributes, builds the question from shared attributes, and places distinctive attributes in hidden context. Algorithm 1 refines questions when there are at least 4 candidates or the question is already answerable, refines context when the answer is not unique, and repairs instances that fail cross-validation, the independent checks of an instance’s validity.

• 3.2.2. VERIFICATION PROCESS first checks that the target satisfies all stated attributes and that question plus context admits only one valid answer; instances with valid alternatives are reconstructed.
• At least 2 annotators paraphrase each question and check candidate-set stability, testing whether missing information causes ambiguity rather than wording.
• Interaction necessity is checked manually against the first five Google result pages and through 5-round non-interactive trials with GPT-5, GPT-5-mini, and Claude-Sonnet-4. Instances solved by at least two models are revised; this check does not test the models’ ability to obtain context through interaction.

3.3. Data Statistics

The dataset contains 210 instances across 9 domains, with Science & Engineering, Humanities, and Entertainment comprising the largest shares. English accounts for 139 instances (66.19%) and Chinese for 71 (33.81%); questions predominantly contain 40–80 words, while contexts range from about 40 to over 200 words and peak at 60–100 words.

• Figure 3 shows domain composition and question/context length distributions; topic coverage is uneven.

Science & Engineering, Humanities, and Entertainment are the largest topic groups. The histograms show initial questions concentrated around 40–80 words and a broader context-length distribution, consistent with reserving distinguishing information for the simulated user.

4. Experiments

The experiments separate knowledge recall, web retrieval, and interactive clarification through controlled configurations. Subsequent analyses examine action budgets, response restrictions, compulsory questioning, failure categories, and historical model performance.

4.1. Experimental Setup

The base architecture is ReAct, a framework combining reasoning and acting, with Answer-only, Answer+Search, Answer+Interact, and Answer+Search+Interact configurations. Main results evaluate 17 models with search and interaction available and a maximum of 10 rounds per instance. Sampling parameters are temperature=0.6, controlling output randomness, and top_p=0.95, limiting the probability mass considered during sampling, where supported; GPT-4o simulates the user at temperature=1.0.

• Reported metrics are average rounds, interaction rate (IR), the percentage of rounds using interact, accuracy, calibration error (C.E.), the discrepancy between confidence and empirical correctness, and estimated total cost in USD, US dollars.
• Calibration uses 5 confidence bins. Appendix B tests responder sensitivity, human agreement, commercial search systems, and consistency across runs.

4.2. Main Results

Table 1 reports GPT-5 at 13.73% (±2.55) accuracy and DeepSeek-R1 at 13.08% (±0.29), the highest result among open-weight models, whose model weights are available. The parenthesized values are standard deviations across runs. Interaction frequency varies sharply: GPT-4o-mini uses interaction in 73.95% of rounds but achieves 7.13% accuracy, whereas GLM-4.5 interacts in only 0.25% and achieves 7.14%.

• GPT-4o-mini has lower C.E. at 37.44 than many models, including Doubao-1.6 at 84.35. Better confidence calibration does not ensure correct identification, and this cross-model comparison does not establish that questioning causes better calibration.
• The paper emphasizes broad performance bands rather than precise rankings because several models differ by only a few accuracy points.

GPT-5 has the highest reported accuracy at 13.73% (±2.55), while DeepSeek-R1 reaches 13.08% (±0.29). GPT-4o-mini’s 73.95% interaction rate and 7.13% accuracy show that frequent clarification alone does not ensure success; estimated total costs also vary substantially across models.

4.3. Ablation Analysis

Table 2 presents an ablation, a comparison that changes available capabilities, across eight models under answer-only, search-only, ask-only, and with-context conditions. GPT-5 reaches 7.62%, 9.52%, 25.24%, and 67.88%, respectively; OpenAI o3 reaches 71.50% with complete context and C.E. of 7.44. Table A2 specifies search access and a one-round limit for the with-context condition.

• Search-only accuracy ranges from 6.74% to 9.52%, while with-context accuracy ranges from 40.93% to 71.50%. This gap supports missing disambiguating information as a substantial bottleneck; it does not show that reasoning errors are absent.
• Ask-only provides access to hidden attributes through clarification. Its performance does not establish that internal knowledge alone resolves the original ambiguous question, and it does not outperform search-only for every model.

GPT-5 reaches 9.52% with search alone, 25.24% with interaction alone, and 67.88% with complete context. OpenAI o3 reaches the highest complete-context accuracy, 71.50%, supporting information acquisition as a major bottleneck while showing that complete context does not eliminate all errors.

4.4. Scaling Analysis

On a 50-question subset, Table 3 increases available rounds from 5 to 10 to 20. GPT-5’s average clarification count rises from 1.14 to 1.76 to 1.90, with accuracy increasing from 14.00% to 16.00% to 20.00%; Claude-Sonnet-4 reaches only 0.78 clarification rounds at the largest budget.

• Figure 4(a) shows limited gains from larger voluntary action budgets; Figure 4(b) compares performance when clarification is required.
• Between 10 and 20 rounds, GPT-4o-mini remains at 8.00% accuracy, Claude-Sonnet-4 improves from 4.00% to 8.00%, and DeepSeek-V3.1 improves from 8.00% to 10.00%. Larger budgets yield modest, model-dependent gains.

Panel (a) shows limited improvement when agents voluntarily use larger round budgets, while panel (b) shows larger but model-dependent gains with required clarification. The horizontal axes differ: maximum available rounds in panel (a) and interaction rounds in panel (b), so the panels do not establish a matched-budget comparison.

On 50 questions, increasing GPT-5’s budget fourfold raises its average clarification count from 1.14 to 1.90 and accuracy from 14.00% to 20.00%. The other models show smaller or non-monotonic changes, limiting the evidence for budget expansion alone.

4.5. Natural Language Interaction Analysis

The askNL variant, interaction with free-form natural-language responses, removes the three-choice response restriction while retaining search and clarification access. Table 4 reports GPT-5 at 25.71% accuracy and DeepSeek-V3.1 at 17.14%, but GPT-5 remains 42.17 percentage points below its 67.88% with-context result.

• Claude-Sonnet-4 reaches 7.62% and Gemini-2.5-Pro 11.43%, so benefits vary by model.
• Richer responses do not close the complete-context gap, supporting difficulty in eliciting distinguishing attributes as a remaining limitation.

Free-form clarification yields 25.71% accuracy for GPT-5 and 17.14% for DeepSeek-V3.1, while Claude-Sonnet-4 and Gemini-2.5-Pro remain lower. All four results remain below their corresponding complete-context accuracies in Table 2.

4.6. Forced Interaction Analysis

Forced interaction requires a minimum of 2–10 clarification questions before an agent can answer. The main text reports GPT-5 reaching 40% accuracy when compelled to ask 8 questions, compared with 20% under the largest voluntary budget; other models benefit less consistently.

• The intervention supports underuse of available clarification as one failure mechanism. The remaining complete-context gap and model-dependent gains also point to limitations in question selection.
• Appendix C.1 reports non-monotonic forced-interaction results and average interaction counts exceeding the 10-round limit stated in Table A2. The precise accounting of forced rounds is therefore unclear.

4.7. Error Attribution Analysis

Failed action sequences are classified as identification errors, choosing the wrong entity despite search and interaction; overconfidence errors, answering with minimal interaction; ineffective questioning, asking about attributes absent from context or about the wrong entity; and retrievability failures, failing to locate evidence despite searches. Among GPT-5’s failures, 70.5% are classified as overconfidence errors; among Claude-Sonnet-4’s failures, 87.8% are identification errors.

• Gemini-2.5-Pro has 23.4% overconfidence errors and 67.6% identification errors.
• The authors distinguish strategic failure, deciding when to interact, from tactical failure, deciding what to ask. These labels explain different observed failure patterns; minimal interaction is the operational definition of overconfidence in this analysis.

4.8. Longitudinal Study

The 15-month historical comparison reports BrowseComp improvement from approximately 10% to 70%, while InteractComp remains around 6–14%. Figure 1 shows that improvements on complete-query search benchmarks did not coincide with comparable improvements on this ambiguity-resolution evaluation; it does not isolate the cause of the difference.

5. Conclusion

The paper concludes that agents often fail to obtain distinguishing user information despite answering more accurately when that information is supplied. It proposes verifiable search outcomes as training signals for better interaction policies, but reports no training experiments establishing that proposal.

Impact Statement

The Impact Statement identifies information retrieval, decision support, and task-oriented assistants as applications of clarification guided by uncertainty. It also highlights user privacy, interaction fatigue, and excessive optimization of information elicitation, motivating evaluation that balances information gain, user burden, and accuracy.

Appendix

• A. Benchmark and Implementation Details begins with A.1. Instance Example, which pairs Hornussen, a Swiss team striking sport, with the Baseball distractor. Shared sport attributes form the ambiguous question; hidden details such as a plastic puck, wooden defensive boards, and a fan-shaped field distinguish Hornussen.
• A.2. Agent Implementation documents A.2.1. AGENT CONFIGURATIONS, A.2.2. SEARCH IMPLEMENTATION, and A.2.3. ACTION SPACE DESIGN. The actions are search(query), answer(response, confidence), and interact(question); search uses the Google Serper API, an application programming interface wrapping Google Search, to send a POST request to https://google.serper.dev/search with a 30-second timeout and return the top 5 organic results, while the standard agent permits one action per round for up to 10 rounds. A.3. Responder Simulation uses hidden context and interaction history to generate one of three responses, prioritizing diagnostic clarity over conversational realism; A.4. Data Construction Pipeline then traces how the Hornussen example is assembled from shared and distinctive attributes.
• A.5. Evaluation Protocol repeats identical context–question pairs across k = 3 trials. A.6. Calibration Error (C.E.). defines Expected Calibration Error (ECE), the sample-weighted confidence–accuracy gap, as \(\mathrm{ECE}=\sum_{b=1}^{B}\frac{n_b}{N}\left|\mathrm{acc}_b-\mathrm{conf}_b\right|\) with B = 5 equal-width bins over [0,1]. Here n_b is the sample count in bin b, N is the total valid sample count, and acc_b and conf_b are the bin’s accuracy and mean confidence; confidence values are normalized to [0,1], and invalid values are discarded.
• B. Additional Robustness Analyses reports B.1. Responder Sensitivity on 50 questions: replacing GPT-4o with a matched responder, the same model used by the answering agent, changes GPT-5 from 14.00% to 20.00%, while all tested models remain far below complete-context performance. B.2. Human Validation of the Simulated Responder uses 100 records and two annotators, obtaining Krippendorff’s α = 0.707, a multi-rater agreement measure, and Cohen’s κ values of 0.756 and 0.693, pairwise agreement measures for GPT-4o and each annotator. B.3. Commercial DeepResearch Systems reports Doubao Deep Search at 10.0% and OpenAI Deep Research at 6.67% on the full benchmark.
• B.4. Ranking Consistency Across Runs reports Kendall’s τ, a measure of rank agreement, of 0.428, 0.595, and 0.557 for Trial 1 versus Trial 2, Trial 1 versus Trial 3, and Trial 2 versus Trial 3, with corresponding p-values of 0.020, 0.001, and 0.002. B.5. Interaction Utility Analysis measures average belief change per question, post-question correctness gain, and usefulness rate on a 50-question subset. GPT-4o-mini asks 269 questions versus GPT-5’s 113, yet has only 2.60% post-ask correctness gain versus 11.80%, supporting evaluation of clarification utility alongside frequency.
• C. Additional Experimental Results and Case Studies includes C.1. Rounds Constraints Results, where forced questioning produces uneven gains: GPT-5 reaches 40.0%, while GPT-4o-mini remains at 2.0–6.0% across forced settings. C.2. Case Study On Failed Case. describes an agent asking four questions and performing three searches before incorrectly answering “Ice Hockey.” with confidence 75 instead of Hornussen; its questions focus on puck and stick attributes without eliciting the distinctive field or defensive equipment. Code is available at https://github.com/FoundationAgents/InteractComp.

Brief Thoughts

The clearest evidence is the within-model gap between accuracy with complete context and accuracy when agents must obtain that context themselves, reinforced by compulsory-questioning and free-form-response interventions. The results support measuring both whether agents ask and whether their questions distinguish candidates, because interaction count alone does not predict success. Generalization is bounded by 210 deliberately constructed instances, uneven domain coverage, and a simulated user with a restricted response protocol. Human agreement supports the simulator as an approximation to user feedback. Several diagnostic analyses use only 50 questions, and forced-round accounting is inconsistent between the implementation description and detailed results. Broad performance gaps are better supported than exact model rankings or precise forced-budget comparisons.