GAIA: a benchmark for General AI Assistants | Summary

This article explains the key points of GAIA: a benchmark for General AI Assistants.

• 2023-11-21 (arXiv)
• Mialon, Grégoire, Fourrier, Clémentine, Swift, Craig, Wolf, Thomas, LeCun, Yann, Scialom, Thomas.
• FAIR, Meta, HuggingFace, AutoGPT, GenAI, Meta
• Paper

Read this article in Korean

Summary

• GAIA evaluates general AI assistants on real-world questions that require reasoning, web browsing, file handling, and other tool use. Human validation annotators achieved 92% overall on valid questions, compared with 15% for GPT-4 equipped with plugins selected by humans for each task.
• The benchmark contains 466 human-designed questions with short answers scored by near-exact matching after type-dependent normalization. It releases 166 annotated developer questions and withholds answers to 300 questions for a leaderboard.
• GPT-4 with plugins scored 30.3%, 9.7%, and 0% on Levels 1, 2, and 3, respectively. Human selection and changing plugin availability limit the reproducibility of this comparison.

1 Introduction

The authors argue that professional examinations and open-ended generation present challenges for reliable evaluation. GAIA targets questions that are conceptually simple for people but may require a sequence of actions to produce a verifiable factual answer.

The examples progress from retrieving a clinical-trial enrollment count to calculating from an image and combining information about astronauts. A short answer can depend on several retrieval or reasoning steps.

2 Related work

Prior assistant evaluations often use closed environments, specific application programming interfaces (APIs), or prescribed solution paths. GAIA leaves tools unspecified and includes open-world interactions; the paper describes OpenAGI as a related multi-step benchmark focused on then-current capabilities.

3 GAIA

GAIA combines a benchmark with a question-design and validation process intended to support extensions.

3.1 A convenient yet challenging benchmark for general AI assistants

The 466 text questions cover personal tasks, science, and general knowledge; some include files such as images or spreadsheets. They are designed for zero-shot evaluation, in which the assistant receives no task examples, and seek a single concise answer.

The design emphasizes real-world tasks, interpretable solutions, resistance to memorization, and ease of use. Answers are intended not to appear in plain text in pretraining data, although the authors acknowledge that contamination remains possible.

3.2 Evaluation

A prefix prompt specifies the answer format, and the scoring function compares the final answer with the reference using type-dependent normalization. Answers are typically numbers, short strings, or comma-separated lists; the benchmark does not automatically grade the reasoning path.

GPT-4 reads an attached spreadsheet and sums food-item columns to obtain 89706.00, illustrating how file access and code execution can produce a checkable final answer.

3.3 Composition of GAIA

Based on annotators’ solution paths, 355 questions involve web browsing, 154 coding, 138 multi-modality, or handling data other than text, and 129 reading diverse file types; these categories can overlap. Systems may solve the same question through different capabilities.

The authors loosely define Level 1 as requiring no tools or at most one tool and no more than 5 steps, Level 2 as roughly 5–10 steps combining tools, and Level 3 as potentially long action sequences with unrestricted tools. These are proxies based on human annotations, not hard constraints; the 466 questions comprise 146, 245, and 75 questions at the respective levels.

Web browsing is the most frequently annotated capability, while step and tool counts overlap across levels. The plotted paths describe human annotators and need not match a model's solution.

3.4 Building and extending GAIA

Question creators record an answer, steps, tools, and time, and two additional annotators independently answer each question. Disagreements lead to repair or removal when needed; 68% of crafted questions met the initial validity criterion, and creation, validation, and potential repair took an estimated two annotator-hours per question.

Web sources can change or disappear, so questions may specify a source version and favor durable evidence. The authors also check whether access to the relevant website is restricted by robots.txt.

4 LLMs results on GAIA

The evaluation compares GPT-4, GPT-4 Turbo, AutoGPT with a GPT-4 backend, GPT-4 with plugins, human annotators, and a search-engine baseline based on the first results page. Where direct API access was available, the authors ran models three times and reported averages.

Human scores were 93.9%, 91.8%, and 87.3% across Levels 1–3; GPT-4 with plugins scored 30.3%, 9.7%, and 0%. AutoGPT scored 14.4%, 0.4%, and 0%, with average times of 7.6 and 11.7 minutes on Levels 1 and 2, respectively.

At Level 1, the plugin-equipped configuration has higher scores in several annotated capability groups. Non-tool models can still answer some questions through alternative solution paths or memorized intermediate information.

The plugin result is an oracle-style estimate: humans chose Advanced Data Analysis, a mode with code execution and file reading, or up to three plugins for each task, and plugin availability changed. API timing estimates averaged 20 questions at one point in time and were not intended to compare GPT-4 and GPT-4 Turbo speeds.

5 Discussion

The authors propose updating the benchmark as web evidence disappears or training-data contamination increases. GAIA evaluates the complete assistant system, so a wrong final answer cannot by itself identify whether browsing, vision, or another component failed.

6 Limitations

Final-answer scoring leaves reasoning traces and tool calls unevaluated, and the paper reports limited tool-call logging for fine-grained analysis. Ambiguities may survive validation, and detailed wording needed for a unique answer can make questions less like underspecified user requests.

All questions use standard English and many rely on English web pages, so the benchmark does not establish assistant usefulness for other languages, dialects, or the non-English web.

Appendix

• A Extended related work surveys single-agent prompting, multi-agent collaboration, tool-augmented assistants, and tooling libraries. B Datacard reports that annotators were based in the US and that questions, answers, and metadata were written in mainstream English.
• C Extended description of GAIA defines capability categories from annotators’ tools and notes that categories overlap; it also reports that answer time correlates more clearly with steps than with the number of tools. The most common attached file types shown are xlsx (29), png (18), and pdf (15).
• D Extended description of our question design framework specifies short, stable, unambiguous answers grounded in sources and independently checked by two annotators. Of 623 newly crafted questions, both validators agreed with the original answer on 55%, one agreed on 27%, and neither agreed on 18%; 68% met the validity criterion, which also accepts a disagreement caused by an annotator error.
• D.1 Extended evaluation provides detailed baseline scores and examples: browsing lets GPT-4 retrieve Saint Petersburg for a specimen-location question, while a Rubik’s cube example ends with an incorrect answer. These traces illustrate individual successes and failures rather than aggregate performance.

Brief Thoughts

GAIA’s concise answers make assistant testing practical, and the level-wise results show a substantial gap under the reported 2023 configurations. Scores should be interpreted alongside human-selected plugins, changing web resources, English-only coverage, and the absence of process-level grading.