BIG-Bench Extra Hard | Summary

This article explains the key points of BIG-Bench Extra Hard.

• 2025-02-26 (arXiv)
• Kazemi, Mehran, Fatemi, Bahare, Bansal, Hritik, Palowitch, John, Anastasiou, Chrysovalantis, Mehta, Sanket Vaibhav, Jain, Lalit K., Aglietti, Virginia, Jindal, Disha, Chen, Peter, et al.
• Google DeepMind, Google Research, UCLA
• Paper
• Github

Read this article in Korean

Summary

• Existing general reasoning benchmarks have become too easy to distinguish the capabilities of recent large language models (LLMs), systems evaluated here through their text responses. Across BIG-Bench Extra Hard’s 23 tasks, the best evaluated reasoning-specialized model, o3-mini (high), scores 44.8%, compared with 9.8% for the best evaluated general-purpose model, Gemini 2.0 Flash, using adjusted harmonic mean accuracy, an aggregate sensitive to weak performance on individual tasks.
• BIG-Bench Extra Hard (BBEH) replaces every task in BIG-Bench Hard (BBH), an earlier general reasoning benchmark, with a harder counterpart covering similar capabilities. The authors revised tasks until two reference models scored below 70% on each, adding longer reasoning chains, unfamiliar rules learned from the supplied context, and error detection in supplied solutions.
• Improvements vary across reasoning skills: o3-mini (high) gains substantially over GPT-4o on counting, scheduling, and arithmetic, but shows limited or negative gains on humor, sarcasm, and causal judgment, meaning judgments about whether an action caused an outcome. Reference-model selection biases task construction, and some low scores also reflect exhausted output budgets, the available response length, or missing extractable final answers.

1. Introduction

Reasoning evaluation has concentrated on mathematics, science, and coding, leaving logical deduction, temporal and spatial understanding, commonsense, and humor less comprehensively tested. BBH provided this breadth, but reported scores above 90% reduce its ability to distinguish recent models.

BBEH preserves BBH’s coverage through 23 task replacements while increasing difficulty and extending the required skills. The introduction’s headline accuracies of 23.9% for the best general-purpose model and 54.2% for the best reasoning-specialized model are micro averages, meaning accuracy pooled across individual examples; they differ from the adjusted harmonic mean results emphasized in the abstract and Figure 1.

o3-mini (high) separates sharply from the other displayed models, while DeepSeek R1 and Distill-R1-Qwen-32B remain close to several general-purpose models. Reasoning specialization alone therefore does not predict a high aggregate score across these tasks.

2. Related Work

The paper places BBEH alongside benchmarks for individual reasoning domains and evaluations that combine multiple tasks. BBEH provides an open collection of difficult tasks with automatic, deterministic scoring, meaning fixed rules determine correctness, and reports task-level results that support analysis of specific failures.

3. Going Beyond BIG-Bench Hard

BBH combines diverse skills and simple correctness checks, but its answer spaces often permit high chance accuracy: 8/23 tasks have binary labels, and another 5/23 have at most five options. Other weaknesses include exploitable shortcuts, a macro average input length of 700 characters, meaning an equal-weight average across tasks, and problems requiring relatively few reasoning steps.

BBEH retains automatic verification while reducing shortcuts and increasing input-processing and reasoning demands. Figure 2 illustrates table reconstruction, unfamiliar arithmetic operations, repeated object swaps, and error detection in solutions for closing nested brackets.

• Many-hop reasoning requires many intermediate steps; long-range dependency requires using information supplied or derived much earlier in the context.
• Learning on the fly means learning rules from the supplied context; reasoning against a strong prior means following those rules when they conflict with familiar knowledge.
• Long-context reasoning combines information across a long input. Needle-in-a-haystack retrieval locates small pieces of relevant information within it, while distractor handling excludes irrelevant information.
• Inductive reasoning derives patterns from examples, constraint satisfaction finds solutions meeting stated conditions, and compositional understanding solves multiple problems combined into one. Knowledge-intensive reasoning requires substantial domain knowledge.

The samples illustrate distinct demands: reconstructing corrupted tables, applying newly defined arithmetic operations, tracking named swaps, and detecting errors in supplied bracket solutions. The task changes extend beyond lengthening familiar calculations.

4. BIG-Bench Extra Hard

Each of the 23 BBH tasks receives a replacement in a similar reasoning domain, with 200 questions per task except Disambiguation QA, a pronoun-resolution task with 120 questions. Table 1 maps the replacements to their targeted capabilities, including deduction from facts and rules, linguistic inference, tracking changes over time, spatial reasoning, and reasoning with domain knowledge.

The mapping shows how replacements preserve broad reasoning domains while changing the required operations. Simple table queries become reconstruction followed by querying, and familiar adjective ordering becomes inference of unfamiliar rules.

The semi-adversarial construction process uses reference-model performance to guide successive increases in task difficulty. The reference models were Gemini 1.5 Flash and Gemini Thinking Experimental, initially its December 2024 version and subsequently Gemini-2.0-Flash-Thinking-Exp-01-21; iteration continued until both scored below 70% on every task.

• Longer Boolean expressions, which combine true-or-false values through logical operators, initially remained easy because the reference model generated Python to evaluate them. Replacing some True and False operands with textual or mathematical statements obstructed this direct evaluation shortcut.
• The chosen reference models bias the benchmark toward particular failures and complicate fair comparisons between reference and non-reference models.

BBEH Mini provides 460 examples, with 20 randomly selected from each task, for faster and cheaper experiments. The benchmark is publicly available at https://github.com/google-deepmind/bbeh.

5. Results and Analyses

The results compare BBEH with BBH, evaluate models by task and overall, and examine how performance differences vary with model type, input length, and generated response length. Appendix A provides additional task-specific construction details and failure analyses.

5.1. BBEH Analysis

Average inputs are longer for all but two BBEH tasks, with the macro average context length about six times that of BBH. Figure 3 shows substantial variation across tasks, with Shuffled Objects, which tracks assignments through swaps, among the longest inputs.

Input lengths increase across most task pairs but by different amounts. The logarithmic vertical scale makes the particularly large changes for Shuffled Objects and Object Counting easier to compare without compressing shorter tasks.

For a fixed model, Gemini 2.0 Flash, average generated responses are longer on every BBEH task than on its BBH counterpart, with a macro average increase of about seven times. The authors use output length as a proxy for required thinking; Figure 4 measures generated length rather than reasoning correctness.

Using Gemini 2.0 Flash for both datasets holds the generating model fixed. Every replacement elicits longer average responses, supporting increased generation demands while leaving reasoning correctness unmeasured by this plot.

5.2. Model Evaluations

The evaluation covers Llama 3.1, Qwen 2.5, Gemma2, Gemma3, Gemini 2.0, GPT-4o version 2024-11-20, DeepSeek R1, its distilled Qwen 32b model, a model trained to reproduce capabilities of another model, and o3-mini (high). Questions request step-by-step reasoning and an explicitly marked final answer, which is scored using deterministic extraction and normalization rules.

BBEH’s primary metric is an adjusted harmonic mean across task accuracies, which penalizes low task scores more strongly than an arithmetic average. The paper states that it adds 1 to every accuracy number to handle zeros. BBEH Mini instead uses micro average accuracy because each task contains only 20 examples.

Table 2 reports adjusted harmonic mean accuracies of 9.8% for Gemini 2.0 Flash, 6.0% for GPT-4o, 6.8% for DeepSeek R1, and 44.8% for o3-mini (high), against 2.4% for random prediction. DeepSeek R1’s micro average is 34.9%, above every evaluated general-purpose model, but weak individual task scores place its harmonic score below Gemini 2.0 Flash and Flash-Lite.

• The best micro averages on full BBEH are 23.9% for Gemini 2.0 Flash and 54.2% for o3-mini (high), while random prediction scores 8.4%. These values use a different aggregation from Table 2’s overall row.
• On BBEH Mini, Gemini 2.0 Flash scores 27.0%, DeepSeek R1 scores 37.2%, and o3-mini (high) scores 56.7%.
• The reference Thinking model achieves 20.2% adjusted harmonic mean accuracy. Table 2 shows o3-mini (high) above 70% on five tasks and DeepSeek R1 on three; the remaining tabulated models never exceed 70%. The accompanying prose incorrectly gives four tasks for o3-mini (high).

Task rankings reveal different strengths: DeepSeek R1 reaches 75.5% on BoardgameQA, a task requiring deductions from potentially conflicting rules, while o3-mini (high) reaches 68.5% on Temporal Sequences, a meeting-scheduling task, and 56.5% on Object Properties, which tracks changes to an object collection. GPT-4o leads on NYCC, the New Yorker Caption Contest task selecting a humorous caption from a textual cartoon description, with 23.0%, and on SARC Triples, which jointly classifies three replies from the Self-Annotated Corpus for Sarcasm, with 38.5%.

• Some scores fall below chance because models exhaust their effective output length or begin producing repetitive or otherwise degraded responses before an extractable final answer appears. These outcomes reflect both reasoning difficulty and limitations in response generation and answer extraction.

The rows explain the sensitivity of the harmonic aggregate to uneven performance. DeepSeek R1 scores well on BoardgameQA and Object Counting but reaches 0.0% on Object Properties and Temporal Sequences; o3-mini (high) scores 56.5% and 68.5% on those two tasks.

The smaller evaluation retains the leading order among o3-mini (high), DeepSeek R1, and Gemini 2.0 Flash. Its pooled accuracy uses 20 examples per task and differs from the full benchmark’s primary harmonic metric.

5.3. Further Analyses of the Results

Comparing o3-mini (high) with GPT-4o, the largest gains occur on formal tasks involving counting, planning, arithmetic, and algorithms. Figure 5 shows limited or negative gains on several tasks involving commonsense, humor, sarcasm, and causation: Object Counting rises from 6.5% to 90.0%, while SARC Triples falls from 38.5% to 24.0%.

The ordered strip places counting, scheduling, and arithmetic toward the largest improvements, with sarcasm and humor toward the smallest or negative differences. It shows that the advantage of o3-mini (high) over GPT-4o varies substantially across skills.

The comparison of Gemini 2.0 Flash with Flash-Lite shows a related but less clear pattern: many-hop and algorithmic tasks often gain more than humor and causal reasoning. Figure 6 places Word Sorting among the largest improvements; SARC Triples is an exception to the broader pattern, which the authors suggest may relate to its requirement to answer three subquestions jointly.

• For this comparison, the authors replace below-chance task scores with the corresponding random baseline to reduce noise from missing final answers. The model comparison does not isolate parameter count as a controlled experimental variable.

Flash’s advantage over Flash-Lite is largest on tasks including Word Sorting and Zebra Puzzles, while several humor, ambiguity, and causal tasks show little improvement. SARC Triples is a clear exception to the broader pattern of small gains on sarcasm-related reasoning.

Figure 7 associates larger o3-mini (high) gains over GPT-4o with both longer average inputs and longer Gemini 2.0 Flash outputs. Flash’s gains over Flash-Lite also increase with input length, but remain approximately flat with output length; Shuffled Objects is excluded because exhausted output budgets and degraded generation add noise.

• These are associations across different tasks, using output length as a proxy for required thinking. They do not establish that increasing input length or reasoning demand causes the observed performance differences.

Both fitted trends rise with input length, but only the o3-mini (high) versus GPT-4o comparison rises strongly with output length. Separate vertical scales and substantial scatter limit direct visual comparison of the two series; the trends describe associations across tasks.

6. Conclusion

BBEH exposes substantial remaining errors across diverse, automatically scored reasoning tasks. The matched zero-shot comparison, which supplies no worked examples, in Table 10 supports its increased difficulty: Gemini 2.0 Flash’s overall reported accuracy falls from 85.2% on BBH to 23.90% on BBEH. The authors caution that some BBH prompts become ambiguous without the examples used in its usual few-shot evaluation, which includes worked examples.

Holding the evaluated model fixed provides direct evidence that most replacements are harder under zero-shot evaluation. Large declines in scheduling, table processing, and state tracking coexist with improved DisambiguationQA accuracy, which the appendix connects to ambiguity in the original instructions.

Appendix

• A. Detailed Description of the Tasks and Task-Specific Insights from Experiments explains the construction of all 23 tasks and analyzes their individual failure modes. These include incomplete proof search, failures to apply unfamiliar rules, errors in tracking state, difficulty handling ambiguity, and failures to combine multiple answers correctly.
• A.1. BoardgameQA uses facts, potentially conflicting rules, and rule preferences to determine whether a statement is proved, disproved, or unknown, with reasoning depths of 6, 7, and 8. Although unknown labels comprise only one-third of examples, GPT-4o predicts unknown in 82.4% of responses containing a recognized label, compared with 39.7% for DeepSeek R1. The authors interpret excessive unknown predictions as evidence of difficulty finding proofs in the supplied facts and rules.
• A.2. Boolean Expressions combines and, or, and not with operands that include textual and mathematical statements. Each question asks which of five expressions is true, giving 20% chance accuracy and requiring interpretation of the statements before logical evaluation.
• A.3. Buggy Tables requires reconstructing a table from corrupted data and an explanation of the corruption, then answering conditional statistical queries. Corruptions include removed null values, appended extraneous values, and merged rows; queries include count, sum, mean, standard deviation, and median over filtered rows.
• A.4. Causal Understanding combines 142 causal-judgment questions with 58 questions about necessary causes, without which an outcome would not occur, and sufficient causes, which suffice to produce it. The judgment subset represents human disagreement through 45 Ambiguous labels, alongside 48 Yes and 49 No labels; three causal-reasoning experts reviewed the other subset and changed six outputs after resolving ambiguities and errors. Gemini 2.0 Flash scores 45% on causal judgment and 71% on necessary and sufficient causes, making 44 errors among the 45 Ambiguous examples.
• A.5. Disambiguation QA contains 120 longer pronoun-resolution examples with more answer options and explicit allowance for unresolved ambiguity. Ten authors created the examples, separate annotators independently checked them, and adjudication or joint revision resolved disagreements; 25 examples required modification.
• A.6. Dyck Languages, a task involving properly nested brackets, asks for the first error in a supplied chain-of-thought, a sequence of intermediate reasoning steps, or a judgment that none exists. The traces come from PaLM 2 Unicorn at temperature = 0, the stated generation setting. Among cases where a trace contains an error and o3-mini (high) names an incorrect error location, 98.7% miss the first error and identify a later erroneous step, rather than flagging an earlier correct step.
• A.7. Geometric Shapes asks models to identify shapes specified by SVG, a language for drawing shapes, using move and line commands. For o3-mini (high), accuracy is 58% with few line breaks versus 47% with many, and 72% without intersections versus 33% with intersections. Both modifications therefore increase difficulty in the reported subsets.
• A.8. Hyperbaton asks models to infer unfamiliar adjective-order rules from 50–250 sentences containing 1 to 3 adjectives, then select all valid options. The examples disclose only partial orderings; an illustrated o3-mini (high) response correctly infers two partial chains but invents a relation between them, selecting ACDEHI when only D is correct.
• A.9. Linguini samples linguistic reasoning problems from translation, fill blanks, num to text, and text to num categories, with the last two converting between numbers and their textual representations. Questions originally requiring several answers are converted into single-answer questions by randomly selecting one requested component.
• A.10. Movie Recommendation changes next-movie prediction into selection among sets of movies. The correct set contains five movies jointly liked by a large group of users in MovieLens, the source movie-rating dataset, while alternatives mix liked and disliked movies. Models must assess which set is most likely to contain only movies liked by that group.
• A.11. Multi-step Arithmetic introduces unfamiliar, sometimes interdependent operators and their compositions, then asks models to evaluate long expressions. The authors observe that models can correctly explain composition while omitting an operation during execution, including an o3-mini example that drops a final subtraction.
• A.12. NYCC makes caption selection harder by expanding beyond pairwise comparisons and using textual cartoon descriptions supplied by prior work. The task tests agreement with crowdsourced humor ratings; in the illustrated question, the correct caption is number 9.
• A.13. Object Properties requires updating a collection whose objects have color, size, origin, smell, and material properties, then answering a count query. A deliberately unspecified lost object leaves the requested final count unchanged, yet Gemini 2.0 Flash declares such problems unsolvable in 93.5% of cases and GPT-4o in 96.5%, compared with 8% for o3-mini (high).
• A.14. Object Counting asks for a sum or absolute difference between counts of two object classes amid irrelevant information, including other people’s possessions. On the sum subset, Gemini 2.0 Flash over-counts in 50.5% and under-counts in 38.5% of problems, whereas GPT-4o over-counts in 17.5% and under-counts in 66.5%, revealing different directional errors.
• A.15. SARC Triples groups three independent Reddit post/reply examples and requires all sarcasm labels in one answer. Filtering the original test set excludes examples shorter than 100 characters or lacking a reply, leaving 679 examples with a 48.4% positive-label rate; 600 are randomly sampled into 200 triples.
• A.16. Shuffled Objects extends object-swap tracking with canceling operations and named actions referenced later. Models often update every state instead of simplifying canceling swaps; Gemini 2.0 Flash exhausts its output budget on 25% of problems. Another observed failure ignores indirect swaps affecting the requested assignment.
• A.17. Spatial Reasoning adds longer paths and intersecting observations that require backward reasoning, inferring earlier positions from later observations. o3-mini (high) scores 58.8% on forward-only problems and 19.2% on backward problems; DeepSeek R1 scores 48.6% and 3.8%, respectively.
• A.18. SportQA selects 200 multi-hop questions from Level 3, the hardest level, covering six sports. Each example contains a main question and subquestions, and every component must be answered correctly, combining detailed sports knowledge with compositional reasoning.
• A.19. Temporal Sequences supplies weekly calendars and constraints such as time zones, breaks, and attendance limits, then asks for both the longest feasible meeting duration and the number of possibilities. o3-mini (high) scores 68.5% on the joint task, compared with 78% when evaluated only on the longest duration, showing the additional difficulty of the combined requirement.
• A.20. Time Arithmetic composes date and time problems from Test of Time, a temporal reasoning benchmark, excluding scheduling problems already covered elsewhere. A first answer X is transformed using X’ = X + (A2 - A1) and inserted into a later question, where A1 is the original first answer and A2 is a number used in the second question. This requires correct propagation of intermediate values.
• A.21. Web of Lies combines 40 examples from a LiveBench benchmark variant with 160 new examples containing cycles of statements and unknown truth values. o3-mini (high) scores 100% on the former and 28.8% on the latter, while Gemini 2.0 Flash scores 77.5% and 3.8%, demonstrating the greater difficulty of the new variant.
• A.22. Word Sorting combines locating the first error in a supplied sorting trace with sorting under a modified alphabet. An illustrated Gemini 2.0 Flash response states the new alphabet correctly but returns words in ordinary English order, showing failure to apply an explicitly recognized rule.
• A.23. Zebra Puzzles uses verbal clues to infer entity-property assignments in grids of sizes 5x5, 6x6, 7x7, and 8x8. Distracting clues appear in the first three sizes; o3-mini (high) scores 55.4% on 7x7 puzzles with distractors and 74.0% on 8x8 puzzles without them. This suggests distractor sensitivity, although size and distractor presence vary together.
• B. BBEH vs BBH Performance compares Gemini 2.0 Flash on corresponding tasks in a zero-shot setting. Accuracy declines on almost every replacement, including Temporal Sequences from 98.8% to 0.5% and Hyperbaton from 94.8% to 4.5%; DisambiguationQA is an exception, rising from 42.0% to 48.3%. The authors identify sensitivity to ambiguous BBH instructions and show that revised wording can change the selected answer.
• C. Reproducibility describes hosted model access through AI Studio, the OpenAI API, and Together AI, using application programming interfaces to submit model requests, alongside local execution on graphics processing units for Llama and Distill R1 Qwen. Answers are extracted using four recognized variants of the prefix “The answer is:” and checked after limited normalization of case, formatting wrappers, quotes, brackets, parentheses, and comma spacing. The appendix specifies answer extraction and scoring but does not provide decoding settings and output budgets for every evaluated model.

Brief Thoughts

BBEH’s diagnostic value comes from reporting diverse task results alongside an aggregate sensitive to weak performance. DeepSeek R1’s 34.9% micro average and 6.8% adjusted harmonic mean show how strong results on some tasks coexist with severe failures elsewhere.

The experiments establish increased benchmark difficulty and uneven improvements across skills. Their support for broader claims about model size or reasoning specialization is narrower: the comparisons do not isolate training, parameter count, or computation used while generating answers, and some failures involve output exhaustion or missing final answers. Task scores and failure traces are therefore necessary for interpreting the aggregate rankings.