Summary explanation: Two-Level Meta-Rubrics for Evaluating Open-Ended Generation: GAMUT, a Benchmark for Factual Completeness
21 Jul 2026 | Paper Review LLM Evaluation Factual Completeness Multimodal Learning Rubric-Based EvaluationContents
- Summary
- 1 Introduction
- 2 Related Work
- 3 A Two-Level Meta-Rubric Framework for Evaluating Open-Ended Generation
- 4 Dataset Construction
- 5 Evaluating Models on Gamut
- 6 Dataset Analysis
- 7 Conclusion
- Appendix
- Brief Thoughts
This article explains the key points of Two-Level Meta-Rubrics for Evaluating Open-Ended Generation: GAMUT, a Benchmark for Factual Completeness.
- 2026-07-21 (arXiv)
- Chen, Xilun, Feizollahi, Zhaleh, Goodwin, Ross, Moon, Seungwhan, Yih, Scott, Donmez, Pinar, Damavandi, Babak, Dong, Luna.
- Meta AI
- Paper
- Github
Summary
- Gamut (Grounded Assessment of Multimodal Factuality) evaluates factual completeness in long-form answers to image-grounded everyday deep-research questions. Its two-level framework authors a structured, evidence-backed meta-rubric and mechanically compiles it into binary checks for LLM judging. The benchmark contains 1,813 human-verified questions in 10 domains; data are at https://huggingface.co/datasets/facebook/GAMUT and code at https://github.com/facebookresearch/GAMUT.
1 Introduction
The paper argues that long-form factuality evaluation has emphasized precision and often treats answer quality as a collection of independent fact checks. Completeness instead may require coverage of an open set of acceptable facts, correct ordering of a process, and differentiated importance, which GAMUT represents before compiling requirements for reliable grading.
The jollof-rice example shows the intended separation of annotation and grading. Required ingredients become individual checks, a flexible ingredient set becomes a coverage threshold plus lower-tier credit, and preparation becomes both presence and order checks; citations and importance tiers connect the two layers.
2 Related Work
Prior work includes precision-oriented decompose-search-verify pipelines, recall-oriented reference-fact evaluation, rubric-based LLM judging, and multimodal knowledge-seeking benchmarks. The authors position GAMUT as addressing the limited ability of flat fact lists to encode grouping, coverage thresholds, ordering, and importance.
2.1 Long-Form Factuality Evaluation
Decompose-search-verify methods break a response into atomic claims and check them against retrieved evidence, while recall methods assess coverage of reference facts. GAMUT targets their shared flat representation, which cannot directly express adequate coverage of a pool of valid answers or a required sequence.
2.2 Rubric-Based Evaluation
Rubric-based LLM evaluation has moved from holistic Likert-style judgments toward explicit criteria and binary checklists, which are intended to reduce variance. GAMUT differs by separating structured rubric authoring from flat checklist grading rather than authoring and grading in the same representation.
2.3 Multimodal Knowledge-Seeking Benchmarks
Multimodal retrieval and agentic-research benchmarks often seek a short verifiable answer or grade a generated report holistically. GAMUT instead scores completeness against a priori reference-fact rubrics for open-ended, image-grounded questions that require multi-source research about everyday subjects.
3 A Two-Level Meta-Rubric Framework for Evaluating Open-Ended Generation
For each question, the framework defines a structured meta-rubric, applies fixed compilation rules to obtain a binary rubric, and asks an LLM judge to assess each check. The division places coverage, ordering, and semantic organization in annotation while limiting the judge to narrow, self-contained requirements.
3.1 Meta-Rubrics
Each meta-rubric item has one importance tier—Answer-Critical, Valuable, or Context—and one of five structures: Simple Knowledge, Strict List, Flexible List, Process, or Relationship. Lists and processes can also include a meta-insight, a synthesis across items that no individual fact captures.
3.2 From Meta-Rubrics to Binary Rubrics
Simple knowledge becomes one check and strict lists become one check per required item. Flexible lists yield self-contained coverage thresholds and lower-tier credit for broader coverage, while processes yield step-presence and sequence checks; relationships and meta-insights are converted to atomic checks. This deterministic mapping makes the compilation inspectable while preserving threshold and ordering requirements.
3.3 Scoring
For each compiled check, the judge returns meets, partially meets, missing, or contradicts. Within each tier, these outcomes receive values 1, λ, 0, and µ; the authors use λ = 0.5 and µ = −2, then combine tier scores with weights wAC = 0.6, wV = 0.3, and wC = 0.1, redistributing the weight of an absent tier. Contradictions are penalized more heavily than omissions.
4 Dataset Construction
Dataset construction first creates everyday deep-research questions grounded in wearable imagery, then builds structured and compiled rubrics for them. Frontier-LLM proposals undergo repeated human review, revision, and evidence checking; 12 entity or question errors were corrected, and examples without sound rubrics were removed, leaving 1,813 entries.
4.1 Question Creation
The authors begin with 1,938 CRAG-MM images and ground-truth entities across 10 everyday domains; roughly 80% are egocentric smart-glasses photographs. Gemini 3.1 Pro proposes questions, two annotators independently review them, and a third adjudicates disagreement. Automatic filtering applies a stranger test for leading questions and a diversification pass, producing 1,843 image-question pairs before later rubric-stage removals.
4.2 Rubric Creation
Gemini 3.1 Pro researches web sources, drafts cited meta-rubrics, and compiles binary rubrics. It then audits supporting snippets and independently researches gaps; a small in-house expert team conducts two rounds of revision, including checks of factual support and compilation correctness. Annotators also inspect cited snippets, removing unsupported evidence and adding missing support.
5 Evaluating Models on Gamut
Fourteen proprietary and open-weight models answer each image-question pair freely, and Gemini 3.1 Pro grades their compiled checks using the fixed scoring procedure. Claude Opus 4.8 and Qwen3-VL 235B provide additional full-dataset judging runs to assess dependence on the primary judge.
5.1 Main Results
Gemini 3.1 Pro achieves the highest reported multimodal Gamut score, 58.7%, while meeting 55.2% of rubric elements. Gemini 3 Flash scores 57.9%, Claude Opus 4.8 scores 53.1%, and the strongest open-weight model, Qwen3-VL 235B, scores 33.0%. Missing requirements dominate failures, while contradiction rates are generally higher for weaker systems and particularly high for the two Qwen3-VL variants.
The table establishes both difficulty and discrimination among the 14 evaluated systems. Gemini 3.1 Pro leads at 58.7%, but its verdict distribution still contains 29.2% missing and 4.1% contradictory checks; the open-weight models range from 33.0% to 5.1%.
The judge comparison reports identical rankings from Gemini 3.1 Pro and Claude Opus 4.8, with per-model scores differing by no more than 1.8 points. Qwen3-VL 235B is more lenient but retains the broad ordering, with swaps only among nearby systems.
5.2 A Text-Only Variant
The text-only variant names the subject explicitly and removes the Answer-Critical identification check that becomes trivial. Seven examples remain image-dependent and are excluded, leaving 1,806 questions. Every evaluated model improves: Gemini 3.1 Pro rises from 58.7% to 73.7%, and the broadly preserved ranking supports the authors’ interpretation that visual identification is a largely shared cost rather than the principal source of separation.
Naming the entity raises every model’s score on the 1,806 shared questions, from 6.8 points for Claude Opus 4.8 to 22.3 points for GPT-4o. The broad preservation of ordering is the evidence for treating visual identification as a largely shared burden.
6 Dataset Analysis
The analysis examines question diversity, rubric scale and evidence grounding, and the prevalence of structure beyond isolated facts. It reports varied domain content, rubrics with substantial depth, widespread structured components, and high citation coverage.
6.1 Question Diversity
Questions have median length 23 words, and the 25 most common opening phrases account for under 60% of the dataset; about one-sixth do not begin with a standard interrogative. Topic profiles also vary by domain: plant questions emphasize habitat and care, local-place questions history and architecture, vehicle questions specifications and reliability, and food questions ingredients and regional variation.
The domain rows exhibit distinct topic mixtures rather than one common question template. Plants & Gardening is led by habitat, Local by history and architecture, Vehicle by specifications, and Food by ingredients; row heights encode differing domain sizes.
6.2 Rubric Statistics
Each question has 15.2 compiled binary checks on average, with median 15: roughly 5.9 Answer-Critical, 5.8 Valuable, and 3.5 Context checks. Rubrics draw on about 10 web snippets per question and cite more than 9,400 distinct pages overall; 97% of meta-rubric items and 94% of compiled checks have supporting citations.
6.3 The Structure Behind the Rubrics
At least one structured component beyond Simple Knowledge appears in 98% of questions. Flexible lists occur in 86%, strict lists in 49%, processes in 17%, and relationships in 6%; flexible-list pools and processes both have median depth four, and 13% of questions include a meta-insight. These counts support the authors’ claim that an independently required fact list would misrepresent many GAMUT questions.
The figure provides the quantitative basis for the meta-rubric design: 98% of questions have a structured component, flexible lists are most prevalent, and most questions use one or two structure types. It also shows median rubric size of 15 checks and citation coverage of 97% for meta-rubric items and 94% for compiled checks.
7 Conclusion
The paper concludes that two-level meta-rubrics separate an expressive representation of complete answers from binary LLM judging. Gamut supplies a multimodal-first and text-only benchmark for factual completeness, but its demonstrated scope is the constructed everyday deep-research setting rather than all forms of open-ended generation evaluation. The accompanying statement reports that Gemini 3.1 Pro was used in data construction under human review and that Claude Opus 4.8 assisted manuscript preparation and implementation.
Appendix
- The appendix supplies the prompts for question generation, selection, rubric generation and refinement, judging, and text-only conversion. It includes complete jollof-rice and Tang sancai examples illustrating all five meta-rubric types and their compilation. For annotation, question workers had to pass scored gold jobs at 95%, while rubric workers passed a 90% quiz, completed two 45-minute trainings, and had 10% of completed jobs audited.
Brief Thoughts
The strongest empirical support for the framework is descriptive: structured components appear in 98% of benchmark questions, and the three-judge experiment yields identical Gemini–Claude rankings and a broadly similar, more lenient Qwen ranking. Important limitations remain: question and rubric proposals originate from Gemini 3.1 Pro before human revision, the reported judge study measures ranking stability rather than direct agreement with human verdicts, and the paper does not present a matched flat-rubric baseline on the same questions.