A3: Android Agent Arena for Mobile GUI Agents with Essential-State Procedural Evaluation | Summary

This article explains the key points of A3: Android Agent Arena for Mobile GUI Agents with Essential-State Procedural Evaluation.

• 2025-01-02 (arXiv)
• Chai, Yuxiang, Tang, Shunye, Xiao, Han, Lin, Weifeng, Li, Hanhao, Zhang, Jiayu, Liu, Liang, Zhao, Pengxiang, Liu, Guangyi, Wang, Guozhi, et al.
• MMLab @ CUHK, Nankai University, EE @ CUHK, vivo AI Lab, SJTU
• Paper
• Github
• Project Page

Read this article in Korean

Summary

• Mobile graphical user interface (GUI) agents, which operate apps through screen interactions, need evaluation that captures partial progress and completion as online content changes. On a held-out Android Agent Arena (A3) test set, the paper’s specialized Android Agent Arena Reward Model (A3RM), an evaluator of task achievement, reaches 98.0% task-level evaluation accuracy versus 95.0% for Gemini-2.5-pro. Its task-level precision, the proportion of predicted successes that are correct, is 96.0% versus 85.7%.
• A3 contains 100 tasks across 20 apps and 20 categories, with success decomposed into essential states: observable semantic milestones required for completion. A multimodal large language model (MLLM), which processes visual and textual evidence, checks these milestones over consecutive screenshot windows without inspecting application internals.
• Under A3RM evaluation, the T3A agent framework powered by Gemini-2.5-pro completes 53.0% of tasks, while InfiGUI-R1, the strongest tested open-source single-model agent, completes 27.0%. These results concern agent execution, distinct from the evaluator accuracy above. Coverage is limited to Android and apps compatible with automated resets, and the learned evaluator remains fallible.

1 Introduction

Static screenshot benchmarks measure individual action predictions but miss accumulated errors and alternative valid interaction sequences. Interactive benchmarks improve realism, yet dependence on instrumented open-source apps, whose code is modified to expose internal states, excludes many proprietary services. A3 addresses this gap with online tasks, semantic progress evaluation, and an interaction toolkit.

The architecture separates device control from agent implementation and subsequent evaluation. This allows recorded interactions to be assessed independently of the agent that generated them.

2 Related Work

The related work reviews agent architectures and benchmark evaluation methods. Its two subsections distinguish how agents control interfaces from how their task achievement is assessed.

2.1 GUI Agents

The paper distinguishes single-model agents, which directly plan and act using a fine-tuned MLLM, from frameworks that add interface perception, interface hierarchies, multi-agent planning, or shortcut guidance. An interface hierarchy represents the screen’s elements and their relationships. A3 evaluates both single-model agents and frameworks.

2.2 GUI Agent Benchmarks

Earlier static benchmarks match predicted screen elements or coordinates, whereas dynamic benchmarks execute tasks in interactive environments. The paper argues that restricted app coverage, unreliable resets, predefined answer matching, and exact internal-state checks limit evaluation of changing online services. Table 1 positions A3 as supporting both state-changing operations and information queries in online apps.

A3 jointly supports dynamic execution, online apps, operations, and information queries in the listed comparison. Its 100 tasks are fewer than several listed dynamic benchmarks, so task count alone does not establish a coverage advantage.

3 Android Agent Arena (A3)

A3 combines curated apps and tasks, essential-state evaluation, model-based judgment, and a software pipeline. It evaluates visually observable task outcomes without requiring access to proprietary application code.

3.1 Apps & Tasks

The authors select one app from each of 20 Google Play Store categories, prioritizing minimal forced login and reliable automated resets; the selected apps average 115 million downloads each. The 100 tasks cover Operation, which requires state-changing actions, and Information Query, which additionally requires extracting information to answer a question.

• Difficulty follows the number of steps N required by a human expert: Easy, N < 7; Medium, 7 ≤ N ≤ 11; Hard, N > 11.
• Tasks follow each app’s core functionality, such as shopping in Amazon and news retrieval in CNN. Reset feasibility constrains which workflows enter the benchmark.

The app collection is paired with two evaluator options. Both commercial judgment and a locally deployable specialized model are supported within the same benchmark framework.

3.2 Essential-State Evaluation

An essential state specifies a required semantic outcome, such as searching a query or selecting its first result, rather than a particular screen-element identifier or pixel location. Essential-State Achieved Rate (ESAR), the fraction of required milestones achieved, is defined as ESAR = N_AES/N_TES, where N_AES counts achieved essential states and N_TES counts all required essential states. Task success requires achieving every associated essential state.

• Two operators independently complete each task using distinct strategies, jointly propose milestones, and submit them to an independent human audit.
• Milestones must be visually verifiable, necessary, and sufficient to cover the task logic. Their definition requires manual work; the authors argue that the definitions remain valid while the core task objective is unchanged.
• A sliding window, a moving segment of consecutive screenshots, traverses the trajectory, meaning the recorded interaction sequence. The evaluator checks which milestones were achieved within each window.

Overlapping screenshot segments provide local evidence of state changes. The Gmail example treats opening the app and displaying starred messages as separate achievements, allowing progress to be recorded before the full task is complete.

3.3 MLLM-Based Judgment

The evaluator uses visual and language understanding because proprietary apps can expose inconsistent interface hierarchies and because outcomes such as selecting the first product require semantic interpretation. The paper contrasts milestone verification with judging an entire trajectory, citing approximately 80% accuracy for the latter approach in prior work. That figure is not a controlled comparison within A3.

• Gemini-2.5-pro serves as a commercial evaluator. Window size and interval, the number of steps advanced between windows, are adjusted to balance accuracy against paid evaluation calls.
• A3RM is fine-tuned from Qwen3-VL-8B for locally deployable essential-state verification. Local deployment removes per-call service fees, while still requiring computation for inference.

3.4 A3 Pipeline Suite

AITK, the Android Interaction Toolkit, manages device interaction, task execution, and human or agent trajectory collection. A translator converts agent outputs to a unified action space, the shared set of supported device actions, and a controller executes them. An agent-independent evaluator processes recorded trajectories; the repository URL is https://github.com/YuxiangChai/AITK.

4 Experiments

The experiments assess evaluator agreement with annotated outcomes and agent performance on the 100-task benchmark. These measure separate capabilities: judging task achievement and executing tasks successfully.

4.1 A3 Reward Model (A3RM)

A3RM uses screenshot windows of size 2 with interval 1. Training data comprise 8241 expert-trajectory samples, including 981 positive and 7260 negative samples, augmented with 1083 negative samples from agent trajectories; successful human demonstrations average three per benchmark task. Positive samples mark transitions where essential states are achieved, while other expert transitions serve as negatives.

• The authors train with dynamic sampling policy optimization (DAPO), their stated reinforcement-learning algorithm, which updates the model using reward feedback. They oversample positive examples fourfold in each epoch, a pass through the training data, to address class imbalance.
• On the held-out A3 test set, A3RM achieves 95.7% essential-state precision, 94.8% recall, 95.3% F1 score, and 96.6% accuracy, compared with Gemini-2.5-pro’s 87.3%, 96.3%, 91.5%, and 89.5%. Recall measures the fraction of actual achievements detected; F1 combines precision and recall, while accuracy measures the fraction of judgments that are correct.
• At the task level, A3RM achieves 96.0% precision, 96.0% recall, 96.0% F1, and 98.0% accuracy. Its essential-state recall is slightly lower than Gemini’s, and the paper explicitly describes A3RM as specialized for the A3 task distribution.

A3RM improves essential-state precision by 8.4 percentage points and task precision by 10.3 points over Gemini-2.5-pro. Essential-state recall decreases from 96.3% to 94.8%, so the improvement does not extend to every metric.

4.2 A3 Benchmark

The benchmark tests seven single-model agents and three configurations of two agent frameworks using official public prompts and inference settings. Under A3RM evaluation, T3A + Gemini-2.5-pro achieves the highest task Success Rate (SR), the proportion of tasks completed, at 53.0%; InfiGUI-R1 leads the single-model agents at 27.0%.

• Qwen2.5-VL achieves 3.0% SR alone, 16.0% with Mobile-Use, and 15.0% with T3A. Keeping T3A while replacing Qwen2.5-VL with Gemini-2.5-pro raises SR to 53.0%, showing that both the framework and underlying model affect the measured performance.
• Qwen3-VL achieves 17.0% SR without a framework. On Hard tasks, UI-Genie, GUI-OWL, Qwen3-VL, and Mobile-Use + Qwen2.5-VL each score 0.0%, while T3A + Gemini-2.5-pro scores 44.0%.
• ESAR exceeds SR for every tested configuration: InfiGUI-R1 reaches 52.1% ESAR versus 27.0% SR, and T3A + Gemini-2.5-pro reaches 66.4% versus 53.0%. The gap shows partial achievement without complete success; the case studies separately document repeated actions and failures to recognize completion.

All configurations achieve higher ESAR than SR, documenting partial progress beyond completed tasks. T3A + Gemini-2.5-pro leads overall, but its 53% success rate leaves 47% of tasks incomplete under A3RM evaluation.

5 Conclusion

The paper contributes an online Android benchmark and a visual evaluation method that records intermediate achievements alongside complete success. The tested agents often make progress without finishing, and the specialized evaluator improves accuracy and precision over Gemini-2.5-pro on the held-out A3 test set.

Limitations

A3 covers Android only; the authors attribute the absence of iOS support to restrictions on virtualization and programmatic control. Authentication and account-reset constraints also exclude categories such as instant messaging. A3RM can still hallucinate achievements, meaning that it may report success unsupported by the recorded interaction.

Appendix

• A Appendix supplies app lists, task examples, evaluator studies, failure cases, and implementation disclosures. A.1 App List and A.2 Tasks & Essential States document the 20 app-category pairs and provide 10 example task decompositions. Examples include selecting CNN’s Science headline and answering its title, filtering Amazon results by material, and configuring an Omio flight search before reporting the cheapest price. A.3 Sliding Window and Interval Study tests window sizes 2–6. Window size 4 with interval 2 achieves 0.95 essential-state accuracy and 0.91 task accuracy at a reported average cost of $0.098, compared with $0.196 for window size 2 and interval 1; the caption defines average cost using the total service cost of three MLLMs over 30 tasks. The authors attribute lower accuracy with larger windows to reduced screenshot legibility, and with non-overlapping small windows to separating actions from their visible results. A.4 Gemini Evaluation reports 58.0% SR for T3A + Gemini-2.5-pro and 29.0% for InfiGUI-R1, compared with 53.0% and 27.0% under A3RM. The authors interpret the generally higher scores as Gemini optimism, consistent with its lower precision in Table 2. The direction is not universal: UI-TARS-1.5 scores 8.0% under Gemini versus 12.0% under A3RM, and GUI-OWL scores 12.0% versus 14.0%. A.5 Case Study, including A.5.1 General Agent Issues and A.5.2 Dynamic Online Apps Specialty, identifies redundant actions, failure to terminate, incorrect visual-element identification, and inaccurate click coordinates. Online-specific difficulties include similar ranked results, advertisements and sponsored content, and dense interfaces with nested filtering menus. These are qualitative observations, with no reported error frequencies. A.6 Generalization Analysis evaluates 25 newly introduced tasks outside the original training distribution. Unchanged A3RM achieves 90.0% essential-state accuracy and 92.0% task accuracy versus Gemini-2.5-pro’s 87.1% and 88.0%, but its essential-state recall is lower, 91.1% versus 93.3%, contradicting the prose claim of superiority across all metrics. A3RM-Continued, additionally trained on data from the new tasks, reaches 95.7% and 96.0% accuracy, so those gains include adaptation to the new tasks. A.7 Risks & License notes that incorrect actions in online, sometimes authenticated apps can affect content. It lists MIT licenses for GUI-OWL and UI-Genie, and Apache licenses for UI-TARS, UI-Venus, InfiGUI-R1, and Qwen2.5/3-VL. A.8 Training Details reports 600 GPU-hour on L40s (48G), using EasyR1 as the training codebase and DAPO as the reinforcement-learning algorithm. A.9 AI Declaration states that Gemini assisted paper writing and served as one evaluation method.

The list documents one selected app per category, spanning news, shopping, navigation, email, and local utilities. It specifies the benchmark’s coverage without establishing that those apps represent every workflow in their categories.

Window size 4 with interval 2 jointly reaches the highest essential-state accuracy, 0.95, and task accuracy, 0.91, at $0.098 under the reported cost calculation. Cheaper settings exist: size 6 with interval 4 costs $0.050 but reduces task accuracy to 0.87.

Gemini evaluation retains T3A + Gemini-2.5-pro as the overall leader and InfiGUI-R1 as the leading single-model agent, while changing absolute scores. UI-TARS-1.5 and GUI-OWL receive lower success scores than under A3RM, showing that evaluator differences are not a uniform upward adjustment.

On the 25 new tasks, unchanged A3RM improves accuracy over Gemini but has lower essential-state recall. A3RM-Continued uses additional data from those tasks, so its stronger results include adaptation rather than measuring unchanged-model transfer alone.

The sequence shows repeated manipulation of the butter ingredient rather than stable recognition of its completed state. It illustrates redundant actions but does not establish how frequently this failure occurs across the benchmark.

The task decompositions require selecting current content or reporting current information, such as a headline or price, rather than matching a permanently fixed answer. They also show the task-specific manual definitions needed to establish which achievements count toward completion.

Brief Thoughts

The combination of semantic milestones and an executable online-app benchmark makes partial progress measurable without proprietary source access. Table 2 supports improved evaluator precision, while Table 3 shows the additional information obtained by reporting intermediate achievement alongside complete success. The 25-task extension provides limited evidence of transfer. Its recall trade-off and the additional training used for A3RM-Continued constrain the generalization claim. The ESAR–SR gap alone does not identify why tasks fail. The qualitative examples document progress-tracking problems and visual grounding errors, where intended actions are mapped to incorrect screen targets, but do not quantify their relative contributions. Agent scores depend on the evaluator, and app selection depends on reset feasibility. These conditions limit how broadly the results describe mobile-agent reliability.