OpenGCUA: Toward a Fully Open Stack for Generalist Computer Use | Summary

This article explains the key points of OpenGCUA: Toward a Fully Open Stack for Generalist Computer Use.

• 2026-10-15 (arXiv)
• Zhou, Hanzhang, Li, Long, Kong, Quyu, Wang, Lei, Tong, Panrong, Chen, Long, Xia, Tianyu, Zhang, Gongjie, Zhang, Xu, Liu, Chen, et al.
• Alibaba Token Hub, Alibaba Group
• Paper

Read this article in Korean

Summary

• OpenGCUA studies how computer-use agents can complete tasks through screen interaction and programmatic access while choosing what to inspect and how to act. On the 82-task offline subset of OSWorld-v2, a computer-use benchmark, its full configuration raises mean partial-credit score by 19.6 points and reduces model calls per task by 53.0% on average across eight models, compared with a graphical user interface (GUI) baseline that batches actions.
• The design combines executable programs with harness-to-model decision transfer: selected observation and task-state decisions move from fixed rules in the agent’s execution system, or harness, to the model. The release includes implementations, prompts, trajectories, and preliminary real-device infrastructure, but no newly trained model. For harness, see Monthly AI Papers: September 2026.
• In selected runs on four macOS tasks, Qwen3.8-Max used 50.2K reported reasoning tokens per task versus 2.6K for GPT-6 Astra and completed three tasks versus four. These preliminary results use one selected run per task and model and do not establish a general model ranking.

1 Introduction

The authors motivate an open research stack by noting that frontier computer-use systems leave key implementation details and execution trajectories unavailable. OpenGCUA releases its harness, prompts, and trajectories at https://github.com/Tongyi-MAI/OpenGCUA, alongside preliminary real-device tools; the project website is https://tongyi-mai.github.io/OpenGCUA/.

The overview separates the benchmark harness from preliminary real-device infrastructure and marks the trained GCUA model as planned.

2 Generalist CUA: Definition and Positioning

The paper characterizes a generalist computer-use agent (GCUA) by the decisions its model makes at each step: what to observe, how to act, and how to organize work. The definition concerns who makes these decisions as well as which interfaces are available.

2.1 Definition

A GCUA can select screenshots or an accessibility tree, a structured representation of interface elements, and combine GUI actions, shell commands, and application programming interface (API) calls in programs. Its working state includes plans, progress, and retained observations; the formal policy jointly selects a view request, a program, and the next working state.

2.2 Relation to Existing Agents

GUI agents use screenshots and fixed interface actions, command-line interface (CLI) agents operate through the shell, and multi-agent hybrids allocate fixed interfaces to sub-agents. The paper treats batched GUI+CLI agents as a restricted case: they choose actions but receive a screenshot each step and follow a fixed context rule.

The comparison identifies which observation, action, and organization decisions are fixed in each agent family and which are assigned to the model.

2.3 Why Now: The Route Flip to Code Execution

A batch fixes its action sequence before execution, whereas a program can use loops, conditions, and checks on intermediate results within one model call. The authors cite a shift toward code-based computer use but do not attribute external latency improvements specifically to code execution.

2.4 Efficiency as a First-Class Metric

The evaluation reports steps, output tokens, reasoning tokens, and estimated cost alongside task success. A step is one model call, reasoning tokens are included in output tokens, and fewer steps alone do not establish lower end-to-end latency.

3 Agent Design: Code Execution and Harness-to-Model Decision Transfer

The study tests two complementary design choices in existing models: programs that coordinate computer operations and model-initiated decisions about observation and task state. Prompts, tool interfaces, and the execution runtime implement these behaviors without training a new model.

3.1 Code Execution over GUI and CLI

The benchmark harness executes a Python program per step using PyAutoGUI and bash(), returning execution output and status. Examples combine interface actions with checks, loops, result-dependent branches, observation preparation, and executable verification; screenshots alone do not expose all interface states as program-readable values.

One program can alternate GUI actions and checks, repeat navigation, or condition an action on a computed result; the email example does not verify delivery.

A contact sheet consolidates 25 visual views, while assertions inspect both intended slide changes and preserved content.

3.2 Harness-to-Model Decision Transfer

The model can request a screenshot(), update a persistent todo record, and state a program’s purpose in an <intent> annotation. The harness still manages context retention, and a claimed_verified task record is the model’s assessment rather than independent verification.

The examples distinguish requesting a new screenshot, recording a task's verification status, and explaining a program's intended change.

3.3 Experiments on Agent Design

The main comparison evaluates eight models on 82 offline OSWorld-v2 v2.1 tasks, excluding 26 tasks requiring external internet access. Mean partial score rises from 45.3 to 64.9 against batched GUI-only execution, while full-task success improves for every model; output tokens decrease for six of eight models.

The table reports per-model score, success, steps, and output tokens; token use rises slightly for GPT-6 Astra and Gemini 3.8 Flash despite fewer steps.

With the same GUI and shell access, code execution outperforms batched GUI+CLI actions in selected runs for three models: GLM-5.3-Flash rises from 55.8 to 63.2, and Qwen3.8-Max falls from 66.1 to 54.9 steps per task. On-demand screenshots raise cache-hit rates, the fraction of input tokens served from cache, from 75.6% to 86.1% and from 61.9% to 76.3% for Qwen3.8-Max and DeepSeek-V4.1-Flash respectively, with higher scores in both selected comparisons.

Across three selected models, the code-execution setting has higher scores and fewer steps than each of the three displayed alternatives.

The full configuration improves scores by 4.4 to 9.1 points over batched GUI+CLI for four selected models, with fewer reported steps.

Individual mechanisms change scores and cache reuse differently: on-demand observation improves cache hits in both selected models, while intent lowers their scores.

A retrospective spending-cap analysis gives Qwen3.8-Flash 60.5 points at approximately $0.38 per task and GPT-6 Astra 75.7 at approximately $15.26. Recorded-time comparisons are exploratory because provider queues, rate limits, concurrency, and tool latency were not controlled across models.

The highest-scoring model changes with budget, while the time curves are exploratory because execution conditions were not controlled across providers.

4 Real-Device Generalist CUA

The real-device work extends the agent design to machines shared with users, using a preliminary harness and cross-device software development kit (SDK). Its evaluation is separate from the controlled OSWorld-v2 benchmark.

4.1 The Gap Between Benchmarks and Real Machines

Real machines have changing windows, existing files and sessions, evolving requests, and potentially simultaneous user activity. Observations and actions must target the intended application, window, or tab, and background observation does not guarantee that background control is supported.

4.2 Cross-Device SDK

The Cross-Device SDK binds operations to explicit resources and provides ui.ax for structured accessibility observations and ui.vision for images. Programs can inspect data locally and use mu.emit to deliver selected information to the model; platform-specific capabilities remain explicit.

Explicit resource handles and platform adapters let a single program coordinate devices while selectively returning observations to the model.

4.3 Real-Device Harness

The real-device harness separates the model, persistent runtime, device SDK, driver, and applications and operating system. It prefers accessibility actions where supported, coordinates foreground input with the user, and preserves state across user interactions.

The layered architecture separates model decisions from runtime execution, resource binding, native delivery, and application state.

On four macOS tasks, selected GPT-6 Astra runs averaged 34.5 steps, 12.9K output tokens, and $18.290 per task and completed 4/4; Qwen3.8-Max averaged 50.8 steps, 73.4K tokens, and $1.494 and completed 3/4. The comparison includes incomplete tasks and uses one selected run per task; the paper describes its case-study trajectories as separate from these runs.

The selected runs show different trade-offs between completion, token consumption, and estimated price; the table covers four tasks and five configurations.

A shopping case illustrates target selection: Qwen3.8-Max captured the foreground desktop instead of the selected Amazon page, while GPT-6 Astra captured the selected page. In a workbook audit, GPT-6 Astra filled 125 required answer cells while Qwen3.8-Max filled 85 and left 40 empty.

The shopping-task examples illustrate differences in observation scope and reported reasoning tokens for screenshot operations.

The saved-file audit links Qwen3.8-Max's fixed row ranges to 40 unfilled cells, while GPT-6 Astra identifies question rows from workbook contents.

5 Roadmap

The roadmap connects model development, execution infrastructure, and real-device evaluation as ongoing project goals.

5.1 From Design Validation to Model Release

The first release validates designs using existing models; a GCUA model that unifies computer and browser use is planned for the next release.

5.2 Research Priorities

The authors prioritize concise reasoning, execution and context efficiency, reliable target selection and recovery, and broader real-device benchmarks, particularly on macOS.

5.3 How to Contribute

The project invites contributions to the harness and SDK, real-device evaluation environments and criteria, and research goals. The roadmap specifies no delivery timeline.

6 Related Work

Related work includes GUI-focused open models, hybrid GUI and programmatic agents, and open infrastructure such as cua and Agent S. The authors position OpenGCUA as an effort to connect a future trained model with real-device infrastructure and execution trajectories.

7 Conclusion

The reported evidence supports studying execution design alongside interface access: the full configuration improves all eight tested models against the batched GUI baseline, and selected ablations show gains from code execution over batched GUI+CLI. The trained model and comprehensive real-device evaluation remain future work.

Appendix

• A Evaluation Details defines task score as mean partial credit on a 0–100 scale and full success as a normalized evaluator score greater than 0.999999. It averages resource use across tasks including failures, retains the latest evaluated attempt, and weights models equally in cross-model summaries.
• B Real-Device Evaluation Details describes workbook reasoning, synthetic-data PDF form filling, research on 22 snack candidates, and a constrained Paris hotel comparison. It supplies translated task instructions and materials, including the workbook’s 125 Level questions and seven worked examples.
• C Prompts documents the benchmark code-execution contract, screenshot requests, task ledger, intent annotation, and full OSWorld-v2 harness prompt. Unlike the persistent real-device interpreter, the benchmark prompt specifies a new Python process for each step.
• D Trajectory Examples presents a GPT-6 Astra slide-background task and a Claude Opus 5 dashboard GIF task, each with a saved evaluator result of 1.0. The supplement describes the exported trajectories as evidence records rather than standalone replay programs.

Brief Thoughts

The comparison with batched GUI+CLI access holds the broad interfaces constant and helps isolate the contribution of execution design; its three-model ablation is narrower than the eight-model main comparison. The four-task real-device study and uncontrolled timing conditions limit claims about general deployment performance or speed.