StateAct: Program State, before Pixels, for Long-Horizon Computer-Use Agents | Summary
24 Jul 2026 | Paper Review Computer-Use Agents Program State Agent LearningContents
- Summary
- 1 Introduction
- 2 Related Work
- 3 The State-Grounding Principle
- 4 StateAct
- 5 Experiments
- 6 Discussion
- 7 Conclusion
- Appendix
- Brief Thoughts
This article explains the key points of StateAct: Program State, before Pixels, for Long-Horizon Computer-Use Agents.
- 2026-07-24 (arXiv)
- Yang, Yan, Jian, Xiangru, Luo, Ziyang, Zhao, Zirui, Dai, Yutong, Shi, Ziji, Yan, Hanshu, Liew, Jun Hao, Savarese, Silvio, Li, Junnan.
- Salesforce AI Research
- Paper
Summary
- StateAct is a code-first multi-agent harness for long-horizon computer use that makes persisted program state—files, application backends, and DOM—the main interface rather than screenshots. On OSWorld 2.0, it raises Claude Opus 4.8 from 20.6% to 26.9% binary success and from 54.8% to 61.6% mean partial success, with reported cost falling from approximately $72 to approximately $7.8 per task.
1 Introduction
The paper argues that screenshots are lossy renderings of task-critical state: identical pixels can conceal formulas, hidden rows, backend values, or save locations. StateAct consequently combines state-grounded action, independent state-grounded completion checking, and context management for long dependent trajectories.
The chart reports StateAct at 26.9% binary success, compared with 20.6% for the Claude Opus 4.8 reference harness and a displayed public frontier below 21%. It also identifies the approximately $7.8 per-task cost versus approximately $72 for the reference.
2 Related Work
Related work improves visual grounding, native GUI action, test-time selection, or hybrid GUI/API control. StateAct instead defaults the main agent to code and program-state operations, delegates direct GUI interaction when necessary, and uses a narration-blind terminal gate rather than trajectory-conditioned self-reflection.
3 The State-Grounding Principle
The state-grounding principle treats program state as $s$ and separates a pixel observation channel, $f_{\mathrm{render}}(s)$, from a queryable state channel, $g(s)$. Since rendering is non-injective, screenshots cannot generally recover all relevant program state; state queries can inspect and edit the persisted artifact directly, although render-only goals still require visual interaction.
The example contrasts screenshot-guided spreadsheet interaction with direct workbook inspection and modification through code. It illustrates why state access can recover the stored formula and persist an exact edit when the artifact is state-addressable.
4 StateAct
StateAct comprises a state-first main agent, an independent finish gate, and long-horizon context mechanisms. Persisted state is the common substrate for action and verification, while disposable GUI, web, and general specialists handle delegated subgoals.
The architecture routes the main agent to program state for observation and action, delegates narrow work to fresh-context specialists, and verifies persisted state through a separate finish gate. It also shows the context manager's compaction and checklist functions and the retry path after gate rejection.
4.1 Act on state
The main agent uses persistent bash, a file editor, a read-only image view, a plan checklist, finish, and delegation, but has no live mouse or keyboard control. It discovers storage through prior knowledge and probing, and calls the GUI specialist only for inaccessible or irreducibly visual subgoals: 28 of 108 tasks and 1.1% of main-agent steps.
4.2 Verify on state
After at least three non-finish steps, a finish call spawns a separate gate with only the task instruction and read-only machine access. The gate does not see the main agent’s history, plan, rationale, or expected values; it independently locates the requested artifact and can return a specified structural gap for up to three correction rounds.
The condensed transcript shows the gate reading a persisted calendar file and detecting that only 8 rather than 14 events were saved. It demonstrates a structural failure the narration-blind gate can identify and return for correction.
4.3 Sustain state
To support episodes of up to 200 main-agent turns, StateAct delegates focused work to fresh-context specialists, compacts old context while stripping images, and re-injects an externalized checklist each turn. The reported mean is approximately 57 main-agent turns plus approximately 98 subagent turns, or approximately 155 model turns per task.
5 Experiments
Experiments test whether state-grounding improves long-horizon computer use, which components contribute, and where failures remain. They include OSWorld 2.0 evaluation, component and delegation-depth ablations, backbone transfer, short-horizon comparison, a manual failure audit, and GUI-subagent substitution across five benchmarks.
5.1 Experimental setup
The main benchmark is OSWorld 2.0, comprising 108 long-horizon GUI tasks. Runs use Claude Opus 4.8 with adaptive thinking and a 200-turn main-agent budget, reporting binary success, mean partial credit, and USD cost per task.
5.2 Main result
Against the same Claude Opus 4.8 reference harness, StateAct gains 6.3 binary-success points and 6.8 mean-partial points while reducing output tokens from 224K to 100K and reported cost from approximately $72 to approximately $7.8 per task. It exceeds the displayed Opus-4.7 entry by 8.7 binary points and GPT-5.5 by 13.9 points.
5.3 Ablation
Removing act-on-state produces the largest component decline, reducing mean partial success from 61.6% to 51.3%; removing verification or sustain mechanisms yields 57.5% and 58.7%, respectively. Flat delegation has the strongest listed mean-partial result, but recursion fired on only 7 of 108 tasks and never nested, so the comparison supports the authors’ flat design choice rather than a general claim about hierarchy.
The three selected trajectories illustrate fully code-based document matching, calendar creation with a narrow GUI delegation, and a finish-gate retry on a stale CRM record. Each ends with score 1.0, but the figure is illustrative rather than an aggregate evaluation.
StateAct averages 155 model turns per task, decomposed into 57 main-agent and 98 subagent interior turns. This exceeds the reference Opus 4.8 entry's reported 103 turns but is below several displayed systems, including Sonnet-4.6 at 253 and GLM-5V at 314.
The paired plots locate StateAct at 26.9% binary success and 61.6% mean partial credit for approximately $7.8 per task. The figure shows that cheaper displayed entries, MiniMax M3 and Qwen3.7, have substantially lower accuracy.
The ablation table shows the largest loss when act-on-state is removed: 18.5% binary and 51.3% partial, compared with 26.9% and 61.6% for StateAct. Its delegation panel reports flat delegation as the strongest listed configuration across the aggregate comparison discussed in the text.
5.4 Additional designs
A bash-only configuration reaches 45.9% partial success, below the 54.8% reference, showing that shell access alone does not reproduce the full system’s result. StateAct raises Claude Sonnet 4.6 from 8.3% to 11.1% binary success, while on short-horizon OSWorld-Verified it is close to the reference at 78.4% versus 77.3%.
The table shows that bash-only action underperforms the reference on Opus 4.8, that StateAct modestly improves Sonnet 4.6, and that it is close to the reference on short-horizon OSWorld-Verified. These conditions qualify the result by separating code access, backbone transfer, and task-horizon effects.
6 Discussion
The discussion links the largest margins to machine-checkable capabilities, including multi-item state, cross-source reasoning, and conflict disambiguation. State grounding reduces perception and persistence errors, but it does not independently resolve incorrect interpretation or value reasoning.
6.1 Why State-Grounding Helps: A Failure Analysis
The manual audit assigns 38 of 79 non-perfect tasks to reasoning, 14 to verifier-weak or wrong-path finishes, 18 to capability bottlenecks, 4 to ambiguity, and 5 to other causes. Of the 76 non-perfect tasks that reached the gate, 8 were correctly rejected and 68 were incorrectly passed, indicating that the gate is limited to structural checks and cannot reliably identify shared value-interpretation errors.
The audit partitions 79 non-perfect tasks, with reasoning the largest category at 38 tasks. It separates reasoning, verifier, ambiguity, capability, and other causes, with finer-grained labels such as wrong value, visual chain, and wrong path.
The gate passes 96 tasks and fails 9, while 3 tasks never call finish. Among 76 reached non-perfect tasks, it correctly rejects 8 and incorrectly passes 68; its 8/9 rejection precision therefore coexists with only a 10.5% correct-rejection rate among reached non-perfect tasks.
6.2 Is a strong GUI subagent necessary?
Replacing the GUI specialist with the compact 31B SFR-CUA changes end-to-end mean-partial performance little on OSWorld-Verified, WindowsAgentArena, AndroidWorld, and MobileWorld. On OSWorld 2.0, however, StateAct with SFR-CUA reaches 18.5% binary and 43.2% partial, below the Claude Opus 4.8 GUI-specialist configuration, so GUI-specialist strength still matters for difficult visual fallbacks.
StateAct with SFR-CUA is close to the Claude Opus 4.8 GUI-specialist version on four benchmarks, but trails it substantially on OSWorld 2.0 at 43.2% versus 61.6% mean partial. This supports a conditional claim that a compact GUI fallback can suffice outside the hardest long-horizon desktop setting.
7 Conclusion
The authors conclude that program state can serve as the primary interface for long-horizon computer-use agents while retaining GUI control as a fallback. Their ablations attribute the largest measured contribution to state-based action and observation, while the failure analysis identifies value correctness and reasoning as the remaining bottleneck.
Appendix
- The provided PDF contains references after the conclusion and no separate technical appendix. The cited literature covers computer-use agents, state verification, hybrid action spaces, self-refinement, and the evaluated benchmarks.
Brief Thoughts
The same-backbone comparison makes the harness intervention practically informative, and the bash-only ablation usefully shows that code access alone is insufficient. However, the primary result is based on 108 tasks, and the gate’s 68 incorrect passes among 76 reached non-perfect tasks means structural verification should not be interpreted as dependable semantic or value verification.