TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks | Summary

This article explains the key points of TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks.

• 2024-12-18 (arXiv)
• Xu, Frank F., Song, Yufan, Li, Boxuan, Tang, Yuxuan, Jain, Kritanjali, Bao, Mengxue, Wang, Zora Z., Zhou, Xuhui, Guo, Zhitong, Cao, Murong, et al.
• Carnegie Mellon University, Independent, Duke University
• Paper
• Github
• Project Page

Read this article in Korean

Summary

• TheAgentCompany measures whether large language model (LLM) agents, systems that use language models to act through computer tools, can complete workplace tasks requiring browsing, coding, and communication. Across 175 tasks, Gemini-2.5-Pro achieves 30.3% full completion, compared with 26.3% for Claude-3.7-Sonnet, with both using OpenHands 0.28.1, a framework for executing code and browser actions.
• The benchmark simulates a software company using resettable internal websites, an isolated workspace, and model-generated colleagues. Weighted checkpoints, intermediate milestones assigned points and checked by evaluators, support a partial completion score that combines progress with a full-completion bonus. Gemini-2.5-Pro receives a 39.3% score and averages 27.2 model calls and $4.2 per evaluated task under the paper’s cost assumptions.
• Results expose weaknesses in document workflows, complex web interfaces, and communication: Gemini-2.5-Pro completes 12.86% of tasks requiring ownCloud, the document storage and editing service, compared with 41.18% requiring Plane, the project management service. The benchmark evaluates selected tasks within jobs; its lack of measured human performance and comprehensive occupational coverage prevents interpreting these percentages as job automation rates.

1 Introduction

TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks addresses the lack of reproducible evidence about agents’ ability to perform varied professional work. Its simulated company combines software engineering, project management, finance, and administrative tasks that require agents to operate tools and obtain information from colleagues.

• The contribution is a benchmark environment and evaluation framework supported by baseline experiments.
• The release provides the website https://the-agent-company.com, code at https://github.com/TheAgentCompany/TheAgentCompany, and evaluations at https://github.com/TheAgentCompany/experiments.

2 Benchmark Desiderata and Comparison to Other Benchmarks

The benchmark specifies five design requirements: multiple work-related task categories, interaction with colleagues, long-horizon tasks requiring many consecutive steps, varied computer interfaces, and self-hosted reproducibility, meaning that researchers can run the environment on their own infrastructure. Intermediate checkpoints record useful progress even when the overall task remains incomplete.

• The comparison with prior benchmarks concerns coverage and environment design. Scores from benchmarks with different tasks and success criteria are not directly interchangeable.
• The combined browser, terminal, program execution, and messaging interfaces support workflows spanning several applications.

3 TheAgentCompany Environment Setup

The local workspace runs in a sandboxed Docker environment, an isolated execution environment containing a browser, code editor, and Linux terminal. The intranet, the company’s internal websites, uses GitLab for code and technical wiki pages, ownCloud for documents and file sharing, Plane for issue and project management, and RocketChat for internal messaging.

• Services can be reset and contain a mixture of real software project data and manually curated company data.
• Sotopia, a platform for creating model-generated human characters, supplies simulated colleagues with names, roles, responsibilities, and project affiliations. Agents contact them through direct messages or channels to obtain information absent from the initial task.
• All simulated colleagues use Claude-3-5-Sonnet-20241022 across experiments, keeping their underlying model fixed while the evaluated agent changes.

4 Task Structure

Each task contains an English task intent describing the requested work, weighted checkpoints, checkpoint evaluators, and initialization and finalization code. Checkpoints assess action completion, output accuracy, and collaboration by examining workspace files, intranet state, conversations, or action histories.

• Most evaluators are deterministic Python functions that apply programmed checks. For example, a software engineering task checks repository cloning, successful building, and server availability.
• Some unstructured outputs or interactions use language model evaluation with predefined scoring rubrics or reference outputs. These evaluators also use Claude-3-5-Sonnet-20241022.

4.1 Evaluation Metrics

Full completion uses S_full = 1 if all checkpoints pass, and 0 otherwise. Partial completion uses S_partial = 0.5 × Result/Total + 0.5 × S_full, where Result is the awarded checkpoint points and Total is the available points; passing every checkpoint adds a full-completion bonus worth half the total score.

• Steps count model calls during execution. They do not measure elapsed time or individual interface operations.
• Cost per instance equals (Prompt token count × Prompt token cost) + (Completion token count × Completion token cost), assuming no prompt caching, or reuse of previously processed prompt content at a reduced charge. This estimates the expense of querying the underlying model under the stated pricing assumptions.

4.2 Workflow

Tasks follow three stages: initialization prepares the workspace, execution performs subtasks and any required communication, and finalization produces and submits the output for evaluation. A task can involve dependent actions across several applications.

5 Task Creation

Task creation combines occupational reference data, task selection, and manual implementation. The process defines the requested work and builds the environment state and evaluators needed to assess it.

5.1 Choosing Task Categories

The authors use O*NET release 29.1, a US Department of Labor occupational database, to identify job categories with substantial employment. They multiply employment counts by median salaries to estimate aggregate occupational value, then select a software company setting that supports several digital job categories.

• Categories requiring extensive physical labor are excluded because the benchmark evaluates computer-based work.
• This selection procedure motivates the setting but does not produce a representative sample of all work or all tasks within an occupation.

5.2 Choosing Tasks

Individual tasks draw on O*NET task lists, experienced coauthors’ knowledge, and brainstorming with language models. Selection emphasizes concrete goals and well-defined success criteria, which support automated evaluation but leave substantial occupational responsibilities outside the benchmark.

• The paper explicitly cautions against using these results alone to judge whether an entire job can be automated.

5.3 Manual Task Curation

For each task, authors define the intent and checkpoints, import or create supporting data, configure initial workspace state, and implement evaluators. Twenty contributors worked for over 2 months, using approximately 3,000 person-hours; some complex tasks required more than 10 hours each to design, implement, test, and verify.

• Quality checks include screenshot evidence that successful execution earns full credit, lead-author code review, and a final manual review of task data, evaluator behavior, and scoring.
• A person who did not curate the tasks checks checkpoint weights for consistency and relative significance.

6 Baseline Agents

The main baseline is OpenHands’ CodeAct Agent with Browsing, an agent that executes code and browser actions. Experiments also include OWL-RolePlay, a framework in which multiple agents collaborate on task execution.

• OpenHands versions 0.14.2 and 0.28.1 are used, with the newer version accommodating newer models.
• OWL-RolePlay uses its recommended model configuration from https://github.com/camel-ai/owl/tree/gaia58.18. The reported configuration combines GPT-4o and o3-mini.

7 Experimental Results

Experiments evaluate closed models and open-weight models, whose learned parameters are available, on the full 175-task benchmark. Simulated colleagues and model-based evaluators remain fixed to Claude-3-5-Sonnet-20241022, so their underlying model belongs to the evaluation environment rather than the tested agent configuration.

• The evaluation primarily varies the model within OpenHands and adds one OWL-RolePlay configuration. Some comparisons also change the OpenHands version or agent framework, limiting attribution of differences to the model alone.

7.1 Result Overview

Gemini-2.5-Pro leads Table 1 with 30.3% success and a 39.3% partial completion score, followed by Claude-3.7-Sonnet at 26.3% and 36.4%. Their average costs are $4.2 and $4.1 per evaluated instance, respectively, while Claude-3.5-Sonnet reaches 24.0% success at $6.3.

• Gemini-2.0-Flash achieves 11.4% success and a 19.0% score at $0.6 per instance, averaging 39.9 calls. Qualitative inspection identifies loops and aimless exploration as sources of prolonged execution.
• Llama-3.1-405b is the strongest open-weight baseline at 7.4% success, a 14.1% score, and $3.2 per instance. Llama-3.3-70b achieves 6.9% success at $0.9, showing lower cost with similar observed success.
• OpenHands with GPT-4o achieves 8.6% success, compared with 4.0% for OWL-RolePlay with GPT-4o and o3-mini. The authors describe lost browsing progress during delegation, but the different configurations do not isolate agent architecture as the sole cause.

Separate columns for full success, partial completion score, model calls, and cost reveal differences between completion and expense. Gemini-2.5-Pro leads at 30.3% success, while Gemini-2.0-Flash costs $0.6 per instance at 11.4% success; the rows also identify OpenHands versions and the different OWL-RolePlay configuration.

7.2 Analysis

Platform and occupational breakdowns show substantial variation. For Gemini-2.5-Pro, success is 33.80% on tasks requiring GitLab, 41.18% on Plane, 29.11% on RocketChat, and 12.86% on ownCloud; tasks can require several platforms, so these groups overlap.

• The occupational categories are Software Development Engineering (SDE), Project Management (PM), Data Science (DS), Administrative (Admin), Human Resources (HR), Finance, and Other. Gemini-2.5-Pro succeeds on 37.68% of SDE tasks and 39.29% of PM tasks, compared with 14.29% of DS tasks, 13.33% of Admin tasks, and 8.33% of Finance tasks.
• The authors connect failures to document understanding, communication, and complex user interfaces. Their explanation that public coding data and coding-focused evaluation favor software tasks is a hypothesis rather than an experimentally isolated mechanism.
• These differences describe performance on selected task groups. Without measured human performance, they do not establish a quantitative relationship between human task difficulty and agent task difficulty.

The platform panel shows lower completion on ownCloud than on Plane for all three displayed configurations. The occupational panel shows that the leading configurations also have low completion on data science, administrative, and finance tasks; these task-group comparisons do not isolate individual capabilities.

7.3 Common Agent Failures

The qualitative failure analysis identifies missed conversational implications, difficulty navigating web interfaces, and invalid shortcuts. One agent receives a recommendation to contact another colleague but stops without doing so; another renames an unrelated RocketChat user when it cannot find the intended person.

• OpenHands’ text-based browsing can become blocked by an ownCloud welcome popup. OWL-RolePlay’s visual browsing handles that obstacle better in the reported examples but more often selects incorrect elements in complex interfaces.
• These examples illustrate failure mechanisms; the paper does not report their frequencies across all failed tasks.

8 Implications and Future Directions

Most selected workplace tasks remain unsolved under the evaluated configurations, including tasks requiring collaboration and professional software interfaces. The authors propose extending coverage to other industries, physical work, ambiguous goals, and longer projects such as developing a product from conception through execution.

• Automated evaluation favors relatively concrete tasks and excludes open-ended creative work such as product brainstorming and system architecture design.
• Only two agent frameworks are tested, no quantitative human comparison is available, and task authors’ experience may differ from actual enterprise task distributions.
• The measured cost and capability differences support comparisons among the tested configurations. Predictions about future efficiency remain prospective.

Appendix

• A Agent Benchmark Comparison compares occupational breadth, colleague interaction, intermediate checkpoints, available interfaces, and self-hosting. It presents their combination as TheAgentCompany’s distinguishing benchmark design.
• B Benchmark Construction describes expert task curation across software engineering, project management, data science, human resources, administration, and finance. The environment includes setup problems, conversational pitfalls, cross-application workflows, and interface distractions to approximate workplace conditions.
• C Example Tasks illustrates dependent workflows involving task management, communication, code execution, reporting, and feedback. Checkpoints reward partial progress while preserving the distinction between intermediate achievements and full completion; evaluation can also assess deliverable quality.
• D Baseline Agent Details explains how OpenHands acts through a shell, interactive Python, and a browser. Its model selects subsequent actions from the execution history and current observations, including command results and structured browser information.
• E LLM-as-a-Judge describes model-based evaluation as a supplement to deterministic checks, mainly for information extraction and classification. Contributor review and final-deliverable checks can correct misjudged intermediate steps, but the appendix provides no numerical evaluator-agreement study.
• F LLM-as-Colleagues reports that inspection of an early benchmark version identified a role-playing prompt ambiguity that the authors fixed. The estimated interaction cost is conditional on assumed input/output token counts and model pricing and excludes prompt caching.
• G More TheAgentCompany Environment Details consolidates the fictional company’s products, employee roles and project assignments, communication channels, quarterly goals, and internal documents. These materials supply shared organizational context and information for task execution.
• H Agent-Simulated Colleagues Conversation Examples illustrates gathering requirements, negotiating within a budget, and coordinating availability. These selected conversations demonstrate the intended interactions rather than a quantitative evaluation of communication reliability.
• I Detailed Experimental Results provides model-performance breakdowns by required platform and occupational task category. Platform groups can overlap because a task may require several services; the results concern the selected task groups.
• J Limitation distinguishes human feasibility checks during task construction from a measured human-performance benchmark. The authors did not collect systematic human results because task duration, cost, and recruitment of participants with relevant backgrounds exceeded their available resources.

Brief Thoughts

The benchmark integrates execution, communication, and outcome checks in a resettable environment. Reporting both 30.3% full success and a 39.3% partial completion score records incomplete workflows, while platform breakdowns identify task groups with low completion.

The results support assessing agents as complete systems that use tools: browsing failures and lost delegation context can prevent success even when a task includes familiar coding or information processing. Occupational automation claims require additional evidence because task selection is not representative, human performance is unmeasured, and some model comparisons also change the agent framework version.