TextArena | Summary

This article explains the key points of TextArena.

• 2025-04-15 (arXiv)
• Guertler, Leon, Cheng, Bobby, Yu, Simon, Liu, Bo, Choshen, Leshem, Tan, Cheston.
• Centre for Frontier AI Research (CFAR), A*STAR, Institute of High Performance Computing, A*STAR, Northeastern University, National University of Singapore, MIT, MIT-IBM Watson AI Lab
• Paper
• Github
• Project Page

Read this article in Korean

Summary

• TextArena addresses the difficulty of evaluating large language models (LLMs) on decisions made over multiple turns against other players, especially when success requires negotiation, deception, and reasoning about opponents. In its preliminary comparison of selected models and games, the pooled human baseline, “Humanity,” ranks above every plotted model in DontSayIt-v0 and TicTacToe-v0, while models lead the other three displayed games.
• The contribution is an open-source game framework with a common interaction interface, online model-versus-model and model-versus-human play, and TrueSkill™, a Bayesian rating system that estimates player skill and its uncertainty from match outcomes. The initial release contains 57+ environments; a publication-time footnote reports 74 games, and the online evaluation section reports 283 evaluated models, including 64 officially hosted models.

Humanity extends beyond the three plotted models on all ten axes, while the models have different profile shapes. Each axis is normalized separately and represents an aggregate of weighted game ratings, so distances across different axes do not establish absolute differences between skills.

• The paper documents the platform and preliminary rankings, but provides limited experimental detail for interpreting score differences. Results combine game strategy with rule and format compliance, and the paper does not report match counts, a complete prompting and inference protocol, or demonstrated training gains.

1 Introduction

The introduction motivates relative evaluation through competitive gameplay as an alternative to repeatedly constructing harder fixed benchmarks. Rankings can distinguish players as long as their capabilities differ, while game outcomes reduce reliance on human preference judgments of generated answers.

• The targeted capabilities include theory of mind, reasoning about other players’ beliefs or intentions, alongside persuasion, deception, spatial reasoning, and long-term planning.
• The framework supplies observations and rewards for offline development and online competition. Reinforcement learning (RL), training through interactions and rewards, and self-play, using games against other agents or model instances to generate training experience, are proposed uses; the paper does not demonstrate training improvements.

The leading player changes across games: Humanity leads DontSayIt-v0 and TicTacToe-v0, claude-3.7-sonnet:thinking leads Poker-v0 and SecretMafia-v0, and gemini-2.5-pro-preview-03-25 leads Snake-v0. The undefined error bars and the caption’s warning about models disclosing cards or roles limit interpretation of close rankings as differences in strategic ability.

2 Design Choices

The design prioritizes adoption and extension through interfaces resembling OpenAI Gym, also called Gymnasium, a framework for standardized environment interactions. Stackable wrappers, interface layers that can be combined around an environment, adapt interactions, while shared game functionality reduces the work required to add environments.

• The example creates agents with ta.agents.OpenRouterAgent, selects environments through ta.make(env_id=[“TicTacToe-v0”, “SpellingBee-v0”]), and applies ta.wrappers.LLMObservationWrapper(env=env).
• After env.reset(num_players=len(agents)), the loop obtains player_id and observation through env.get_observation(), generates an action with the corresponding agent, and advances the game with env.step(action=action). It retrieves rewards through env.close().

3 Environments

The environment collection includes existing games, modified games, and novel games for single-player, two-player, and multi-player settings; every game is either inherently text-based or adapted to text. Figure 3 illustrates formats ranging from grids and cards to dialogue, negotiation, and debate.

• The environments cover reasoning, vocabulary, memory, planning, risk assessment, deception, persuasion, and resource management. The paper describes these intended capabilities without experimentally establishing that individual games isolate them.
• The appendix’s Table 3 maps multi-player games to player counts and skill categories, including Poker and Snake for 2–15 players and SecretMafia for 5–15 players. Table 2 provides the corresponding map for single-player and two-player games; both tables use asterisks to identify games that are not fully implemented, qualifying the reported catalog coverage.

The examples show grids, cards, numeric feedback, and conversations within the environment collection. They illustrate the range of game representations, without establishing that each representation measures a distinct capability.

4 Online Evaluation

Online evaluation matches frontier models, community submissions, and humans, updating TrueSkill™ ratings after every match. Each model starts with \(\mu = 25\) and \(\sigma = \frac{25}{3}\), where μ represents estimated skill and σ its uncertainty; human players are collectively represented by the leaderboard identity “Humanity.”

• The authors select TrueSkill™ for individual and team competitions, varying player counts, and uncertainty handling. They report faster convergence than Elo, another competitive rating system, but provide no quantitative convergence results or detailed comparison protocol.
• Skill profiles use weighted environment labels from ten categories: Strategic Planning, planning actions toward a goal; Spatial Thinking, reasoning about positions; Pattern Recognition, identifying regularities; Theory of Mind; Logical Reasoning, drawing deductions; Memory Recall, retaining and retrieving information; Bluffing, misleading opponents; Persuasion, influencing other players; Uncertainty Estimation, assessing incomplete information; and Adaptability, adjusting behavior as play changes. A model’s category score is a weighted average of relevant environment scores, and Figure 1 normalizes each category separately.
• The main text states that each environment has up to five skill labels, although the appendix descriptions for Jaipur and MonopolyGame each list six. The paper does not explain how this discrepancy relates to the implemented weighting.

Figure 2 presents preliminary TrueSkill™ rankings for DontSayIt-v0, Poker-v0, TicTacToe-v0, Snake-v0, and SecretMafia-v0. Humanity leads the displayed DontSayIt-v0 and TicTacToe-v0 rankings, claude-3.7-sonnet:thinking leads Poker-v0 and SecretMafia-v0, and gemini-2.5-pro-preview-03-25 leads Snake-v0.

• The figure includes error bars, but the paper does not define their statistical interpretation or provide the number of matches behind each score. The rankings support descriptive comparisons within the plotted subset; the significance and stability of close differences remain unclear.
• The caption identifies a confound: some reasoning models reveal private cards or roles during play. Lower ratings can therefore reflect disclosure and failures to follow rules or formats as well as weaker strategy.
• Submissions have no formal restrictions, and the authors encourage distinct names for different model variants. The paper reports 283 evaluated models but does not specify standardized prompts, sampling settings, or computational budgets across submissions.

5 Related Work

The related-work comparison records environment counts and four capabilities: a Gym-compatible application programming interface (API), online evaluation, model-versus-model play, and model-versus-human play. Table 1 reports 16 single-player, 47 two-player, and 11 multi-player environments for TextArena, totaling 74, and marks it as the only listed framework supporting all four capabilities.

• The comparison includes Clembench, LMRL-Gym, GameBench, Game-theoretic LLM, LAMEN, GTBench, GameArena, and SPIN-Bench. It compares reported platform features rather than evaluation accuracy under a shared experimental protocol.
• For SPIN-Bench, Table 1 lists 21 single-player environments, while the accompanying prose says one. The PDF does not resolve this discrepancy.

TextArena’s row reports 16 single-player, 47 two-player, and 11 multi-player environments and support for all four listed technical capabilities. These entries document the authors’ reported platform coverage among the compared frameworks; they do not establish greater evaluation accuracy or validity.

The matrix associates social games with categories such as Theory of Mind and Persuasion, and Snake and Surround with Strategic Planning and Spatial Thinking. Player counts vary across games, and Pit’s asterisk identifies it as not fully implemented.

Single-player rows frequently include Logical Reasoning and Pattern Recognition, while two-player rows also include social and strategic categories. The indicators record assigned categories rather than measured skill effects, and asterisks distinguish entries that are not fully implemented.

6 Future Directions

The proposed directions are RL training on game environments, further community contributions, release of human–model gameplay trajectories, meaning sequences of interactions during a game, and VideoGameArena, an extension to competitive environments presented as successive frames with real-time directional and key-based inputs. The paper presents these as plans and reports no training improvements, released trajectory dataset, or VideoGameArena results.

• The planned trajectory release includes interactions with OpenAI o1, Claude-3.7-Sonnet, and Gemini-2.5-Pro. The authors also report that 64 hosted models are available for free online play.
• The documented resources are https://www.textarena.ai/, https://www.textarena.ai/leaderboard, and https://github.com/LeonGuertler/TextArena.

Appendix

• A Covered Games catalogs environments and their assigned cognitive or strategic skills. A.1 Single-Player Games includes puzzles such as FifteenPuzzle, Mastermind, Minesweeper, and Sudoku, alongside word games. Sudoku is associated with pattern recognition, logical reasoning, and memory recall; TowerOfHanoi is associated with strategic planning and logical reasoning.
• A.2 Two-Player Games spans board games, games with information hidden from opponents, bargaining, coordination, and dialogue. Examples include Chess, KuhnPoker, IteratedPrisonersDilemma, SimpleNegotiation, Debate, and TruthAndDeception. Table 2 records single-player and two-player skill assignments and marks incomplete implementations, including CarPuzzle and MathProof in the single-player group and several entries in the two-player group.
• A.3 Multi-Player Games describes auctions, discussion, word association, trading, bluffing, navigation, and alliance formation. Table 3 includes Codenames for 4 players, Diplomacy for 3–7, and Two Rooms and a Boom for 6+; Pit is marked as not fully implemented. SecretMafia appears in the table without a corresponding prose entry.
• B Online Model Play demonstrates registering a model through ta.make_online with model_name, model_description, email, and env_id=[“SpellingBee-v0”, “SimpleNegotiation-v0”, “Poker-v0”]. It wraps observations with ta.wrappers.LLMObservationWrapper(env=env), calls env.reset(num_players=1), and uses the observation–action loop until termination. The appendix notes that the setup may change over time.

Brief Thoughts

TextArena’s main contribution is the combination of a reusable text-game interface and live competition against models and humans. The code examples, rendered environments, feature comparison, and reported online participation support its use as research infrastructure.

The skill profiles aggregate game ratings through assigned labels, so interpreting them as separate measurements of each skill requires validation beyond the evidence presented. Pooled human ratings, unspecified match counts, and private-information disclosure also complicate comparisons. A controlled evaluation separating strategic errors from rule and format failures would help establish what the ratings measure.