Build, Don't Play: Self-Improving LLM Research Agents Develop a Hearthstone Battlegrounds AI That Rivals Top Players | Summary

This article explains the key points of Build, Don't Play: Self-Improving LLM Research Agents Develop a Hearthstone Battlegrounds AI That Rivals Top Players.

• 2026-10-24 (arXiv)
• Ma, Changlian, Wang, Limin.
• Nanjing University, Shanghai AI Laboratory
• Paper
• Github

Read this article in Korean

Summary

• The paper asks whether large language model (LLM) research agents can build a competitive system for Hearthstone Battlegrounds, an eight-player game with hidden information and random battles. On each player’s last 20 public leaderboard records, the resulting TAVERNAGENT averaged an estimated placement of 1.60 and 80% first places, compared with 2.34 and 59% for the server’s ten highest-rated players combined.
• AUTOTAVERN coordinates specialized agents through versioned research records, verification checks and feedback from live games. In 20 days it built a simulator and delivered 23 versions of a reinforcement learning policy; on a shared TAVERNBENCH test, that policy scored 91% on play tasks versus 63% for the best directly answering LLM.
• The evidence covers one game and patch, a short leaderboard window and a benchmark with different simulator access across systems. Human researchers set goals, resolved ambiguous rules and pressed the keys named by the agent during live play.

1 Introduction

The authors study whether research agents can carry a problem from raw game data to a system evaluated against people. AUTOTAVERN uses discrepancies between simulation and live play to revise the training environment, opponents and research process.

2 Related Work

Prior Battlegrounds tools simulate limited parts of the game, omit features or advise human players. The paper combines agent-built environment engineering with proximal policy optimization (PPO), a reinforcement learning method, and measures progress on the live ranked ladder.

3 Hearthstone Battlegrounds

Each player recruits and arranges up to seven minions, the units that fight automatically against an opponent; players are eliminated until one remains. Patch 36.6.3 contains 978 game objects, and a game lasts about 15 turns and 450 actions per player, with only partial information about opponents.

4 The AUTOTAVERN System

Six role agents coordinate through typed, versioned records of findings, decisions, experiments and lessons. Five verification gates require tested builds, compatible model and simulator versions, comparisons within a build, checks on unseen random seeds and replay of live games.

The diagram links specialized agents and shared memory to simulator training and live-game replay; its chart shows first-place rates rising across delivered versions on both exams.

4.1 The iteration cycle

Each iteration observes, diagnoses, plans, implements, trains, evaluates, deploys and reflects. Training operates over seconds to minutes, research over hours and live-game feedback over days; a scheduler queues experiments within a resource budget.

4.2 Autonomy in numbers

Over 20 days the agents registered about 10,000 experiments and delivered 23 versions; the human contributed 418 messages, or 0.6% of all messages. The project recorded 375 rule verifications and 184 fixes prompted by live games.

The counts show the scale of the 20-day project and the reported human involvement: 418 messages accompanied about 10,000 registered experiments.

5 TAVERNGYM: Reproducing the Real Game

TAVERNGYM is a Rust game engine with Python bindings, all 978 objects in the studied patch and 245 discrete actions with a mask that identifies legal actions. A seed and action sequence reproduce a game exactly; with scripted bots the engine runs 190–280 complete eight-player games per second on one core, and unit tests, randomized checks and live-game replays support its fidelity checks.

6 TAVERNAGENT: Learning to Play the Live Game

TAVERNAGENT is an 18M-parameter transformer trained with PPO against seven opponent seats drawn from libraries of game states calibrated to human statistics and live games. On one simulator build, three unseen seeds and 24,576 games per row, its main-exam first-place rate rose from 1.9% for TA-1 to 41.5% for TA-23; calibrated opponents improved a matched training comparison from 20.9% to 27.2%.

The final agent improves on earlier versions in both exams, while the matched opponent ablation isolates a gain from calibration under the same starting checkpoint and training budget.

Across 392 recorded live games on days 5–18, its rating rose from 6,000 to 14,007. Excluding games flagged for client failures, TA-19–23 averaged 2.37 placement and 32.6% first places across 46 games linked with high confidence to a single version.

TAVERNAGENT combines high play accuracy with a 0.03 s decision time; most tested LLMs lie beyond the one-second threshold, while the chart also shows their different strengths by task.

The rating trajectory includes two pauses following live deployment problems, showing uneven progress and corrective feedback from deployment.

7 TAVERNBENCH: Build, Don’t Play

TAVERNBENCH contains 960 tasks across 40 scenarios; all systems were compared on 368 tasks. TAVERNAGENT received structured engine state and simulated 256 combats per candidate for short-term decisions, while LLMs received text without tools: their best play score was 63%, compared with 91% for TAVERNAGENT.

Under the paper’s one-second decision budget, TAVERNAGENT retained a 90% score, DeepSeek-V4.1-Flash 26%, and Jev, a decision model that selects typed options without generating text, 37%. LLM latency was estimated from published API medians, while TAVERNAGENT’s 0.03 s was measured.

The table separates play accuracy, the one-second score, rule execution and timing; TAVERNAGENT is not scored on rule execution because its simulator performs those rules.

8 Lessons from Self-Improvement

The checks exposed misleading signals: copied random-number states made search appear to gain 23.6 first-place percentage points, but reseeding reduced the gain to −0.65 ± 1.14 points. TA-10 improved its simulated first-place rate yet lost live rating, prompting investigation of client failures and weaknesses hidden by the exam; a separate input-encoding mismatch led to a bit-level comparison gate.

9 Conclusion

The authors conclude that the research agents built a stronger and faster game-playing system than the tested LLMs playing directly under the benchmark conditions. They state that TAVERNGYM and TAVERNBENCH will be open-sourced at https://github.com/MCG-NJU/AutoTavern.

Appendix

• A The AUTOTAVERN system in detail: The 184 live-game fixes comprised 46 rule and card-data fixes, 73 input-encoding fixes, 51 client-controller fixes and 14 opponent-library or exam recalibrations.
• B TAVERNGYM details: The approximately 63,000-line engine uses seeded randomness, versioned rules and exact save and restore; every mechanic has a unit test, according to the paper.
• C TAVERNAGENT details: Each 300-iteration training stage covers about 310 million decisions and 570,000 games. In an early equal-recipe size comparison, the 18M-parameter model achieved better placement and top-four rates than the 3.5M and 8M models on both exams.
• D Live deployment: The authors used their own account, and a human pressed the shortcut key specified by the policy. Of 388 games with valid placements, 96 had logged client failures; including those failures, TA-19–23 averaged 2.54 placement over 56 games.
• E TAVERNBENCH details: Simulator labels use repeated combat simulations, while strategy labels use 128 game continuations per plan by TA-21. On 144 short-term tasks, giving GPT-6 Astra simulator tools raised its accuracy from 62% to 97%, versus TAVERNAGENT’s 99%, with a reported 21 s per task.
• F Compute resources: Main-line training used eight H200 GPUs and 128 CPU cores. The authors report about 1,900 GPU-hours for delivered stages, 16,700 for other training, 1,300 for evaluation and analysis, and about 100 million research-agent output tokens.

Brief Thoughts

Live-game feedback exposed failures that the simulated exams missed, including the TA-10 regression and an encoder mismatch. The leaderboard comparison uses only 20 recent records per player, with placements estimated from rating changes and records potentially covering multiple games.

The benchmark comparison depends on access conditions: TAVERNAGENT has structured engine state and simulator lookahead, while the main LLM comparison uses text and no tools. The tool-assisted experiment shows that simulator access closes much of the short-term decision gap, although it adds latency.