SIMA 2: A Generalist Embodied Agent for Virtual Worlds | Summary

Contents

Summary
1. Introduction
2. Background & Related Works
3. Methods
4. Results
5. Discussion
Appendix
Brief Thoughts

This article explains the key points of SIMA 2: A Generalist Embodied Agent for Virtual Worlds.

2025-12-04 (arXiv)
SIMA team, Bolton, Adrian, Lerchner, Alexander, Cordell, Alexandra, Moufarek, Alexandre, Bolt, Andrew, Lampinen, Andrew, Mitenkova, Anna, Hallingstad, Arne Olav, Vujatovic, Bojan, et al.
Google DeepMind
Paper

Summary

The paper addresses how to turn a pretrained multimodal model, which processes images and language, into an embodied agent that observes and changes diverse 3D game environments through actions. On held-out tasks within training environments in the updated evaluation suite, SIMA 2 achieves 65% success under human evaluation and 66% under automatic evaluation, compared with SIMA 1’s 33% and 30%, respectively. SIMA stands for Scalable Instructable Multiworld Agent.
The agent adapts Gemini Flash-Lite through supervised fine-tuning (SFT), learning from examples, followed by reinforcement learning (RL), improving behavior through rewards. Human gameplay teaches keyboard-and-mouse control, while synthetic bridge data, examples connecting dialogue and reasoning to actions, supports interactive behavior; non-gameplay pretraining data is mixed in to preserve general capabilities.
SIMA 2 transfers to ASKA, a Viking village-building survival game, and MineDojo, a Minecraft task benchmark, without embodied training in those environments, although absolute success remains low. Qualitative demonstrations extend to The Gunk, a story-driven action-adventure game, and Genie 3, a model that generates interactive virtual environments. A separate Gemini Pro model can provide higher-level instructions and text summaries carried between calls.
A Gemini task setter, which proposes achievable instructions, and reward model, which scores recorded attempts, support learning from self-generated experience in ASKA and Genie 3 environments. The experiments demonstrate improvement over a finite sequence of training iterations; very long tasks, short interaction memory, precise control, and complex scene understanding remain limitations, and general reasoning benchmark scores decline after adaptation.

1. Introduction

The paper motivates embodied learning through the gap between reasoning over static internet data and acting successfully in an environment. SIMA 1 followed short, direct instructions from screen images but could neither converse nor accept image-based instructions, and it generalized unreliably to new situations.

SIMA 2 integrates Gemini to interpret high-level goals, formulate multiple steps, discuss its behavior, and act through the visual and keyboard-and-mouse interface used by human players.
Virtual games provide the experimental setting. Transfer to physical robotics is a proposed application, not an evaluated result.

Overview of SIMA 2’s dialogue, reasoning, embodied action, task performance, and self-improvement.

The related work connects game-based agent research, world models, pretrained embodied models, and autonomous task and reward generation. SIMA 2 is a vision-language-action model, a model that combines visual and language inputs with action outputs, fine-tuned for 3D virtual worlds.

Games & Simulation Driving Agent Research traces the progression from constrained games to visually complex, open-world settings that permit exploration and require multiple skills and explicitly defined goals.
World Models describes models that predict future outcomes from observations and actions; Genie supplies newly generated interactive environments for transfer and learning experiments.
Foundation Models in Embodied Agents and Open-Ended Self-Improvement motivate combining pretrained knowledge with control, then using separate foundation models as agent, task setter, and reward model. Open-ended learning refers here to the aim of continually generating new tasks and acquiring skills.

3. Methods

The method combines training across multiple environments, a common screen-and-input interface, human and synthetic gameplay data, and subsequent reward-based training. Its evaluation separates unseen starting states in familiar environments from entirely unseen environments.

3.1. Environments

Training uses ten environments: Construction Lab, Playhouse, WorldLab, Goat Simulator 3, Hydroneer, No Man’s Sky, Satisfactory, Space Engineers, Valheim, and Wobbly Life. Space Engineers is newly added relative to SIMA 1 and includes resource extraction, construction, variable gravity, and jetpack movement with six degrees of freedom, allowing translation and rotation along three axes.

3.1.1. Held-Out Environments distinguishes saved starting states absent from training from entirely new games with unfamiliar visuals, menus, and mechanics.
ASKA and Minecraft are evaluated quantitatively. The MineDojo subset contains 50 tasks from Combat, Harvest, and Tech Tree, each evaluated with 15 random seeds specifying environment configurations; Tech Tree tasks involve progression through crafting dependencies.
The Gunk requires navigation and use of a suction tool. Genie 3 supplies newly generated naturalistic and urban photorealistic environments, scenes with realistic visual appearance, for qualitative evaluation.

Sample scenes from the environments used to train and evaluate SIMA 2.

3.2. Agent-Environment Interface

The agent receives 720p RGB video frames, color images represented by red, green, and blue channels, together with a history of instructions and its own generated text. It receives no privileged underlying game state and produces structured text that is deterministically parsed into action chunks, sequences of controls applied to the environment, dialogue, or internal reasoning.

The action interface includes 96 standard keyboard keys, mouse clicks, and discretized relative mouse movements.
The agent chooses which output types to produce at a given step. Figure 3 illustrates their interleaving during a burger-cooking task.

Interleaved visual observations, reasoning, dialogue, and actions in the agent-environment interface.

3.3. Data, Agent & Training

Gemini Flash-Lite is fine-tuned using a mixture of gameplay and Gemini pretraining data. The authors report that retaining non-gameplay data is crucial for preserving vision understanding, dialogue, reasoning, and responsiveness to prompts.

Human trajectories, recorded sequences of observations and actions, constitute most training data by volume and teach low-level control.
Synthetic bridge data connects high-level user interaction with reasoning and action. The paper does not disclose numerical dataset sizes or mixture proportions.

3.3.1. Human Data combines gameplay video, keyboard-and-mouse actions, and language annotations from consenting, compensated participants. Collection includes free-form play with later annotation, live Setter-Solver interaction, predefined Game-Tasks, and human judgments of trajectory success or relative quality.

In Setter-Solver collection, one participant issues instructions before another executes the corresponding actions, linking language to subsequent behavior more directly than retrospective annotation. Game-Tasks are episodes with a specified instruction and starting state.
Preprocessing resizes frames, filters low-quality data, reweights environments and datasets, and splits trajectories into spans containing one instruction and its associated frames and actions.
Gemini provides additional synthetic language and reasoning annotations.

3.3.2. Bridge Data starts from a relatively small set of high-quality successful trajectories across training environments. Gemini Pro adds reasoning and dialogue consistent with the visible scene and the agent’s first-person behavior, producing examples that interleave language with actions.

Examples cover error correction, instruction following and chaining, answering questions about visual observations, memory use, and behavior over extended sequences.
Training prompts are varied to encourage robustness, and no-ops, steps with no action, teach the agent to remain still after completion.
The annotations align generated reasoning with recorded behavior; the paper does not report a controlled test isolating the contribution of this reasoning text.

3.3.3. Reinforcement Learning uses online experience on verifiable tasks defined by an initial game state, text instruction, and verification function. Rewards follow task completion or a correct environment-grounded answer; some tasks add shaped rewards, supplementary feedback intended to improve instruction following and controllability.

Participants propose achievable tasks from random states, and verification functions identify additional task completion points in human trajectories.
Tasks are filtered for human completion within a specified time limit. Dialogue tasks pair screenshots with human-suggested questions and answers.
This training phase excludes held-out environments such as ASKA and MineDojo.

3.4. Evaluations

Evaluation uses ground-truth checks based on underlying environment state, programmatic checks based on screens and actions, or five independent human ratings per video. Programmatic checks include optical character recognition, detecting on-screen text, together with pixel-color and action heuristics; their coverage is limited to outcomes these rules can detect.

SIMA Evaluation Suite 2.0 expands tasks and strengthens completion criteria through persistent text detection, limits on actions after completion, and sequential tasks that require every sub-task to succeed.
The stricter suite lowers SIMA 1’s scores relative to its original report, so the relevant comparison is between agents on the updated suite.
Human baselines generally use matching task time limits. Humans receive all steps of sequential tasks at once, whereas agents receive each instruction after completing the preceding task; experienced players assess familiar environments, and players unfamiliar with ASKA and MineDojo assess transfer.

4. Results

The results examine new interaction capabilities, task success in familiar and unfamiliar environments, comparison with unadapted Gemini, composition with Gemini Pro, and learning from self-generated experience. Aggregate task evaluations measure success rates, while selected demonstrations illustrate particular behaviors.

4.1. New Capabilities

SIMA 2 supports Embodied Dialogue, including answering questions by taking actions to obtain information, and Basic Reasoning, interpreting indirect instructions using its pretrained knowledge. In the illustrated examples, it approaches objects in No Man’s Sky to read their material information and interprets a house colored like a ripe tomato as a red house.

Complex Instructions includes multilingual instructions in French, German, and Mandarin Chinese despite English-only embodied training, emoji instructions, and multiple navigation steps.
Multi-modal Prompting includes sketches specifying objects, locations, or paths. A tree sketch leads the agent to identify and chop a tree.
The paper provides examples but no aggregate success rates for multilingual, sketch-based, or indirect instructions.

Examples of embodied dialogue and interpretation of an indirect navigation instruction.

4.2. Embodied Task Performance

4.2.1. Performance in Training Environments reports success averaged over environments: 65% for SIMA 2 versus 33% for SIMA 1 under human evaluation, and 66% versus 30% under automatic evaluation. Experienced humans achieve 76% and 78% under the corresponding time limits, rising to 86% in both evaluation types without those restrictions.

These are held-out tasks and starting states within environments seen during training.
Some time limits are as short as three seconds. The paper identifies human inattention, infrastructure latency, and fine-motor demands as factors affecting this comparison.

Average embodied-task success in training environments for SIMA 1, SIMA 2, and human baselines.

Environment-level results show gains across the evaluated environments, with especially large improvements in several commercial games. Across eight skill categories, SIMA 2 approaches human performance more closely in interaction and object management, while resource gathering and combat retain larger gaps.

The human-evaluated gains shown in Figure 8 range from 18 percentage points in Satisfactory to 46 percentage points in Goat Simulator 3.
Figure 9 shows automatic-evaluation gains ranging from 5 percentage points in Playhouse to 57 percentage points in Goat Simulator 3 and Satisfactory.
Combat requires rapid decisions and precise movement; the paper’s Valheim deer-hunting example illustrates these remaining motor-control difficulties.

Task performance across eight embodied skill categories.

Human-evaluated success by training environment, with timed and untimed human baselines.

Automatically evaluated success by training environment.

4.2.2. Performance in Held-Out Environments shows improvements of 12 percentage points in ASKA and 13 percentage points in MineDojo over SIMA 1, although SIMA 2’s plotted success rates remain below 15%. In MineDojo, SIMA 2 completes tasks in 26 of 50 task categories, whereas SIMA 1 succeeds only in harvest dirt and combat spider.

ASKA uses programmatic evaluation, and MineDojo uses ground-truth state evaluation. Illustrated successes include locating a campfire, using an unfamiliar villager-selection menu, mining coal, and fighting a zombie.
First-attempt human baselines are roughly 19% on a 16-task MineDojo subset and 32% on a 25-task ASKA subset. These subset results do not establish a directly matched full-suite human comparison.
The authors caution that humans frequently fail because of time limits, while agents often fail through suboptimal exploration.

Success rates in the held-out ASKA and MineDojo environments.

Selected SIMA 1 and SIMA 2 trajectories in held-out ASKA and Minecraft environments.

With manual instructions, SIMA 2 progresses through the first 15–20 minutes of The Gunk to the Campsite checkpoint, handling scanning, ledges, gaps, and clearing two areas of gunk. In Genie 3 environments, qualitative examples show navigation to targets across naturalistic and urban photorealistic scenes.

In The Gunk, the agent uses the on-screen ABSORB and HOLD cues to select a sustained left-mouse action for the unfamiliar suction tool.
The Genie 3 demonstrations include approaching a bench, a red flower, orange coral, leafy greens, an espresso machine, doors, and a fire hydrant.
No aggregate success rate or physical-world evaluation is reported for these demonstrations.

Manually instructed progression and example interactions in The Gunk.

Navigation examples in naturalistic and urban Genie 3 environments.

4.3. Comparison to Baseline Gemini Models

Without SIMA action-data adaptation, Gemini Flash-Lite achieves 3.2% success and Gemini Pro 7.0% across the paper’s programmatic evaluations in the ten training domains, despite efforts to prompt correct output formatting. General capability retention is assessed using LiveCodeBench (LCB), a code-generation benchmark, AIME, the American Invitational Mathematics Examination dataset, and GPQA Diamond, a difficult scientific question-answering benchmark.

After supervised fine-tuning, relative score changes from the baseline Gemini model are -4.0% for LCB (Code), -25.5% for AIME (Math), and -16.3% for GPQA Diamond (STEM). STEM denotes science, technology, engineering, and mathematics.
After supervised fine-tuning plus reinforcement learning, the corresponding changes are -8.4%, -15.4%, and -19.5%. These are relative changes in benchmark scores, not percentage-point differences in accuracy.
The results show retained capabilities alongside measurable regressions. Reinforcement learning improves the AIME comparison while worsening the coding and GPQA Diamond comparisons; the table provides no uncertainty estimates or significance tests.

Relative changes in general benchmark scores after SFT and SFT + RL.

4.4. Gemini Instructing SIMA 2

A hierarchical configuration places Gemini Pro above SIMA 2: Pro reviews recent video every k steps, issues natural-language instructions, and passes a text summary into its next call as recurrent memory, information carried between successive calls. This separates slower higher-level reasoning from the lower-latency action model.

The main example uses an illustrated Valheim campfire guide. The combined system interprets the diagram, gathers materials, equips a hammer, and selects and places a campfire.
The demonstration shows that this composition can complete the illustrated task, but the paper does not report a controlled aggregate comparison with SIMA 2 alone.

Following an illustrated campfire-building guide with Gemini Pro and SIMA 2.

4.5. Self-Improvement

The self-improvement process couples SIMA 2 with a Gemini-Based Task Setter and a separate Gemini-Based Reward Model. The setter proposes instructions achievable from the current scene and can redirect practice toward weaker skills; the reward model scores trajectory videos using the instruction and visible evidence without internal game-state variables.

The scoring rubric assigns scores from 0 to 100 for completion and directedness, avoiding unnecessary actions. A score of 50 or greater counts as success.
The rubric prompt is calibrated against human preference pairs on a small trajectory dataset.
Scored self-generated trajectories become training data. This model-based scoring process differs from the verifiable rewards used in the initial reinforcement-learning stage.

4.5.1. ASKA first isolates improvement on a fixed task set covering resource gathering, interaction, navigation, and menu use. The initial agent succeeds on less than a quarter of these tasks, while successive training iterations eventually exceed the reward-model success threshold across all tasks and surpass the average score of experienced human reference trajectories.

These comparisons use Gemini reward-model scores for both agent and human trajectories; they do not establish general superiority to human players.
With automatic task generation focused on game progression, the agent learns unfamiliar crafting and menu skills from its own experience.
The resulting agent is evaluated through manual instructions along the ASKA technology tree and builds a shelter within one hour, progressing beyond the original SIMA 1 and SIMA 2 agents.

Reward-model scores over successive self-improvement iterations on fixed ASKA tasks.

Self-improvement data collection and subsequent ASKA technology-tree progression.

4.5.2. Genie 3 splits generated environments into urban training settings and natural test settings, with tasks primarily involving navigation. Self-generated experience improves scores on nearly all training tasks, often by 25 points or more, and improves a majority of held-out natural-environment tasks.

Figure 17 includes improved navigation to a lollipop in an urban candy store and a red mushroom in a natural scene.
Some task scores decline, particularly in the test set. Improvement is broad but non-uniform.
These experiments provide preliminary evidence of transfer across generated environment types after adaptation; they do not test indefinitely continuing learning.

Training and held-out test score changes after self-improvement in Genie 3.

5. Discussion

The discussion identifies very long tasks, multi-step reasoning and goal verification, limited interaction memory, precise keyboard-and-mouse control, and robust understanding of complex 3D scenes as unresolved challenges. A short context window, the recent interaction history available to the model, supports low-latency interaction but limits memory, and physical-world transfer remains a prospective direction.

SIMA 2 is presented as a limited research preview with early access for a small group of academics and game developers.
SIMA 2 Team lists contributors alphabetically by first name and provides the collective citation author SIMA Team, year 2025, for SIMA 2: A Generalist Embodied Agent for Virtual Worlds.

Appendix

A. Embodied Skill Categories defines Interaction, Navigation, Menu Use, Tool Use, Construction, Object Management, Resource Gathering, and Combat. Table 2 supplies descriptions and examples, clarifying the categories used in the main skill-level comparison.
B. Additional Results Combining Gemini Pro & SIMA 2 provides qualitative examples of Abstract Reasoning, applying a remembered rule to subsequent requests, and Active Exploration, choosing movements and questions to obtain information. The combined agent retains an instruction to perform the opposite of subsequent requests, and it plays 21 questions by exploring the environment and using the user’s answers to identify an object.
C. Additional Self-Improvement Results illustrates ASKA behaviors acquired after adaptation: approaching a rain collector and extinguishing a campfire. The ASKA Technology Tree diagram shows the minimal dependencies needed to summon the first villager, providing context for the main game-progression experiment.

Brief Thoughts

The clearest contribution is the combination of pretrained multimodal knowledge with explicitly trained low-level control: the gains over SIMA 1 and the low success of unadapted Gemini support the value of embodied training on these tasks. The paper provides less direct evidence about how much each data component, generated reasoning, or reward-based training contributes because it does not report detailed component ablations, controlled comparisons that remove individual training components. Generalization is meaningful but incomplete: unfamiliar-game success rates remain below 15%, while The Gunk and initial Genie 3 results are qualitative. Self-improvement is supported by rising rubric scores and concrete acquired behaviors. Additional independent evaluation would help assess reliability beyond the reward-model scores and the manually instructed progression demonstration. Capability retention has measurable costs: the final model shows relative declines of 8.4%, 15.4%, and 19.5% on the three reported coding and reasoning benchmarks.