SPADE: Self-Play in Adaptive Synthetic Executable Environments | Summary

This article explains the key points of SPADE: Self-Play in Adaptive Synthetic Executable Environments.

• 2026-08-19 (arXiv)
• Liu, Bo, Yu, Simon, Jiang, Yiding, Qu, Ao, Zhao, Andrew, Liu, Zichen, Kim, Junsu, Zhou, Zijian, Kim, Seungone, Ren, Tongzheng, et al.
• University of Washington, Stanford University, Northeastern University, Carnegie Mellon University, Massachusetts Institute of Technology, National University of Singapore, Seoul National University, Stevens Institute of Technology, University of Chicago
• Paper

Read this article in Korean

Summary

• SPADE trains one LLM in two prompted roles: an Environment Designer that writes executable Gym-style Python environments and a Reasoning Agent that solves them. The proposed adaptive curriculum combines corpus grounding, environment memory, and a hint-based regret signal, and is evaluated in game-based reasoning and multi-step tool use.

4 SPADE: Self-Play in Adaptive Synthetic Executable Environments

The framework alternates environment generation and validation with hinted and unhinted Agent rollouts, then applies joint GRPO updates to shared parameters. The generated code implements state transitions, rewards, and verification through a reset()/step() interface, covering both answer-oriented and stateful multi-turn tasks. For stateful, see this post.

The overview connects the shared-policy dual-role loop to the reported transfer curves: the 30B-A3B games curve exceeds the fixed-environment baselines, while tool-use improvements vary across the three benchmarks.

4.1 Dual-Role Self-Play and Code-as-Environment

Role-specific prompts select Designer or Agent behavior from the same policy πθ. Candidates must pass syntax and execution checks, while training uses role-specific advantage normalization, delayed Designer updates with truncated importance sampling, asymmetric clipping, and a difficulty anchor targeting a 0.4–0.6 Agent win-rate band.

4.2 Hint-Based Regret Reward

The Designer reward is the difference between mean Agent return with and without a privileged hint, rD(e) = r̄A(e | h) − r̄A(e). A large gap identifies tasks that hints make solvable but the unaided policy has not mastered; training floors raw regret at zero and blends it with the difficulty anchor.

The diagram specifies that corpus passages and memory condition environment generation, and that both Designer regret and Agent task reward update the same policy through GRPO.

The logged examples illustrate the behavioral basis of positive regret: hints suggest a productive probe pattern or parameter range and are associated with successful hinted trajectories where unhinted play fails or remains partial.

4.3 Environment Design

Fresh documents supply topical material and a memory buffer of prior environments, skill labels, and regret scores supplies frontier-oriented seeds. Games use 10k mathematics and 5k science documents from DCLM and MegaScience, while tool use uses 15k Nemotron pretraining-code documents.

Four checkpoints from one 30B-A3B run illustrate the authors' proposed curriculum evolution, from a car-ownership environment to a thermodynamics laboratory environment with state-gated interaction.

5 Experimental Setup

The authors train Qwen3-4B-Instruct-2507, Qwen3-8B, and Qwen3-30B-A3B-Instruct-2507 with GRPO for 400 rollouts of 24 environments. Every reported evaluation benchmark is held out from training.

5.1 SPADE Training Recipe

The common recipe uses a 24-environment rollout batch, group size G=16, learning rate 1 × 10−6, and delayed Environment Designer updates. The Agent makes 16 × k unhinted plays and 16 hinted plays per environment, with k=4 for games and k=8 for tool use; the RL backbone is slime.

5.2 Domain: Games

In games, each rollout contains 24 self-contained Python environments across three rotating skills drawn from Mathematical Reasoning, Logical Deduction, Spatial Reasoning, Pattern Recognition, Optimization, and Causal Inference. Fixed-env GRPO and Fixed-env RLVE are retrained with the same 400-iteration budget, and evaluation covers AIME 2025/2026, GPQA-Diamond, LiveCodeBench-v6, and four hard Reasoning-Gym categories.

5.3 Domain: Tool Use

For tool use, the Designer creates simulated OpenAI function-calling tools, mutable backend state, and three to five sequential user instructions with state-based criteria. Tool environments must pass deterministic reset checks and an LLM solvability screen; evaluation uses BFCL v4 multi-turn, τ²-bench, and ACEBench-Agent, while cited comparison rows use different models, budgets, harnesses, and sometimes benchmark variants.

6 Experimental Results

The experiments test transfer from adaptive synthetic environments to held-out reasoning, code, and agent-interaction tasks, then analyze the generated curriculum and Agent behavior. Evidence includes benchmark tables, learning curves, component ablations, diversity measurements, and qualitative traces.

6.1 Quantitative Analysis

At 30B-A3B in games, SPADE reaches a suite average of 58.3, versus 50.2 for the untrained base and 53.0 for the strongest fixed-environment baseline, Fixed-env RLVE. In tool use at 30B-A3B, it improves BFCL v4 multi-turn by +5.7, τ²-bench by +3.6, and ACEBench-Agent by +13.9 over its corresponding base.

Across the three Qwen3 backbones, SPADE has the highest suite average: 44.1 at 4B, 55.5 at 8B, and 58.3 at 30B-A3B. At 30B-A3B, its largest gains over base are on Reasoning-Gym Math (+18.3), Algorithmic (+14.1), and Cognition (+14.7).

The checkpoint plots show SPADE preserving competition-math performance while separating most clearly from fixed-environment baselines on procedural reasoning, GPQA-Diamond, and LiveCodeBench-v6.

SPADE improves every displayed tool-use benchmark relative to its corresponding Qwen3 base; at 30B-A3B it reaches 54.7 on BFCL v4 multi-turn and 75.9 on ACEBench-Agent. The table and appendix explicitly caution that the cited reference-system rows are not protocol-matched.

6.2 Qualitative Analysis

In the full 30B games run, the share of environments with Agent win rate in [0.2, 0.8] reaches roughly one third late in training. Corpus grounding is associated with higher reported diversity, Vendi/n 0.68 with the corpus versus 0.04 without it, while trajectory examples show the Agent moving from long up-front derivations toward probing and evidence gathering before committing to a solution.

The full configuration raises the late-stage fraction of learnable environments, whereas corpus-, memory-, or Designer-training ablations decline or collapse. This figure measures the supplied training environments rather than direct held-out-task performance.

The t-SNE plot is illustrative, but the reported quantitative Vendi/n difference is large: 0.68 with corpus grounding and 0.04 without it. This supports the paper's association between external documents and avoidance of generation collapse.

The examples and formula-reveal statistic show later physics environments withholding solution methods more often while retaining multi-turn structure; the no-corpus condition repeats RotatingMazeEnv 41 times from steps 290–312.

The selected traces depict a qualitative shift from repeated unparsable long-form answers, through short hypothesis testing, to probing before deriving a hidden rule and solving at turn 4.

7 Ablations

The ablations assess online Designer adaptation and the reward used to train it. On Qwen3-30B-A3B games, the complete configuration attains the highest selected suite average; removing memory, corpus grounding, or Designer training reduces the selected result and some partial configurations peak early before declining.

7.1 Ablation: Environment Designer Adaptation

Removing both Environment Designer training and memory yields a suite average of 40.5, below the 50.2 untrained base, whereas a static corpus-grounded GPT-5.5-generated pool reaches 53.0. These controls favor the full adaptive configuration, but they alter multiple factors and therefore do not isolate a single causal component.

The full setting scores 58.3, compared with 53.2 without memory, 53.5 without corpus grounding, 40.5 without Designer training and memory, and 53.0 using a fixed GPT-5.5 environment pool. These best-checkpoint comparisons support the adaptation ablation but do not independently manipulate all factors.

The full SPADE configuration remains strongest late in the run, while no-memory and no-corpus variants peak near step 111 and subsequently decline on several procedural metrics. The figure provides trajectory context for the selected-checkpoint values in Table 3.

7.2 Ablation: Environment Designer Reward

Replacing hint-based regret with EMA-based learning potential gives 55.9 rather than 58.3 on the eight-benchmark suite. The paper attributes the difference to the EMA signal’s unsigned deviation from a running skill mean, which can score mastered and intractable environments alike and requires history before becoming informative.

Hint-based regret produces the strongest six-benchmark trajectory average and separates from EMA learning potential after approximately 50 steps; both trained-Designer variants exceed the no-Designer-training-and-memory control.

8 Scaling Results

The average games gain over base rises from +5.2 at 4B to +5.7 at 8B and +8.1 at 30B-A3B, while Fixed-env GRPO remains near +1.2. A six-skill curriculum reaches 58.3 versus 53.7 for a two-skill variant, indicating that broader curriculum coverage contributes materially to the reported transfer.

The left panel summarizes increasing gains with model size, while the right panel shows a limitation of the finite-sample signal: estimated hint regret is negative for long intervals at 4B and 8B even though both models improve over their bases.

The restricted two-skill curriculum improves over the untrained base but reaches 53.7 rather than 58.3 for the six-skill curriculum. The comparison indicates that curriculum breadth is an important contributor to the suite-level gain.

9 Discussion

The reported results support adaptive executable-environment generation as a post-training mechanism in the tested Qwen3 settings. The authors note that environment complexity remains bounded by model scale and context, GRPO is human-designed, hint-based regret lacks a practical optimal-curriculum guarantee, and fixed benchmarks do not establish open-ended capability growth.

Appendix

• The appendix formalizes an idealized expected-regret game: under sound generation, articulated hints, and internalizability assumptions, every pure Nash equilibrium has zero hint regret and hint-free optimality on every valid environment. It also provides prompts, hyperparameters, reproducibility details, extended ablations, diversity analyses, and generated-environment examples; raw game-code executability is reported as 84.9% as emitted and 90.3% after stripping Markdown fences, with the pipeline sanitizer repairing remaining failures.

Brief Thoughts

SPADE’s operational contribution is to train an environment generator online rather than treating synthetic environments as a static dataset. Its controlled comparisons show consistent gains, but the generated examples and stated validation limits make environment quality, reward specification, and protocol-matched external comparisons important targets for independent verification.