Language Self-Play For Data-Free Training | Summary

This article explains the key points of Language Self-Play For Data-Free Training.

• 2025-09-09 (arXiv)
• Kuba, Jakub Grudzien, Gu, Mengting, Ma, Qi, Tian, Yuandong, Mohan, Vijai, Chen, Jason.
• Meta Superintelligence Labs, UC Berkeley
• Paper

Read this article in Korean

Summary

• Language-model post-training, further optimization of an already trained model, depends on externally supplied task prompts. Starting from Llama-3.2-3B-Instruct, Language Self-Play (LSP), which trains one model to generate challenges and answer them, achieves a 36.4% overall win rate on AlpacaEval, an instruction-following evaluation, without an external prompt dataset, compared with 26.5% for the base model and 38.8% for reinforcement learning (RL), optimization using reward feedback, trained on Alpaca data.
• The model acts as Challenger, which generates queries that expose weaknesses, and Solver, which answers those queries. RL updates both roles, while self-rewarding, scoring interactions with a fixed reference copy of the initial model, encourages useful interactions and reduces reward hacking, behavior that increases a reward score without producing a helpful answer.
• Standalone LSP improves the reported scores from 48.7 to 52.4 on MATH and from 77.5 to 79.4 on GSM8K, two mathematics benchmarks, and from 97.2 to 98.1 on HumanEval, a coding benchmark. Adding subsequent data-based RL raises the overall AlpacaEval win rate to 39.5%, although this sequence does not outperform data-based RL on every benchmark.
• The results demonstrate improvement without additional externally supplied training examples, conditional on an already trained model and reward models, models that score generated answers. Experiments use one base model, and the authors identify reward quality and mismatch between generated queries and evaluation prompts as constraints; the reported benchmark comparisons do not establish indefinite improvement.

1 Introduction

Conventional language-model RL still requires task examples in the form of prompts, even when it does not require target answers. The paper treats prompt generation as a learnable action: one role generates challenging queries while another learns to answer them, allowing the query distribution and answering behavior to evolve together.

• Both roles share one language model, avoiding a separate adversarial model.
• Training begins with Llama-3.2-3B-Instruct; the absence of an additional prompt dataset does not remove its prior training.

2 Background

The background assumes access to a pretrained language model and introduces supervised fine-tuning (SFT), learning from supplied query-answer pairs, alongside RL. These approaches contrast fitting existing responses with optimizing rewards for generated responses.

2.1 Supervised Fine-Tuning

SFT maximizes the log-likelihood of provided answers conditioned on their queries, increasing the probability assigned to those answers. The paper argues that this objective lacks an explicit measure of answer quality, motivating reward-based optimization.

2.2 Reinforcement Learning

The RL formulation maximizes expected reward \(R(q,a)\) for a query q and answer a, supplied either by a verification engine that checks an answer or by a trained reward model. Group-Relative Policy Optimization (GRPO), an approach that compares sampled answers to the same query, estimates a query’s value as the mean reward of G answers and assigns each answer an advantage, its reward minus that mean.

• LSP uses this group-relative comparison to update Solver and estimate how difficult each generated query is for the current model.

3 Language Self-Play

Challenger generates queries under a special challenger prompt, <cp>, while Solver answers them using the same model parameters. In the initial minimax game, opposing optimization of the same expected task reward, Challenger minimizes Solver’s reward and Solver maximizes it. For N queries with G answers each, Solver’s advantage is its answer reward minus the query’s mean reward, and Challenger’s advantage is the overall mean query value minus that query’s value.

• Box 1 provides task and template placeholders, {task} and {template}, to configure query generation.
• Kullback–Leibler (KL) divergence regularization, a penalty for departing from a reference model’s output distribution, also discourages semantically meaningless adversarial queries.
• The paper calls this initial formulation Language Self-Play Zero (LSP-Zero), where Zero denotes the zero-sum game: the two roles receive opposing task rewards.

The two colored roles represent different behaviors of one shared model. Feedback from answering a generated challenge updates both behaviors, making the query distribution part of the learning process rather than a fixed external input.

Early experiments with OpenAssistant’s reward-model-deberta-v3-large-v2 produced unhelpful Python responses to ordinary queries, illustrating reward hacking. LSP adds a quality self-reward \(R_Q\), generated by the reference model using seven binary criteria in Box 3: Solver receives task reward plus quality reward, and Challenger receives negative mean task reward plus mean quality reward.

• The criteria assess whether the task is identifiable, instruction clarity, response comprehensibility, partial coverage, comprehensive usefulness, writing quality, and likely user preference for the response’s form and style; the additive score ranges from 0 to 7.
• Adding quality reward to both roles makes the game no longer zero-sum and encourages difficult but useful interactions.
• Algorithm 1 combines the role losses as L_Self-Play = L_Sol + α_Ch L_Ch, where α_Ch weights the Challenger loss, and updates their shared parameters.

4 Related Work

The paper distinguishes LSP from synthetic-data bootstrapping, learning from filtered model-generated examples, methods that edit existing training data, and self-play methods that optimize responses to supplied prompts. LSP learns query generation and answering together without requiring a specialized language game or a game-specific SFT phase.

• Compared with concurrent autonomous challenger-solver approaches, the authors emphasize less curated query generation and a combination of learned preference rewards, scores of answer desirability, and self-rewards.
• This reward design accommodates open-ended tasks, whereas the concurrent methods discussed emphasize deterministic, verifiable answers and majority-voting rewards, which use agreement among sampled answers.

5 Experiments

All experiments use Llama-3.2-3B-Instruct as the base model. The study first compares LSP-Zero, LSP, and data-based GRPO on AlpacaEval, then evaluates LSP as standalone training and as an intermediate stage before data-based RL on mathematics, coding, and instruction-following tasks.

• Table 1 lists 8 queries, β = 0.05 for KL regularization, 2 gradient accumulation steps, combining gradients before an update, and batch size 4 for all three methods.
• The learning rate is 1 × 10⁻⁶ for GRPO and 3 × 10⁻⁶ for both LSP-Zero and LSP; only LSP uses self-reward.

5.1 The Importance of Self-Rewarding

The self-reward ablation, a comparison that removes the quality self-reward, evaluates methods initialized from the same base model, with GRPO trained on Alpaca examples and the self-play methods generating their own queries. All three use Skywork-Reward-V2-Llama-3.2-3B as the reward model. AlpacaEval uses Llama-4-Maverick as judge, compares responses against a copy of Llama-3.2-3B-Instruct, and evaluates with sampling temperature τ = 0.01, a setting that limits randomness in generation.

• Overall win rates are Base 26.5%, GRPO 38.8%, LSP-Zero 32.0%, and LSP 36.4%; LSP exceeds the zero-sum variant by 4.4 percentage points.
• Results vary by prompt subset: LSP reaches 48.1% on Koala versus 37.8% for GRPO, but reaches 33.3% on Helpful-Base versus 44.2% for GRPO.
• The authors report that improved reward scores from the earlier OpenAssistant reward model did not translate into improved AlpacaEval scores, motivating their use of Skywork-Reward-V2.

Quality-regularized LSP improves the overall win rate by 9.9 percentage points over Base and by 4.4 points over LSP-Zero, while remaining 2.4 points below data-based RL. Its advantage on Koala and weaker result on Helpful-Base show substantial variation across prompt subsets.

Subsequent data-based GRPO increases the overall win rate from 36.4% to 39.5% for LSP and from 32.0% to 36.3% for LSP-Zero. LSP+RL exceeds standalone data-based RL’s 38.8% overall result by 0.7 percentage points, while the subset results show that additional RL can reduce performance on some prompt distributions.

• For LSP, Koala decreases from 48.1% to 42.9% and Vicuna from 36.2% to 26.2% after the additional RL stage.
• These comparisons favor quality-regularized LSP over LSP-Zero in the reported setting, but the figures provide no uncertainty intervals.

Adding data-based RL to LSP raises the overall win rate from 36.4% to 39.5%, above RL alone at 38.8%. Improvements are uneven: LSP+RL gains on OASST, Helpful-Base, and Self-Instruct but loses performance on Koala and Vicuna relative to standalone LSP.

5.2 Benchmark Evaluation of LSP

For MATH and GSM8K, the same standalone LSP model is trained with task=’Pure & Applied Mathematics’ in the challenger prompt. Data-based GRPO uses each benchmark’s training set, except that HumanEval has no training set and therefore uses MBPP, a programming-problem dataset, for training; evaluation uses the test sets.

• Figure 4 reports Base, RL, LSP, and LSP+RL scores respectively: MATH 48.7, 54.6, 52.4, 52.9; GSM8K 77.5, 79.8, 79.4, 79.9; HumanEval 97.2, 98.7, 98.1, 98.4; Alpaca 26.5, 38.8, 36.4, 39.5.
• Standalone LSP improves every reported benchmark while remaining below data-based RL on each aggregate score.
• LSP+RL slightly exceeds RL on GSM8K and Alpaca but remains below it on MATH and HumanEval. The authors hypothesize that generated-query mismatch, carried into subsequent RL through the KL constraint, limits the gains.

Bar heights show percentage-point gains over Base, while labels show absolute scores. LSP improves all four reported benchmarks; adding RL improves LSP further, but the resulting model exceeds RL alone only on GSM8K and Alpaca.

6 Conclusion

The conclusion presents LSP as a framework for improving a pretrained language model together with its self-generated training distribution, with favorable results on conversational tasks. The experiments use preference-based reward models; application to verifiable rewards is proposed rather than demonstrated here.

• The authors identify reward-model judgment quality and existing human knowledge about the physical world as constraints when ground-truth verification is unavailable.
• Their suggestion that embodied systems, systems able to interact with the physical world, could collect empirical data and expand knowledge is a future possibility.

Appendix

• A Example Plays reproduces Challenger queries and Solver responses at 500, 1000, and 1500 iterations: treasure-map instructions, a bridge-building task constrained to a 5-year-old, 10 wooden blocks, and 10 minutes, and a request for 2048-line assembly code subtracting two 16-bit numbers in consecutive 32-bit registers. These illustrate generated interactions rather than verified task success: the bridge response allocates six blocks, then two, then two, and then another two, and the assembly response does not supply the requested 2048 lines.
• B An Example Of Unregularized Play gives the full unhelpful Python response discussed in the main text, including unimplemented methods and a draw method that calls itself. C Hyper-parameters supplies Table 1, documenting the shared query count, KL coefficient, gradient accumulation, and batch size, along with the different learning rates and self-reward settings.

Brief Thoughts

The method supplies an explicit learning signal for query generation: low Solver reward identifies weaknesses, while shared quality reward discourages useless interactions. The 4.4-percentage-point advantage over LSP-Zero supports the quality-reward design in this experiment; the documented reward-hacking example illustrates the failure it addresses.

The reported gains make LSP relevant when an external prompt dataset is unavailable, provided that a trained model and suitable reward models are available. Evaluation with one base model, different learning rates for GRPO and self-play, no reported uncertainty intervals, and imperfect appendix responses leave generality, sensitivity to training settings, and sustained autonomous improvement unresolved.

The methods share 8 queries, β = 0.05, 2 gradient accumulation steps, and batch size 4. GRPO uses learning rate 1 × 10⁻⁶, while both self-play variants use 3 × 10⁻⁶; LSP-Zero and LSP share the listed settings except self-reward, whereas the GRPO comparison also changes learning rate.