Black-Box On-Policy Distillation of Large Language Models | Summary

Contents

Summary
1 Introduction
2 Method
3 Experiments
4 Related Work
5 Conclusion
Appendix
Brief Thoughts

This article explains the key points of Black-Box On-Policy Distillation of Large Language Models.

2025-11-13 (arXiv)
Ye, Tianzhu, Dong, Li, Chi, Zewen, Wu, Xun, Huang, Shaohan, Wei, Furu.
Microsoft Research
Paper
Project Page

Read this article in Korean

Summary

The paper addresses how to train a smaller language model from a teacher’s text responses while also learning from answers generated by the student itself. Under the same three-epoch training schedule, Generative Adversarial Distillation (GAD), which trains the student against a discriminator that scores teacher and student responses, raises Qwen2.5-14B-Instruct’s score on the LMSYS-Chat conversational test set to 52.1, compared with 50.6 for sequence-level knowledge distillation (SeqKD), which fits the student to teacher responses, and 51.7 for the GPT-5-Chat teacher. These are averaged GPT-4o evaluation scores based on each answer’s rating relative to a GPT-4o-generated reference answer, rather than accuracy percentages.

GPT-4o evaluation scores across Qwen2.5 student sizes on LMSYS-Chat and the average of Dolly, SelfInst, and Vicuna.

GAD enables black-box distillation, which uses teacher text without access to internal probabilities, and on-policy learning, which learns from responses sampled from the current student. Its discriminator scores prompt-response pairs, learns to rank teacher responses above student responses, and acts as a reward model that supplies feedback for student training; both models are updated throughout training.
With GPT-5-Chat as teacher, GAD outperforms SeqKD and the initial instruction-tuned models, models trained to follow instructions, across all five tested students and four evaluation datasets. Human comparisons favor GAD over SeqKD with win/tie/loss rates of 52%/40%/8% for Qwen2.5-7B-Instruct, 68%/8%/24% for Qwen2.5-14B-Instruct, and 44%/40%/16% for Llama-3.1-8B-Instruct.
The frozen-discriminator experiment supports continuously updating the reward model: the frozen version produces responses reaching about 1300 tokens, whereas the adaptive version maintains comparatively stable response lengths. Teacher comparability is limited to the reported automatic evaluation protocol; the paper provides neither uncertainty estimates for the main scores nor a comparison with SeqKD at matched computational cost.

1 Introduction

Knowledge distillation transfers a teacher model’s behavior to a student model. White-box distillation uses the teacher’s internal probabilities or representations, whereas black-box distillation uses generated text; proprietary teachers and incompatible tokenizers, the systems that map text to model tokens, make direct distribution matching difficult.

Kullback-Leibler divergence (KLD), a measure of the difference between probability distributions, supplies supervision in common white-box methods but cannot be directly computed from teacher text alone.
The introduction motivates an adaptive reward signal for student-generated responses, extending on-policy learning to teachers that expose only text.

2 Method

The method studies conditional generation: a student distribution qθ(y|x), parameterized by θ, should approximate a teacher distribution p(y|x), where x is a prompt and y is a response. Black-box training observes sampled teacher responses rather than the teacher’s predictive distribution, so GAD learns a discriminator to provide feedback on the student’s own outputs.

2.1 GAD: Generative Adversarial Distillation

GAD constructs a dataset of prompt-teacher response pairs and treats the student G as a generator, the model that produces responses. The discriminator D receives the concatenated prompt and response, projects the last token’s final hidden state to a scalar score, and is initialized from the generator’s model parameters with an additional prediction head.

The two-player minimax objective, in which the generator maximizes and the discriminator minimizes the same value function, is max_G min_D E_(x,y_t)∼T[−log σ(D(y_t)−D(G(x)))]. Here T is the training dataset, y_t is a teacher response, σ is the sigmoid function, and D(y) abbreviates the score conditioned on the prompt.
Bradley-Terry loss, a pairwise preference loss based on score differences, trains the discriminator to favor teacher responses. The student attempts to increase the discriminator’s score for its own responses.
Figure 2 shows the distinct update paths: gradients train the discriminator, while policy-gradient updates train the student through sampled text.

Teacher, student generator, and discriminator update paths in GAD.

2.2 Training

The student maximizes the expected discriminator score E[D(G(x))]. Because sampling text is not differentiable with respect to the student’s parameters, GAD uses policy gradient, which updates generation behavior from sampled responses and their rewards, with GRPO, the group-based policy optimization algorithm used in the experiments.

The discriminator minimizes Bradley-Terry loss on teacher-student response pairs, continually adapting its scores to current student outputs.
Before adversarial training, both models receive one warmup epoch: the generator fits teacher responses with cross-entropy loss, the supervised loss used to increase their likelihood, and the discriminator learns pairwise rankings.
Removing either model’s warmup reduces the reported evaluation scores in the warmup ablation.

2.3 Implement GAD with Reinforcement Learning Frameworks

GAD is implemented within reinforcement learning, which optimizes a policy using rewards, through frameworks such as verl. The generator serves as the policy model that samples responses, the discriminator serves as the reward model, and D(G(x)) supplies the reward; GAD updates the reward model alongside the policy.

Algorithm 1 first warms up both models, then repeatedly samples student responses, updates the generator using discriminator rewards, and updates the discriminator using teacher-student pairs.
The paper relates its discriminator to reward models in reinforcement learning from human feedback (RLHF), which uses human feedback to guide policy training. GAD’s pairwise labels instead come from teacher-versus-student identity.

Correspondence between reinforcement learning components and GAD components.

3 Experiments

The experiments compare GAD with the student before distillation and with SeqKD, using Qwen2.5 and Llama3 instruction models. They combine automatic scoring, human response comparisons, training-behavior analysis, and ablations (controlled changes to individual training components) of warmup, discriminator loss, and discriminator size.

3.1 Setup

Training uses 200K prompts sampled from LMSYS-Chat-1M-Clean, a cleaned conversational dataset, paired with GPT-5-Chat responses. Students are Qwen2.5-3B-Instruct, Qwen2.5-7B-Instruct, Qwen2.5-14B-Instruct, Llama-3.2-3B-Instruct, and Llama-3.1-8B-Instruct.

All models receive three training epochs with batch size 256 and approximately 2400 optimization steps. Prompt and response limits are 2048 and 1536 tokens, respectively; training temperature, the setting controlling sampling randomness, is 0.8, and checkpoints are saved every 50 steps.
Evaluation uses 500 held-out LMSYS-Chat-1M-Clean samples, 500 Dolly samples, 252 SelfInst samples, and 80 Vicuna questions. The latter three assess out-of-distribution generalization, performance on datasets different from the training source.
GPT-4o generates reference answers and scores model answers against them. Checkpoints are selected by the highest GPT-4o score subject to an acceptable response-length range; the paper specifies neither that range nor a separate checkpoint-selection split.
The dataset source is https://huggingface.co/datasets/OpenLeecher/lmsys_chat_1m_clean, and the conversational platform is https://lmarena.ai.

3.2 Main Results

Table 2 reports GAD improvements over both baselines for every GPT-5-Chat teacher experiment. For Qwen2.5-14B-Instruct, the GAD/SeqKD scores are 52.1/50.6 on LMSYS, 50.4/48.2 on Dolly, 51.1/49.4 on SelfInst, and 51.6/49.7 on Vicuna; the teacher scores are 51.7, 49.8, 49.7, and 49.9, respectively.

On LMSYS, Qwen2.5-3B-Instruct with GAD scores 48.9 versus 49.2 for Qwen2.5-7B-Instruct with SeqKD. Qwen2.5-7B-Instruct with GAD scores 50.8 versus 50.6 for Qwen2.5-14B-Instruct with SeqKD, supporting the paper’s approximate comparisons across model sizes.
The Llama students also improve: LMSYS scores rise from 47.6 to 48.1 for Llama-3.2-3B-Instruct and from 49.7 to 50.3 for Llama-3.1-8B-Instruct when replacing SeqKD with GAD.
Human win/tie/loss rates against the initial model are 68%/4%/28% for Qwen2.5-7B-Instruct, 56%/28%/16% for Qwen2.5-14B-Instruct, and 60%/28%/12% for Llama-3.1-8B-Instruct.
Against SeqKD, the corresponding rates are 52%/40%/8%, 68%/8%/24%, and 44%/40%/16%. GAD wins more often than it loses in all three comparisons, although the Llama comparison has a win rate below 50%.

Automatic evaluation scores for five students across LMSYS, Dolly, SelfInst, and Vicuna.

Human win, tie, and loss rates for GAD against initial students and SeqKD students.

3.3 Analysis

The analysis examines lexical overlap, distribution matching in a toy problem, and the stability of adaptive versus frozen discriminators. These experiments illustrate the proposed training mechanism, but do not directly measure the authors’ interpretation that GAD transfers global stylistic characteristics.

For Qwen2.5-14B-Instruct, SeqKD has higher N-gram overlap, similarity in contiguous sequences of n tokens, with the teacher across n=1 through n=6, despite a lower GPT-4o score. Figure 4 measures this overlap using an F1 score, an overlap metric combining precision and recall.
The toy teacher is a discrete Gaussian mixture, a distribution combining multiple Gaussian components, over classes 0,…,9. A single-Gaussian student trained only from teacher samples concentrates on a reachable high-probability region with GAD, called mode-seeking, but spreads probability more broadly with SeqKD, called mode-covering.
The toy GAD student uses REINFORCE, a sampled-reward policy-gradient algorithm. This example illustrates optimization under restricted student capacity; it does not establish that the same mechanism explains the language-model gains.
For the frozen-discriminator comparison, the student first receives one SeqKD epoch, the discriminator is trained for two epochs with that student frozen, and the discriminator is then frozen during student optimization. The paper reports reward hacking, exploitation of the reward signal through undesirable outputs, after around 300 training steps, with responses reaching about 1300 tokens; the adaptive version maintains comparatively stable response lengths in the displayed run.

Teacher-student N-gram overlap for Qwen2.5-14B-Instruct trained with SeqKD or GAD.

Teacher and student distributions in a discrete Gaussian-mixture distillation experiment.

Response-length trajectories with frozen and adaptive discriminators.

3.4 Ablations

The ablations favor warming up both models, using Bradley-Terry loss, and matching discriminator size to student size in the tested configurations. They support the default configuration within these comparisons.

For Qwen2.5-7B-Instruct, full GAD scores 50.8 on LMSYS and 50.0 on the average of the other datasets. Removing generator warmup lowers these to 49.7 and 49.7; removing discriminator warmup lowers them to 49.0 and 47.7.
For Qwen2.5-3B-Instruct, Bradley-Terry loss scores 48.9 on LMSYS and 47.9 on the other-dataset average, versus 47.9 and 46.4 with binary cross-entropy, a loss that classifies responses as teacher or student outputs.
For a 3B student, increasing the discriminator from 3B to 7B lowers scores from 48.9/47.9 to 47.8/46.9. For a 7B student, increasing it from 7B to 14B changes scores from 50.8/50.0 to 50.5/49.9.
The authors attribute these patterns to generator-discriminator balance. The tested larger discriminators do not improve scores, but the comparisons do not isolate balance as the causal explanation.

Generator and discriminator warmup ablations for Qwen2.5-7B-Instruct.

Discriminator loss comparison for Qwen2.5-3B-Instruct.

Evaluation scores for equal-size and larger discriminators with 3B and 7B students.

The related-work discussion distinguishes white-box methods that align output probabilities, hidden states, or attention scores from black-box methods that fit teacher-generated text. It positions GAD as a way to learn from student-generated responses when teacher internals are unavailable.

The paper identifies SeqKD and supervised fine-tuning on teacher reasoning traces, generated intermediate reasoning steps, as established text-only approaches.
Its adversarial formulation draws on generative adversarial networks, which train a generator against a discriminator, and earlier adversarial methods for sequence generation and imitation learning.

5 Conclusion

The conclusion presents GAD as a text-only distillation framework whose adaptive discriminator supplies feedback for on-policy student optimization. The experiments report improved conversational evaluation scores over SeqKD across the tested model families and datasets, including a Qwen2.5-14B-Instruct result comparable to the teacher under the specified GPT-4o protocol.

Appendix

A Experimental Details — A.1 Implement GAD with GRPO: each prompt produces a group of student responses, and the discriminator score supplies each response’s reward. The response advantage, its reward relative to the group, is calculated by subtracting the group mean and dividing by the group standard deviation; each student response is paired with the same teacher response for discriminator training. Equation (7) is explicitly abbreviated: the paper omits the KLD regularizer and clipping operator, the constraints used in policy optimization, from its displayed student objective.
A.2 Training Details: GAD uses one warmup epoch followed by two adversarial epochs, with the discriminator trained for 10 steps before joint warmup updates. The learning-rate search is [1e-6, 5e-6]; SeqKD uses 5e-6, while GAD with GPT-5-Chat uses 1e-6 in both stages, and GAD with the Qwen2.5 teacher uses 5e-6 for warmup and 1e-6 afterward. Group size is \(N=8\), the KLD weight is \(\beta=0.001\), and distilling Qwen2.5-14B-Instruct takes about 30 hours on 16 H100 GPUs.
A.3 Automatic Evaluation Details: generation uses greedy sampling, selecting the highest-probability token at each step, with a response limit of 1536 tokens. Figure 7 supplies the training and evaluation wrapper with “### Instruction:”, “{instruction}”, and “### Response:”; Figure 8 asks GPT-4o to score helpfulness, relevance, accuracy, and detail on a scale of 1 to 10. The reported score divides the student’s rating by the sum of the student and reference ratings, so the table values are neither accuracy percentages nor direct pairwise win rates.
B Additional Results — B.1 Additional Automatic Evaluation Results: Table 6 adds response lengths to the GPT-5-Chat teacher results; on LMSYS, Qwen2.5-14B-Instruct averages 438.9 tokens with GAD, 319.3 with SeqKD, 322.1 before distillation, and 329.1 for the teacher. Table 7 tests a Qwen2.5-14B-Instruct teacher with Llama students whose tokenizers are incompatible with the teacher’s: GAD outperforms SeqKD in seven of eight model-dataset comparisons, with Llama-3.2-3B-Instruct on SelfInst scoring 47.3 versus 47.6. For Llama-3.1-8B-Instruct, GAD scores 49.6, 49.9, 50.5, and 49.7 on LMSYS, Dolly, SelfInst, and Vicuna, respectively.

Brief Thoughts

GAD provides a concrete mechanism for training on student-generated text with a text-only teacher: a discriminator refreshed as the student changes. The main-table gains, warmup and loss ablations, and frozen-discriminator failure provide complementary evidence for this approach. The toy experiment and lexical-overlap analysis suggest possible explanations for the gains without establishing a complete causal account.

The evaluation leaves several questions unresolved: the paper reports no confidence intervals, detailed human-annotation sample counts, or clearly separate checkpoint-selection set. Table 6 shows longer responses with GAD than with SeqKD, making sensitivity to response length and detail relevant to interpreting the judging scores. Table 7 extends the results to a different teacher and incompatible tokenizers, while the reported 30 hours on 16 H100 GPUs provides a training-cost reference rather than a cost comparison with SeqKD.

Instruction and response prompt wrapper used for training and evaluation.

GPT-4o judging prompt for comparing two assistant responses.

GPT-5-Chat teacher evaluation scores and average response token lengths.

Llama student evaluation scores with Qwen2.5-14B-Instruct as teacher.