Summary explanation: LLM-as-a-Coach: Experiential Learning for Non-Verifiable Tasks

Contents

• Summary
• 1 Introduction
• 2 Method
  • 2.1 Preliminaries: RL with an LLM-as-a-Judge
  • 2.2 Experiential Learning with an LLM-as-a-Coach
  • 2.3 From Judge to Coach
  • 2.4 Feedback-Bandwidth Intuition
• 3 Experiments
  • 3.1 Setup
  • 3.2 Main Results
  • 3.3 Experiential Learning Generalizes Better
  • 3.4 Ablations
  • 3.5 Analysis
• 4 Discussion
• 5 Related Work
• 6 Conclusion
• Appendix
• Brief Thoughts

This article explains the key points of LLM-as-a-Coach: Experiential Learning for Non-Verifiable Tasks.

• 2026-07-20 (arXiv), arXiv
• Ye, Tianzhu, Dong, Li, Chen, Guanheng, Zhu, He, Wu, Xun, Huang, Shaohan, Wei, Furu.
• Microsoft Research, Tsinghua University, Peking University
• Paper

Read this article in Korean

Summary

• The paper studies post-training for non-verifiable open-ended tasks, where rubric-based LLM assessments are commonly reduced to scalar rewards. It proposes Experiential Learning (EL), which turns the feedback model into an LLM-as-a-Coach that extracts transferable experiential knowledge and internalizes it through on-policy context distillation.

A travel-guide response illustrates the distinction between a single score of 7 and coach-produced advice about specificity, sensory detail, and grounding. The displayed 3.3-bit versus 17,600-bit comparison is an upper-bound bandwidth illustration under the paper's stated discrete-score and textual-channel assumptions.

1 Introduction

RL is effective when correctness can be automatically checked, but open-ended responses must be assessed across dimensions such as factuality, relevance, completeness, organization, and style. The authors argue that a scalar reward discards the evaluator’s written analysis and makes responses with the same score indistinguishable, especially near the top of the reward scale.

2 Method

EL trains a policy πθ from prompt-and-rubric pairs (x, Rx). For each on-policy response, a feedback model produces reusable experiential knowledge, which conditions a teacher that supplies dense token-level supervision to the policy.

2.1 Preliminaries: RL with an LLM-as-a-Judge

The RL baseline uses feedback model M as an LLM-as-a-Judge, extracting r = Extract_Reward(M(x, y, Rx)) for y ∼ πθ(·|x) and maximizing expected reward with GRPO. Although the judge may generate detailed analysis, optimization receives only the sequence-level scalar score.

2.2 Experiential Learning with an LLM-as-a-Coach

In EL, the coach extracts experiential knowledge e = Extract_Knowledge(M(x, y, Rx)) from each rollout. The policy minimizes token-level reverse KL to πteacher(·|e, x) along its own trajectory, converting the coach’s textual guidance into a context-conditioned target distribution that is absorbed into the policy weights without requiring the context at inference.

The diagram contrasts the update paths: RL maximizes a judge-provided reward, while EL conditions a teacher on coach-produced experiential knowledge and minimizes a reverse-KL objective. Both approaches begin with prompts, policy rollouts, and rubrics.

2.3 From Judge to Coach

The judge and coach can use the same feedback model and prompt-specific rubrics, but expose different learning signals. A judge provides a scalar assessment, whereas a coach supplies transferable guidance intended to improve similar future responses; the distinction concerns the optimizer’s interface rather than different intrinsic capabilities of the feedback LLM.

The table isolates EL's conceptual change: both interfaces use the same LLM and rubrics, but the judge evaluates with a scalar score whereas the coach supplies transferable guidance with higher stated feedback bandwidth.

2.4 Feedback-Bandwidth Intuition

The feedback-bandwidth comparison is presented as intuition rather than a measurement of usable information. A discrete 1–10 reward has at most log2(10) ≈ 3.3 representational bits per sample, whereas a 1024-token context from a 150,000-token vocabulary has a theoretical upper bound of about 17,600 bits; natural-language redundancy and imperfect extraction can substantially reduce effective supervision.

3 Experiments

Experiments compare EL with Rubric-as-Reward RL on held-out WildChat-IF and four unseen open-ended benchmarks. The study uses Qwen3-8B and OLMo-3-7B-Instruct policies, with either the initial frozen policy checkpoint or GPT-4o as feedback model, and also examines teacher updates, alternative teacher contexts, and a controlled feedback-quantization analysis.

3.1 Setup

Training uses 7,500 WildChat-IF prompts with GPT-4o-generated rubrics; both methods use batches of 256 prompts, 8 responses per prompt, learning rate 1e-6, temperature 1, and 3 epochs. Evaluation uses 250 held-out WildChat-IF prompts plus AlpacaEval v2.0, WildBench, ArenaHard v2.0, and CreativeWritingV3, with GPT-4o as evaluator; the authors report a consistency check using a stronger proprietary evaluator.

3.2 Main Results

EL outperforms RL on most reported benchmarks across policy and feedback-model combinations. For Qwen3-8B with Qwen3-8B feedback, EL obtains 80.0 on WildChat, 40.0% AlpacaEval-v2 win rate, 21.8 on WildBench, 31.0% ArenaHard-v2 win rate, and 76.0 on CreativeWritingV3, versus RL’s 79.2, 37.3%, 18.4, 30.5%, and 74.3; with GPT-4o feedback, EL is lower on AlpacaEval-v2 but higher on WildBench, ArenaHard-v2, and CreativeWritingV3.

Across Qwen3-8B and OLMo-3-7B policies and two feedback-model choices, EL generally improves held-out WildChat and unseen-benchmark metrics relative to RL. The table also shows exceptions, including Qwen3-8B with GPT-4o feedback on AlpacaEval-v2, where EL's 37.4% win rate is below RL's 38.2%.

3.3 Experiential Learning Generalizes Better

On a 1,000-example WildChat training subset, RL gains more than EL (+1.7 versus +1.3), whereas EL gains more on WildChat test (+2.0 versus +1.1), AlpacaEval (+5.5 versus +2.8), and WildBench (+4.2 versus +0.8). The paper hypothesizes that scalar-reward optimization can favor patterns that inflate training rewards without transferring, while experiential guidance transfers more effectively.

The bars show that RL has the larger training-set gain, whereas EL has larger improvements on WildChat test, AlpacaEval, and WildBench. This is the paper's direct empirical evidence for stronger transfer beyond the training distribution.

3.4 Ablations

Iterative teacher updates raise Qwen3-8B’s WildChat score from 80.0 with a fixed teacher to 80.7, but reduce IFEval accuracy from 83.1 to 74.9. Mixing Tulu3 general prompts at a 1:0.25 ratio preserves the 80.7 score and recovers IFEval to 79.9; extracted experiential knowledge also exceeds Full Critique, Rubrics Only, and Multiple-Choice teacher contexts on both reported WildChat score and OOD accuracy in the Qwen3-1.7B experiment.

Teacher iteration improves the reported WildChat score but causes a substantial drop in IFEval OOD instruction-following accuracy. Adding general-prompt on-policy distillation partially restores this capability while retaining the reported WildChat score.

Extracted experiential knowledge achieves the best WildChat score, 68.6, and the best listed OOD accuracy, 68.8, in this Qwen3-1.7B experiment. Full Critique and Rubrics Only improve task score but lower OOD accuracy, while the low-bandwidth Multiple-Choice context preserves OOD accuracy but yields smaller task gains.

3.5 Analysis

The analysis characterizes EL’s external textual-feedback channel as O(L), compared with RL’s O(1) scalar signal, while distinguishing it from ordinary on-policy distillation, which can draw O(N) information from a stronger teacher’s parameters. In a V = 200 toy distribution-matching setting, binary rewards approximate a binary target similarly to direct reverse-KL matching, while five quantized reward levels yield a staircase approximation of a bimodal target that direct distributional matching recovers more smoothly.

The table classifies RL as O(1) scalar feedback, conventional on-policy distillation as O(N) feedback from a larger teacher, and EL as O(L) feedback from coach-produced experiential context. It is qualitative and distinguishes information source as well as scaling notation.

For the constructed binary target, RL and direct distributional matching produce similar curves. For the bimodal target quantized into five reward levels, RL forms plateaus while direct distributional matching follows the smooth target; the paper presents this only as intuition about quantization.

4 Discussion

EL is intended to mitigate reward hacking caused by compressing feedback between the evaluator and policy, not errors originating in a biased or miscalibrated feedback model. It performs distribution matching toward a context-conditioned teacher rather than scalar-reward maximization, and the paper reports that its additional teacher forward pass and token-level distillation are small relative to shared rollout and feedback-generation costs in its setting.

5 Related Work

The work relates EL to learning from experience, textual feedback, context distillation, on-policy distillation, and on-policy self-distillation. Its specific design extracts transferable knowledge from rubric-based response assessment and uses that knowledge as external context for on-policy distillation on non-verifiable tasks.

6 Conclusion

The paper concludes that experiential knowledge provides a richer post-training signal than scalar rubric rewards for the reported open-ended tasks, with stronger held-out and unseen-benchmark results in most comparisons. The evidence remains bounded by LLM-based evaluation, the selected models and benchmarks, and a bandwidth argument explicitly described as theoretical intuition rather than a measure of effective supervision.

The template directs the coach to evaluate rubrics, assign a score, and distill general reusable strategies inside <experience> tags. EL extracts this latter field as e rather than using the score as its learning signal.

This template prepends extracted experience and asks the teacher to solve the original prompt. It implements the context conditioning that produces the teacher distribution used in EL's reverse-KL objective.

The Multiple-Choice ablation limits feedback to one of nine predefined directives, with an empty fallback, producing ten possible contexts. The paper uses it as a low-bandwidth comparison with the same log2(10) maximum capacity as a 1–10 score.

Three examples show training rubrics for a broad analytical response, SQL query construction, and recipe scaling. Each rubric contains named criteria and weights, including a negative-weight pitfall criterion in the recipe example.

The examples emphasize strategies abstracted beyond a single response, including maintaining an emotional arc, prioritizing factual precision, and structuring a technical report. They illustrate the operational definition of experiential knowledge as transferable rather than instance-specific feedback.

Appendix

• The appendix provides EL knowledge-extraction and teacher-conditioning prompts, the RL scoring prompt, teacher-context ablation templates, GPT-4o-generated rubric examples, extracted experience examples, and an end-to-end EL example. The default coach prompt requests rubric analysis, a 1–10 score, and reusable high-level strategies enclosed in <experience> tags; EL uses the extracted experience to condition the teacher.

Brief Thoughts

The core comparison is controlled in that RL and EL use the same rubrics and can use the same feedback model, isolating the effect of the learning interface. However, the generalization and reward-hacking interpretation is suggestive rather than conclusive: most outcomes rely on GPT-4o evaluation, and the quantization experiment is explicitly a toy construction rather than a simulation or proof about all scalar-reward methods.