Learning to Reason without External Rewards | Summary

Version note. This retrospective review covers arXiv:2505.19590v1, published on 26 May 2025.

Xuandong Zhao, Zhewei Kang, Aosong Feng, Sergey Levine, and Dawn Song investigate INTUITOR, a reinforcement-learning method that scores a language model’s sampled answers using its own distributional confidence. Paper, Code.

Read this article in Korean

Summary

• Reinforcement learning with verifiable rewards (RLVR) needs an answer checker, such as a gold mathematical solution or code tests. INTUITOR substitutes the model’s own self-certainty for that external training reward in group-relative policy optimization (GRPO).
• In the reported Qwen2.5-3B MATH-training comparison, INTUITOR scores 0.612 versus GRPO’s 0.636 on MATH500, but 0.153 versus 0.085 on LiveCodeBench v6 and 0.416 versus 0.341 on CRUXEval-O. These values do not imply that confidence guarantees correctness. [p. 6, Table 1]
• The paper reports that a fixed confidence scorer can be exploited, while an online scorer remains stable in the tested run. Larger models needed modified training settings after an initial collapse. The preprint also disagrees with itself on some hyperparameters, which must be checked against code before reproduction. [pp. 9–10, 15–16]

1. Introduction

RLVR improves reasoning where an answer can be checked, but requires a domain-specific verifier and, in the math setup, reference solutions. Code verification similarly needs tests and execution infrastructure. The authors ask whether a model can improve from a signal computed from its own predictions and call this broader setting reinforcement learning from internal feedback (RLIF). Human-defined questions, the pretrained model, and external evaluation still matter: the absence of external training rewards is not the absence of data or evaluation. [pp. 2–4]

2. Related Work

The paper situates INTUITOR among preference-based feedback, verifiable rewards, and self-generated training signals. Those approaches differ in what supplies the feedback; the presence of self-generated text alone does not make their reward mechanisms identical. [p. 3]

3. Method

3.1 From External Supervision to Internal Feedback

The authors contrast a learned preference reward with a verifiable task reward and discuss their dependence on external supervision or checking infrastructure. [pp. 3–4]

3.2 Reinforcement Learning from Internal Feedback (RLIF)

RLIF replaces the external reward in the objective with an internal signal, while retaining a reference-policy KL term. [p. 4]

3.3 INTUITOR: Policy Optimization with Self-Certainty

For question \(q\), sampled response \(o\), vocabulary \(V\), and uniform distribution \(U\), self-certainty is the mean token-position divergence

\[ u(q,o)=\frac{1}{|o|}\sum_{i=1}^{|o|}D_{\mathrm{KL}}\!\left(U\,\|\,p_{\pi_\theta}(\cdot\mid q,o_{<i})\right). \]

The order is \(D_{\mathrm{KL}}(U\|p)\), not \(D_{\mathrm{KL}}(p\|U)\): it is not simply negative token entropy. Higher values are interpreted by the authors as greater self-certainty. For each question, GRPO samples several completions, INTUITOR scores each completion, and the scores are normalized within the group to form relative advantages for policy updates. A KL term constrains deviation from a reference policy. A confident model can still be wrong, so the useful link between the signal and correctness remains an empirical question. [pp. 4–6]

The source diagram shows how sampled responses are scored and normalized. The reference model supplies the regularizing reference; it does not serve as a gold-answer verifier.

4. Experimental Setup

The principal experiments use Qwen2.5-1.5B and 3B models and the 7,500-problem MATH training split. The authors describe seven completions per question, 128 questions per update, and matching settings for the main GRPO and INTUITOR comparison. GRPO-PV uses plurality voting rather than gold answers. Evaluation uses greedy decoding and includes GSM8K, MATH500, LiveCodeBench v6, CRUXEval-O, MMLU-Pro, and length-controlled AlpacaEval. Evaluation can use external correctness criteria even when the training reward does not. [p. 6]

INTUITOR-Code is a separate Codeforces-trained condition, with 14 completions per question and 50 steps. The text describes a learning-rate change from \(3\times10^{-5}\) to \(1\times10^{-5}\) and a default MATH KL penalty of 0.005. Appendix Table 5 instead lists MATH and Codeforces learning rates of \(3\times10^{-6}\) and \(1\times10^{-6}\), and a MATH KL penalty of 0.0005. For Qwen2.5-7B and 14B, the prose reports 16 completions per question, whereas Table 5 lists a group size of 14. This inconsistency is present in v1 and has not been resolved here. The code and configuration history need checking before reproducing the run. [pp. 6, 16]

5. Results and Analysis

Table 1 reports the following for Qwen2.5-3B. GRPO and INTUITOR both train on MATH; the base model is not fine-tuned in this comparison. [p. 6]

The original table includes additional evaluation columns and the separate Codeforces-trained condition; the four-column table above isolates the comparison discussed here.

INTUITOR is lower than GRPO on the displayed math results and higher on the two code benchmarks. Relative to the 0.093 base score, 0.153 is approximately a 65% relative increase; the absolute increase is six percentage points. On Qwen2.5-1.5B, LiveCodeBench rises from 0.000 to 0.099 with INTUITOR versus 0.056 with GRPO. The base model frequently produces repetitive or malformed output under the chat-style evaluation, so the comparison also reflects improved valid instruction-following. [pp. 6–7]

5.1 Learning to Follow Instructions

The paper describes fewer repetitive or invalid chat-formatted outputs for the smaller base model after training. [p. 7]

5.2 Fostering Structured Reasoning

At training step 10, the displayed table gives INTUITOR higher GSM8K and MATH values than GRPO for both tested small model sizes. This is an early-checkpoint observation, not a universal learning-speed guarantee. The paper also reports that some model responses include free-form reasoning before a required JSON answer or code. Its qualitative LiveCodeBench analysis uses ten randomly selected questions, so the samples illustrate behavior rather than proving a broad causal mechanism. [pp. 7–9, Table 2 and Figures 5–6]

In Figure 4, MATH500 accuracy rises early, preceding gains on LiveCodeBench. In the two displayed INTUITOR curves, LiveCodeBench scores rise after MATH500 accuracy levels off; this pattern is not shared by every displayed curve. These temporal patterns are compatible with transfer but do not establish that mathematical improvement causes the later code gains. [p. 8]

5.3 Understanding Emergent Long-Form Reasoning

The authors interpret some sampled pre-code reasoning as a possible contributor to performance; the displayed qualitative sample does not prove that mechanism. [pp. 8–9]

5.4 Online Self-Certainty Prevents Reward Exploitation

In a reduced-batch experiment the paper compares online self-certainty, computed by the evolving policy, with an offline scorer fixed to the base model. In the offline run, around update 100 the policy appends an already-solved auxiliary problem, response length increases, and validation accuracy collapses. The online run remains stable in the reported comparison. That result does not establish immunity to reward hacking in all other settings. [pp. 9–10, Figure 7]

The figure plots accuracy on the left and completion length on the right. The offline curve's falling accuracy and rising length occur together in this particular experiment.

Figure 8 compares self-certainty distributions for correct and incorrect MATH500 responses. The reported Mann–Whitney analysis gives U = 30910, p = 0.52, r = -0.00 for the base model generating and grading, and U = 45415, p = 8.2e-24, r = 0.45 for INTUITOR generating and grading. These are associations within the reported evaluation conditions; they do not establish a causal mechanism or guarantee the correctness of an individual high-confidence answer. [pp. 9–10]

5.5 Ablation Studies

The KL-penalty ablation shows that code benchmarks vary substantially with the penalty in the displayed settings. In the appendix, the original training recipe for Qwen2.5-7B and 14B initially collapsed; the authors simplified the system prompt, reduced the learning rate, and increased the sample count without a comprehensive sweep. They additionally report a Llama3.2-3B-Instruct experiment. [pp. 9–10, 15–16]

6. Discussion and Future Research

The paper shows that a self-generated distributional signal can be useful under particular training conditions. Its evidence does not establish that confidence is correctness, that other policy-gradient algorithms will work as well, or that performance transfers to arbitrary domains. The difference between the learned reward and the external evaluation criteria is central to interpreting its results. [pp. 4–6, 10–11]

7. Conclusion

The authors propose internal self-certainty as a training signal and report the results and limits above. [p. 11]

Appendix

• A.1–A.3 report larger-model settings, a Llama3.2-3B-Instruct experiment, and hyperparameters; the main-text/appendix discrepancy is retained above. [pp. 15–16]
• B.1–B.6 give prompts and sample model completions, including code and CRUXEval-O examples and training rollout evolution. These examples illustrate output behavior rather than a new aggregate benchmark result. [pp. 17–29]

Brief Thoughts

Internal confidence is worth studying as a training signal, but external evaluation remains necessary to distinguish confident output from a correct answer.