J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learning | Summary

Contents

Summary
1 Introduction
2 J1: Thinking-LLM-as-a-Judge via Reinforcement Learning
3 Experimental Setup
4 Results
- 4.1 Main Results on Benchmarks with Pairwise-J1 and MultiTask-J1
- 4.2 Ablations and Analyses of J1 in Pointwise vs. Pairwise Setups
5 Related Work
6 Conclusion
Reproducibility Statement
Appendix
Brief Thoughts

This article explains the key points of J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learning.

2025-05-15 (arXiv)
Whitehouse, Chenxi, Wang, Tianlu, Yu, Ping, Li, Xian, Weston, Jason, Kulikov, Ilia, Saha, Swarnadeep.
FAIR at Meta
Paper

Summary

The paper studies how to train an LLM-as-a-Judge, a large language model that evaluates other models’ responses, to reason before deciding and remain correct when response order changes. Using 22K synthetic preference pairs, each containing a preferred and a rejected response, J1-Qwen-32B-MultiTask reaches 76.8% accuracy on Preference Proxy Evaluations (PPE) Correctness, a benchmark of verifiable response pairs, compared with 66.5% for its Qwen3-32B base model when both are evaluated by comparing two responses together.
J1 uses reinforcement learning, training from rewards assigned to generated outputs, to optimize evaluation reasoning and final decisions together. It makes judgment rewards checkable against constructed preference labels, trains with both response orders, and combines pairwise evaluation, comparing two responses together, with pointwise evaluation, scoring each response independently.
Across five benchmarks using their respective evaluation metrics, the multitask model obtains an overall score of 80.8, compared with 77.3 for Qwen3-32B and 78.4 for DeepSeek-R1-671B. Pointwise evaluation improves accuracy requiring correctness under both response orders, while pairwise evaluation performs better under a randomly selected order. Constructed-label validity, remaining order sensitivity, and reporting discrepancies qualify the findings.

1 Introduction

J1 investigates online training, which updates the judge using newly generated evaluations, instead of relying only on prompting or offline preference training. Its three contributions are unified training on synthetic preferences, optimization of chain-of-thought, intermediate reasoning generated before a judgment, and a multitask judge designed to reduce position bias, the dependence of a verdict on response order.

Figure 1 illustrates evaluation criteria, reference-answer generation, reassessment, and response comparison. These examples show observable output patterns without measuring how frequently those strategies occur.

Illustrative reasoning patterns for pairwise and pointwise J1 evaluation.

2 J1: Thinking-LLM-as-a-Judge via Reinforcement Learning

The primary judge receives an instruction x and responses a and b, generates intermediate thought tokens t, and returns a verdict y identifying the preferred response. Figure 2 connects synthetic preference construction, batches containing both response orders, rule-based rewards, and GRPO, the online reinforcement-learning optimizer used to train reasoning and judgments jointly.

Synthetic preference construction, reward computation, and online training for J1 variants.

2.1 Synthetic Training Data Generation

Training uses 22K synthetic preference pairs: 17K WildChat prompts, drawn from user conversations, and 5K MATH prompts, drawn from mathematical problems. For WildChat, a model generates a modified instruction and an answer to it; that answer becomes a rejected response to the original instruction. For MATH, rejected generations fail to reach the gold answer.

Both (x, a, b) and (x, b, a) enter the same training batch, enabling rewards that inspect correctness across response orders.
The construction makes agreement with the preference label verifiable. For subjective prompts, it does not independently establish that every constructed label represents human preference.

2.2 Reward Modeling

Verdict Correctness assigns +1 when the judge identifies the preferred response and 0 otherwise. Verdict Consistency assigns +1 only when the judge is correct under both response orders, with 0 if either ordering is incorrect. Additional rewards enforcing <think> tags provide no noticeable benefit.

2.3 J1 Formulations and Training

J1 explores five formulations with different outputs and reward rules. The multitask formulation jointly trains pointwise evaluation and pairwise evaluation with scores, allowing the same model to operate in either setting.

Pairwise J1 with Verdict (PaV) outputs (t, y); Pairwise J1 with Scores (PaS) outputs (t, s_a, s_b), where s_a and s_b score the two responses, and selects the higher score; Pairwise J1 with Scores&Verdict (PaVS) outputs (t, s_a, s_b, y), with rewards determined only by y.
Pointwise J1 (PoS) outputs reasoning and a score between 0 and 10 for one response. Distant supervision, learning individual scores from pairwise labels without individual score annotations, rewards the model when the preferred response receives the higher score.
Multitask Pairwise & Pointwise J1 (MT) combines PoS and PaS training. The main benchmark comparisons evaluate this model in its pairwise setting.

3 Experimental Setup

All variants train on the same 22K synthetic pairs using verl, the training framework used in the experiments. Evaluation covers PPE, RewardBench, RM-Bench, JudgeBench, and FollowBenchEval, with model selection based on overall RewardBench accuracy and inference performed through vLLM, the authors’ inference engine.

PPE Preference contains 10.2K human preference pairs spanning 20 models and 121+ languages. PPE Correctness contains 12.7K samples across MMLU-Pro, MATH, GPQA, MBPP-Plus, and IFEval, covering knowledge, mathematics, question answering, coding, and instruction following.
JudgeBench uses 350 challenging preference pairs in the GPT-4o response subset. RM-Bench contains 4K samples testing subtle content differences and style biases; FollowBenchEval contains 205 pairs testing multiple response constraints; RewardBench contains 3K samples across chat, chat-hard, safety, and reasoning.
PPE, RewardBench, and RM-Bench use random-order accuracy, correctness under a randomly ordered response pair. JudgeBench and FollowBenchEval use position-consistent accuracy, requiring a correct verdict under both response orders.

4 Results

The results compare J1 with base judges, scalar reward models, which output a numerical reward directly, and generative judges, which generate reasoning before a score or verdict. Subsequent analyses examine evaluation format, response-order consistency, reward design, seed prompts, generation length, and additional inference-time sampling.

4.1 Main Results on Benchmarks with Pairwise-J1 and MultiTask-J1

Table 1 reports PPE Correctness accuracies of 59.2 for J1-Llama-8B, 72.9 for J1-Llama-70B, 74.6 for J1-Qwen-32B, and 76.8 for J1-Qwen-32B-MultiTask. The multitask result exceeds Qwen3-32B by 10.3 percentage points, EvalPlanner-Llama-70B by 6.6 points, and DeepSeek-GRM-27B by 17.0 points, using the displayed table values.

The multitask model scores 85.0 on MMLU-Pro, 94.3 on MATH, 68.6 on GPQA, 66.3 on MBPP-Plus, and 69.5 on IFEval.
EvalPlanner uses the same synthetic data distribution with direct preference optimization (DPO), an offline method for learning from preferred and rejected outputs. This comparison supports the J1 training recipe but does not isolate every training difference.

PPE Correctness results across base judges, reward models, and J1 models.

Table 2 gives J1-Qwen-32B-MultiTask scores of 71.8 on PPE, 93.6 on RewardBench, 90.3 on RM-Bench, 71.4 on JudgeBench, and 77.1 on FollowBenchEval. It exceeds both OpenAI-o3 and DeepSeek-R1-671B on RewardBench, RM-Bench, and FollowBenchEval; OpenAI-o3 scores higher on PPE and JudgeBench. The PPE column here includes both preference and correctness subsets, so its 71.8 measures a different evaluation set from Table 1’s 76.8.

J1-Llama-8B and J1-Llama-70B exceed their corresponding DeepSeek-R1-Distilled-Llama models on all five benchmarks.
The RewardBench comparison in Table 6 reports the highest overall score among listed models at 93.6, only 0.1 above SF-Foundation/TextEval-Llama3.1-70B. J1 does not lead every category.

4.2 Ablations and Analyses of J1 in Pointwise vs. Pairwise Setups

Table 3 shows that pointwise evaluation improves position-consistent accuracy while retaining ties, cases where both responses receive equal scores. For J1-Qwen-32B-MultiTask, pairwise evaluation reaches 76.8 and 76.2 under the two orders but only 67.0 when both must be correct; pointwise evaluation reaches 70.6 with a 10.5% tie rate.

A verdict flip occurs when swapping response order changes the preferred response. The pairwise multitask model has a 17.0% verdict-flip rate, compared with 14.5% for separately trained J1-Qwen-32B, so multitask training improves accuracy without uniformly improving order sensitivity.
Figure 3 shows larger chosen-minus-rejected score differences in pairwise evaluation. The authors attribute this to access to both candidates within one input, which pointwise evaluation lacks.

Random-order accuracy, position-consistent accuracy, verdict flips, and ties on PPE Correctness.

Absolute scores and chosen-minus-rejected score differences for multitask J1.

Figure 4 shows that averaging scores over more sampled generations improves position-consistent accuracy and reduces ties or verdict flips for both 8B and 70B models. Additional results distinguish score averaging from majority voting over verdicts: 32-sample pointwise score averaging raises J1-Llama-70B from 65.0 to 74.8, whereas pairwise majority voting raises its reported accuracy from 72.9 to 73.7.

Effects of additional inference-time samples on accuracy and tie or verdict-flip rates.

Table 4 reports an overall score of 61.8 with positive rewards for correct verdicts, 60.4 after adding negative rewards for incorrect verdicts, and 61.0 after adding a format reward. Thinking and Plan & Execution seed prompts, the initial instructions used to elicit evaluation reasoning, perform comparably at 61.8 and 62.1, respectively. Figure 5 shows rising training rewards, with most pairwise generations settling around 500 tokens and pointwise generations typically using 300–400 tokens.

Qualitative examples show reference-answer creation and error diagnosis. In Figure 6, J1 checks that the sum over all two-element subsets of {1,2,3,4,5,6} is 105 and identifies the rejected response’s incorrect total of 47.
These examples show structured evaluation behavior, but the paper does not quantify its prevalence or establish whether the generated reasoning faithfully explains the model’s decisions.

Reward-scheme and seed-prompt ablations for Pairwise-J1 8B.

Training reward, generation length, and validation reward for J1-Llama-8B variants.

The paper situates J1 among traditional scalar reward models, judges that generate critiques or scores, and concurrent approaches that train reasoning judges with reinforcement learning. Its distinguishing combination is synthetic-only preference training, explicit response-order analysis, pointwise scoring learned from pairwise supervision, and joint pointwise–pairwise training.

The related-work discussion contrasts earlier prompting, rejection-based fine-tuning, which trains on selected self-generated reasoning, and offline DPO with online optimization of generated judgments. Reinforcement learning for reasoning judges is shared with several concurrent methods.

6 Conclusion

The conclusion presents a training recipe: construct preference pairs, reward judgments against their labels, and optimize generated reasoning and decisions online. Experiments at 8B, 32B, and 70B support gains over the corresponding base models, while the multitask formulation supports both comparative evaluation and independent response scoring.

Reproducibility Statement

The Reproducibility Statement identifies open-weight Llama and Qwen foundations, appendix prompts for training and data generation, a training implementation based on the open-source verl repository, and hyperparameters in Appendix C. It does not provide a dedicated J1 code or checkpoint URL.

Appendix

A J1’S THINKING TRACES provides additional examples for a verifiable geometry problem and a non-verifiable writing prompt. The geometry example diagnoses an incorrect triangle-area calculation; the writing example compares instruction relevance and unnecessary content. These are representative illustrations rather than a systematic error analysis.
B PROMPT TEMPLATES supplies prompts for verdict-only pairwise evaluation, plan-and-execution evaluation, pairwise scores, scores plus verdict, independent pointwise scoring, and synthetic rejected-response generation. Score prompts specify a range of 0 to 10 with up to 0.1 precision. Figures 9 and 10 identify the Thinking and EvalPlanner-style prompts, respectively, although Table 4 reverses these figure references.
C EXPERIMENTAL SETUP specifies 5 rollouts, sampled evaluations, per prompt at temperature 1.0, learning rate 1e−6, batch size 512, and maximum sequence length 4096 tokens for both inputs and outputs. Pairwise J1-Llama-70B later uses 3e−7; the selected KL coefficient, controlling divergence from the reference model, is 0.01 for J1-Llama-8B and 0 for J1-Llama-70B. Training uses 8×A100, 32×A100, and 64×A100 graphics processing units for the 8B, 32B, and 70B models, respectively; inference uses 8×A100, tensor parallelism 8, which divides model computations across eight devices, and maximum generation length 4096 tokens.
D ADDITIONAL RESULTS examines output formulations, order-bias mitigation, decoding, training regularization, a scalar baseline trained on the same data, and inference-time aggregation. In Table 8, the consistency-reward variant reaches PPE Correctness position-consistent accuracy 43.9 with verdict flips 28.7, compared with 39.1 and 36.8 for both-order training. Table 9 reports greedy decoding, selecting the most likely next token, versus temperature-0.6 sampling results of 72.9 versus 72.8 ± 0.1 for J1-Llama-70B and 76.8 versus 77.0 ± 0.1 for the multitask model across 8 sampling seeds; Table 11 reports gains over the same-data scalar baseline on four of five benchmarks.

Brief Thoughts

The strongest evidence combines improvements over corresponding base models with a direct analysis of response-order failures. The distinction between random-order and position-consistent accuracy is operationally useful: a high single-order score can coexist with substantial verdict instability.

The experiments support benchmark gains from learning constructed preferences, but do not establish universally correct subjective judgments. RewardBench also serves as the model-selection criterion, and the reported experiments do not test improvements from using J1 to train a downstream response-generating model. Generated reasoning and multiple-sample aggregation require additional computation, while the paper provides no latency or cost comparison.

Some numerical reporting requires caution. The main text gives an EvalPlanner comparison of 70.2 → 76.6 and a 6.8% gain, whereas Table 1 gives 76.8; Tables 7 and 10 report an Overall value of 63.9 for the verdict variant even though its displayed benchmark values match the 61.8 row in Table 4. These discrepancies leave the affected aggregate values unresolved; the comparisons above retain the explicitly identified table values.