Inference-Time Scaling for Generalist Reward Modeling | Summary

This article explains the key points of Inference-Time Scaling for Generalist Reward Modeling.

• 2025-04-03 (arXiv)
• Liu, Zijun, Wang, Peiyi, Xu, Runxin, Ma, Shirong, Ruan, Chong, Li, Peng, Liu, Yang, Wu, Yu.
• DeepSeek-AI, Dept. of Computer Sci. & Tech., Tsinghua University, Institute for AI Industry Research (AIR), Tsinghua University
• Paper

Read this article in Korean

Summary

• Reward models score language-model responses, but improving these judgments with additional inference computation is difficult when queries lack directly verifiable answers. Across Reward Bench, PPE Preference, PPE Correctness, and RMB—benchmarks of response preferences and correctness—DeepSeek-GRM-27B reaches an overall score of 72.8 with 32 sampled evaluations and a learned evaluator that filters them, compared with 69.9 under greedy decoding, which selects the most probable next token at each step, and 71.3 for the reported GPT-4o results.
• Self-Principled Critique Tuning (SPCT) trains a generative reward model, which expresses evaluations in text, to produce principles, meaning evaluation criteria, followed by critiques and an individual score for each response. It combines rejective fine-tuning, which retains selected correct evaluation sequences, with online reinforcement learning, which updates the model from newly sampled evaluations using rewards for correct judgments.
• Parallel sampling and score aggregation improve DeepSeek-GRM-27B more than the reproduced generative and semi-scalar baselines, where semi-scalar models combine textual critiques with numerical reward outputs. A meta reward model, a learned evaluator of the generated evaluations, further improves aggregate performance by selecting samples before aggregation. Generative evaluation remains expensive, reference-free correctness evaluation trails the strongest reproduced scalar baseline, and the model-size comparison does not establish an equal-compute advantage.

1 Introduction

Generalist reward modeling must handle diverse response formats and evaluation criteria across domains where reference answers are often unavailable. The introduction identifies four requirements: flexible inputs, accurate rewards across domains, improved rewards with more inference computation, and learned behaviors that support this scaling.

• The proposed pointwise approach assigns each response its own score through generated text, supporting single, paired, and multiple responses.
• The authors train DeepSeek-GRM-27B from Gemma-2-27B and investigate whether generating more principles and critiques improves judgment through aggregation.

DeepSeek-GRM improves as the number of sampled evaluations increases, and meta-guided aggregation has the strongest aggregate scaling curve. The plot compares sampled evaluations with fixed public-model reference scores, so it establishes a performance comparison without controlling computational cost.

2 Preliminaries

The preliminaries separate the representation of a reward from the scoring pattern used to evaluate responses. This distinction motivates an approach that can generate varied evaluations while accommodating different numbers of candidate responses.

2.1 Comparisons of Different RM approaches

Scalar reward models output numerical values; semi-scalar models generate text alongside a numerical reward; generative reward models produce textual evaluations from which scores can be extracted. Pointwise scoring assigns an individual score to each response, whereas pairwise scoring selects a preferred response, usually from two candidates.

• Bradley-Terry models learn pointwise scalar scores from paired preferences; PairRM produces a relative scalar judgment; CLoud produces scores conditioned on critiques; and LLM-as-a-Judge uses a large language model to generate a textual preference.
• The paper focuses on sampling-based scaling: fixed scalar outputs offer little diversity across repeated evaluations, while generated critiques and scores can vary.
• DeepSeek-GRM normally extracts integer scores from 1 to 10. Pointwise scores permit ties within individual evaluations and support inputs containing one or multiple responses.

The diagram separates reward representation from scoring pattern, explaining the authors' choice of pointwise generated scores: this combination supports varied input counts and multiple sampled evaluations. Its scalability labels concern the paper's sampling-and-aggregation setting.

2.2 Boosting Reward Quality with Principles

Principles are explicit evaluation criteria that guide a critique. In preliminary experiments on Reward Bench Chat Hard, a subset of difficult chat comparisons, and IFEval, an instruction-following evaluation subset of PPE, GPT-4o-2024-08-06 generates principles and scores four times per example; principles associated with correct preference judgments are retained for subsequent evaluation.

• For GPT-4o, unfiltered self-generated principles change Chat Hard from 76.1 to 75.9 and IFEval from 56.0 to 55.6; filtered principles raise them to 77.8 and 57.5.
• For Gemma-2-27B-it, filtered principles improve Chat Hard from 59.1 to 68.0 and IFEval from 56.1 to 57.3. Unfiltered self-generated principles improve Chat Hard to 64.0 but reduce IFEval to 55.8.
• These results motivate learning to generate useful principles, but filtering uses preference labels and therefore does not demonstrate a label-free improvement.

Principles selected using correct preference judgments improve both tested models on both subsets, whereas unfiltered principles have mixed effects. Gemma-2-27B-it benefits from unfiltered principles on Chat Hard but loses slightly on IFEval, showing that the effect depends on the model and task.

3 Self-Principled Critique Tuning (SPCT)

SPCT trains a pointwise generative reward model in two phases: rejective fine-tuning establishes evaluation behavior and output format, then rule-based online reinforcement learning improves the generated principles and critiques. The method aims to learn evaluations that benefit from aggregating multiple samples at inference time; training rewards individual judgment correctness.

3.1 Unpinning Principles from Understanding to Generation

The model generates query-specific principles as part of its evaluation instead of receiving them only as fixed preprocessing input. Principle generation and critique generation share the same language model and output head, allowing training to modify both behaviors together.

• Each sampled evaluation can introduce different criteria and weights before producing a critique and response scores.
• The proposed explanation for scaling is that aggregation incorporates more judgment perspectives and produces finer score distinctions. The paper does not establish that sampled principles are independent or collectively exhaustive.

3.2 Rule-Based Reinforcement Learning

Rejective fine-tuning mixes general instruction data with sampled evaluation sequences, discarding incorrect sequences and examples for which every non-hinted sample is correct. Hinted sampling optionally supplies the labeled best response to recover training examples for difficult queries, although the authors observe shortcuts in some resulting critiques.

• For multiple responses, correctness requires the uniquely labeled best response to receive a strictly higher score than every alternative; for a single response, its score must match the label. Hinted sampling generates one evaluation per example and rejects it only if incorrect.
• Online training uses group relative policy optimization (GRPO), which normalizes rewards within groups of sampled outputs, with an outcome reward of +1 for a correct judgment and −1 otherwise.
• No separate format reward is used. A Kullback-Leibler (KL) penalty, which constrains deviation from a reference policy, helps preserve format and limit severe performance deterioration in particular domains.

The training and inference paths connect generated principles to critiques, extracted scores, and correctness feedback. At inference, aggregation combines several evaluation sequences, while the meta reward model selects sequences before their scores are combined.

4 Inference-Time Scaling with SPCT

At inference time, DeepSeek-GRM samples multiple sequences of principles and critiques and sums their pointwise scores for each response. Candidate responses are shuffled before sampling to reduce positional bias and increase diversity.

• Summing scores from a small discrete scale expands the range of possible aggregate rewards and allows finer distinctions than a single evaluation.
• The meta reward model is a scalar classifier trained with binary cross-entropy, a loss for binary prediction, to predict whether a generated evaluation makes the correct judgment. Its input includes the query, candidate responses, principles, and critique.
• Meta-guided voting retains the evaluations with the highest meta scores before summing their response scores; by default it retains half of the sampled evaluations. Training examples include non-hinted samples from rejective fine-tuning and samples from the DeepSeek-GRM model being guided.

5 Results on Reward Modeling Benchmarks

The experiments evaluate overall judgment quality, gains from additional inference samples, component ablations, and comparisons with larger model backbones. Reproduced baselines provide comparisons within the Gemma-2-27B model family, while reported public-model scores supply broader reference points.

5.1 Experiment Settings

Reward Bench tests chat, reasoning, and safety preferences; PPE contains Preference and verifiable Correctness subsets; RMB tests helpfulness and harmlessness with paired and multiple responses; ReaLMistake tests errors in single responses. Evaluation uses best-response selection accuracy for the first three benchmarks and the area under the receiver operating characteristic curve (ROC-AUC), which measures discrimination across classification thresholds, for ReaLMistake.

• The reproduced baselines are LLM-as-a-Judge, DeepSeek-BTRM-27B, CLoud-Gemma-2-27B, and DeepSeek-PairRM-27B, using Gemma-2-27B and compatible training data and settings.
• Default evaluation uses greedy decoding; inference-time scaling uses temperature = 0.5, which controls sampling randomness. The meta reward model also uses Gemma-2-27B.
• Larger backbones include DeepSeek-V2-Lite, DeepSeek-V2.5, and DeepSeek-V3. These are mixture-of-experts models, which route computation through subsets of model parameters.

5.2 Results and Analysis

DeepSeek-GRM-27B scores 69.9 overall under greedy decoding, compared with 68.8 before online reinforcement learning and 67.8–69.0 for the reproduced baselines. Under temperature = 0.5 sampling, its overall score rises from 67.9 with one sample to 70.6 with eight and 71.0 with 32; meta-guided voting reaches 72.0 and 72.8 with eight and 32 samples, respectively.

• At eight samples, plain aggregation gains 2.7 points over one-sample evaluation, versus 0.6 for LLM-as-a-Judge and 0.3 for CLoud-Gemma-2-27B. These gains use the one-sample stochastic baseline, rather than the 69.9 greedy-decoding result.
• Removing principle generation lowers greedy performance from 69.9 to 67.5 and eight-sample performance from 70.6 to 68.0. Online reinforcement learning without rejective sampled critique data improves 66.1 to 68.7; removing general instruction data from rejective fine-tuning lowers its score from 68.8 to 63.3.
• At 32 samples, meta-guided DeepSeek-GRM scores 63.2 on PPE Correctness, below DeepSeek-BTRM-27B at 66.7. On RMB, meta guidance lowers the eight-sample score from 69.5 to 68.8, although it improves the aggregate score.
• On Reward Bench, 32-sample plain voting reaches 88.5, comparable to the 671B model’s 88.4, while eight-sample meta-guided voting reaches 89.8. Larger models receive different training, and the comparison does not normalize total training and inference computation.
• DeepSeek-R1-0120 is evaluated on only 300 downsampled Reward Bench examples because of inference cost. Its weaker overall result supports a limited comparison, while its strong Reasoning subset result shows that extended reasoning can help specific judgment tasks.

The best aggregate score is 72.8 with 32 samples and meta guidance, but individual benchmarks retain different leaders. DeepSeek-BTRM-27B scores 66.7 on PPE Correctness versus 63.2 for the meta-guided model, and reported GPT-4o results remain higher on RMB.

The +4.9-point gain for 32-sample meta-guided evaluation is measured from the 67.9 one-sample stochastic result. The separate 69.9 greedy result uses a different decoding condition, which changes the apparent magnitude of the scaling gain.

Removing principle generation reduces both greedy and eight-sample performance, while online reinforcement learning improves a model trained without rejective sampled evaluations. Retaining eight or 16 of 32 evaluations gives similar aggregate scores, 72.7 and 72.8; retaining only one gives 71.5.

On Reward Bench, the 27B model with eight meta-guided samples exceeds the plotted 671B greedy result, and 32-sample plain voting is comparable to it. Different model backbones, unequal training recipes, and the 300-example DeepSeek-R1 evaluation limit conclusions about computational efficiency.

6 Related Work

The related work covers generative reward models trained with offline or online reinforcement learning, evaluators augmented with tools or external knowledge, and inference-time scaling through sampling or extended reasoning. The paper emphasizes training adaptive principle generation for scalable generalist reward evaluation.

• The authors do not establish an effective method for inducing long-horizon reward generation, meaning extended reasoning before judgment, for generalist evaluation. They leave its combination with principle-guided evaluation to future work.
• Using scalable reward models to improve a separate response-generating policy is discussed as a possible application rather than demonstrated here.

7 Conclusion and Future Work

The conclusion attributes improved reward quality and sampling-based scalability to adaptive principles and critiques learned through SPCT, with additional gains from meta-guided voting. Proposed applications include online reinforcement-learning reward systems, joint inference-time scaling with response-generating policies, and offline model evaluation; these remain future directions.

Ethics Statement

The Ethics Statement identifies risks from biased training data, automatically generated criteria, and potentially unfaithful critiques, whose explanations may not reliably reflect the judgment process. The authors advocate continued human oversight, public scrutiny of released models, and evaluation across benchmarks and practical settings. More balanced performance across benchmark domains does not establish the absence of social bias.

Appendix

• A Additional Related Work reviews Constitutional AI, which uses guiding principles to align model behavior, alongside scalar and semi-scalar reward models. B Limitations and Future Directions identifies generative evaluation cost and weaker performance on verifiable tasks as current limitations. It proposes tool use, separating principle generation from critique generation, offline evaluation, and longer reasoning as future research; evaluating intermediate reasoning steps with DeepSeek-GRM is also suggested but not extensively tested.
• C Implementation Details and C.1 Model Training report 1256K rejective fine-tuning examples, comprising 1070K general instruction examples and 186K sampled evaluations, plus 237K reinforcement-learning examples. Data combine internal sources with MATH, a mathematical problem dataset, and the preference datasets UltraFeedback, OffsetBias, Skywork-Reward-Preference-80K-v0.2, and HelpSteer2-Preference. Sampled evaluations are generated with DeepSeek-v2.5-0905 using three non-hinted samples per example; training uses group size 4 and KL coefficient 0.08 for the 27B model, versus 0.002 for the 16B model.
• C.1 Model Training reports 128 A100 graphics processing units (GPUs), 900 steps per stage, learning rates of 5 × 10⁻⁶ and 4 × 10⁻⁷, and batch sizes of 1024 and 512 for rejective fine-tuning and reinforcement learning, respectively; the stages take 19.2 h and 15.6 h. Models larger than 27B receive only 50K rejective sampled examples and no rule-based reinforcement learning because of resource constraints. C.2 Baseline Implementation documents method-specific restrictions, including pairwise-only reinforcement-learning data for LLM-as-a-Judge and exclusion of single-response rating data when training the scalar value heads, which map model representations to numerical rewards.
• D Experiment Details, D.1 Hyper-Parameters, D.2 Benchmarks, and D.3 Detailed Results supply evaluation settings and benchmark breakdowns. Reward Bench prior sets, additional preference datasets reported separately, are excluded from overall score calculations; reported GPT-4o scores use gpt-4o-2024-08-06 for Reward Bench and PPE and gpt-4o-2024-05-13 for RMB. On RMB best-of-N subsets, which require selecting the best of multiple responses, baselines evaluate the labeled best response against each alternative separately, whereas DeepSeek-GRM receives the full response list; the appendix examines this protocol difference.
• E Additional Experiments and E.1 Input Flexibility of the Pointwise GRM Approach test multiple, single, and reference-supported inputs. E.1.1 Generating Rewards for Many Responses finds pair-versus-list differences of 0.2 points for helpfulness and 0.5 for harmlessness; E.1.2 Generating Rewards for Single Responses reports ReaLMistake ROC-AUC of 72.2 for DeepSeek-GRM-27B and 74.4 with eight samples, versus 74.3 for GPT-4o-2024-08-06. E.1.3 Generating Rewards with Reference raises PPE Correctness from 59.8 to 91.6 by supplying ground-truth references, materially changing the information available to the evaluator.
• E.2 Transferability of Generated Principles shows GPT-4o reaching 78.1 on Chat Hard and 58.3 on IFEval with DeepSeek-GRM-generated principles, compared with 76.1 and 56.0 by default. E.3 Generalization beyond Training Data finds that removing MATH reward-modeling data lowers the 27B model’s Reward Bench score from 86.0 to 83.0, including a Chat Hard decline from 78.3 to 70.4, while Chat improves from 94.1 to 96.1. E.4 Response Length Analysis for Rule-Based RL reports increases from 260 to 376 tokens on Reasoning, unchanged 259-token length on Chat Hard, and a decrease from 245 to 218 on Safety; the comparison with DeepSeek-R1 uses different tokenizers, which divide text into model-specific units.
• F Qualitative Analysis and F.1 Case Study illustrate a correct distinction missed by a scalar baseline, a programming judgment corrected through aggregation, and a failure involving real-time information and criterion weighting. F.2 Failure Mode Analysis manually examines 10 incorrect examples from each of four benchmark groups: incorrect critiques account for 4, 5, 7, and 5 cases, while imbalanced principle weights account for 4, 4, 3, and 2, respectively. The remaining categories are improper principles and annotator disagreement with benchmark labels; this small sample of errors supports a qualitative diagnosis rather than population-level failure rates.
• G Prompt Templates provides DeepSeek-GRM (Default), DeepSeek-GRM (Training on Rating Single Response), Meta RM, and LLM-as-a-Judge templates. The default model generates Specific Criteria, Analysis, and integer Scores from 1 to 10; single-response training uses binary correctness scores, while ReaLMistake evaluation uses the usual score range. The Meta RM receives candidate responses and generated principles and critiques, and LLM-as-a-Judge outputs the preferred response index.

Brief Thoughts

The principle-removal ablation and weaker scaling of the fine-tuning-only model support the contribution of learned criteria generation to repeated evaluation. They do not isolate principle quality from every other effect of training. The results establish improved benchmark judgments, while downstream policy improvement and equal-compute superiority remain untested.