HelpSteer3: Human-Annotated Feedback and Edit Data to Empower Inference-Time Scaling in Open-Ended General-Domain Tasks | Summary

This article explains the key points of HelpSteer3: Human-Annotated Feedback and Edit Data to Empower Inference-Time Scaling in Open-Ended General-Domain Tasks.

• 2025-03-06 (arXiv)
• Wang, Zhilin, Zeng, Jiaqi, Delalleau, Olivier, Egert, Daniel, Evans, Ellie, Shin, Hoo-Chang, Soares, Felipe, Dong, Yi, Kuchaiev, Oleksii.
• NVIDIA
• Paper

Read this article in Korean

Summary

• HelpSteer3 addresses how to improve language-model answers to open-ended tasks without relying on a verifiable correct answer. With dedicated 70-billion-parameter Feedback and Edit models that critique and revise drafts, its scaled system reaches 92.7 on Arena Hard, a model-judged benchmark of challenging user queries, using Llama-3.1-Nemotron-70B-Instruct for initial answers and Llama-3.3-Nemotron-70B-Select for final selection, compared with reported scores of 90.4 for OpenAI o1-preview-2024-09-12 and 92.3 for DeepSeek R1 as of 5 Mar 2025.
• The dataset contains 81,642 responses with human feedback, 14,461 original responses with human edits, and 3,274 original responses supporting edit preferences. On Arena Hard, dedicated feedback and editing models outperform prompting the initial model to critique and revise itself.
• Inference-time scaling, increasing computation while generating an answer, operates here through additional drafts, actionable feedback, and candidate edits. At approximately 16 times the generated tokens of greedy decoding, which selects the highest-probability next token at each step, the combined setup scores 91.0 against 88.5 for selecting the best of 16 initial answers. Replacing the final selector raises the score to 92.7, although the paper does not establish a statistically reliable advantage over DeepSeek R1.

1 Introduction

The introduction argues that detailed natural-language feedback can identify what to change in a response more precisely than a preference or scalar rating. Many methods for training inference-time reasoning depend on verifiable answers, leaving a gap for tasks such as research proposals, speeches, and software planning.

• The proposed system separates initial response generation, feedback generation, and editing into three model roles.
• The contribution combines human-annotated feedback and edits, dedicated Feedback and Edit models, and experiments that scale each generation stage.

The chart places the final 92.7 system score alongside 90.4 for o1-preview and 92.3 for DeepSeek R1. It summarizes reported benchmark standings as of 5 Mar 2025, but does not show uncertainty or establish statistical separation between nearby scores.

2 Dataset

HelpSteer3 covers General, STEM, Coding, and Multilingual tasks; STEM denotes science, technology, engineering, and mathematics. The released data supports learning to produce feedback, apply feedback through edits, and distinguish satisfactory edits from unsatisfactory ones.

2.1 Data Collection

Coding and Multilingual prompts come from ShareGPT, a collection of voluntarily shared user conversations, while General and STEM prompts come from WildChat, a larger conversation dataset. Two responses are generated per prompt using models of different sizes and capabilities at temperature 0, top p 0.9, and a maximum of 3072 tokens; temperature controls sampling randomness, and top p restricts sampling to a cumulative-probability threshold. Intermediate assistant turns are regenerated rather than copied from ChatGPT conversations.

• Potentially unsafe material is filtered, and annotators are asked to skip remaining unsafe tasks and tasks containing personally identifiable information.
• More than 7,000 annotators from more than 80 countries participate through Scale AI and Translated. Each task receives 3–5 annotators, with relevant qualifications required for STEM and Coding.
• Feedback is written in English, including for multilingual responses, and is requested to contain 2–10 sentences, or 50–250 words, beginning with a five-level helpfulness judgment. Annotators may consult internet resources for fact-checking but may not use large language models.
• Separate annotator pools edit responses using feedback from the three annotators whose helpfulness judgments agree most closely. Responses rated perfectly helpful or not helpful by most annotators are excluded from editing because the authors consider them either unsuitable for meaningful improvement or in need of complete rewriting.

2.2 Data Pre-processing

Feedback Demonstration retains three closely agreeing critiques per response and removes cases with substantial remaining disagreement. Edit Demonstration removes perfectly helpful feedback and uses Llama-3.3-70B-Instruct to assess whether annotator-written change summaries address each critique, providing distant supervision, an indirect label, for which feedback was applied.

• All permutations of the retained feedback order are included to teach editing that does not depend on critique order.
• Edit Preference pairs good edits with two equally represented negative types: edits that depart from the supplied feedback and unchanged copies of the original response. This dataset covers only General and STEM.
• The unchanged-copy negative addresses a proposed training shortcut: token-level imitation can reward preserving most of an original answer even when feedback calls for changes.

2.3 Data Analysis

Feedback Demonstration contains 81,642 responses, Edit Demonstration contains 14,461 original responses, and Edit Preference contains 3,274 original responses. More than half of each demonstration dataset comes from specialist categories, and the dataset covers 14 programming languages and 13 natural languages.

• Feedback averages 437.8 characters overall, with longer Coding feedback at 518.6 characters and shorter Multilingual feedback at 339.2 characters.
• Edit Demonstration averages 2.3 feedback items per edit. Coding edits increase response length by 782.0 characters on average, while average changes in General, STEM, and Multilingual remain within 100 characters.
• In Edit Preference, bad edits shorten answers by 122.1 characters on average, whereas good edits increase length by 32.5 characters. The authors attribute the shortening partly to editors making unnecessary deletions, but the statistics alone do not establish its cause.

The category breakdown shows that specialist data forms more than half of both demonstration datasets. Coding edits increase answer length much more than edits in the other categories, making response length relevant when interpreting downstream results.

The 3,274 original responses are limited to General and STEM. Bad edits shorten responses by 122.1 characters on average, compared with a 32.5-character increase for good edits, documenting an association between edit quality and length change.

3 Experiments

The experiments evaluate whether trained feedback and editing models improve existing instruction-following models, which are trained to answer user requests. They also isolate component contributions, vary inference-time computation, and test whether selected system outputs can train a model to improve without running the full pipeline.

3.1 Evaluation

Evaluation uses AlpacaEval 2.0 Length Controlled, a model-judged preference measure adjusted for answer length; MT Bench, a scored collection of multi-turn questions; and Arena Hard. The paper treats these as approximately easy, medium, and hard evaluations, respectively, and reports mean MT Bench response length to contextualize changes in verbosity.

• The default system samples 10 critiques per initial answer at temperature 0.7 and top p 0.9, removes critiques judging the answer perfectly helpful, and randomly selects up to three remaining critiques.
• The edited answer is generated greedily from the prompt, initial answer, and selected critiques. MT Bench is also used for checkpoint selection, choosing a saved training state by validation performance.

3.2 Model Training

Feedback and Edit models start from Llama-3.3-70B-Instruct and undergo supervised fine-tuning, learning to imitate annotated outputs. The Edit Reward Model, which scores edits, and the reinforcement-learning Edit model, which learns from those scores, both start from the supervised Edit model. Each dataset is divided into 95% training and 5% validation.

• Feedback fine-tuning runs for 1 epoch, one pass through the training data, with 1,858 steps at global batch size 128; step 1,400 is selected. Edit fine-tuning runs for 1 epoch, 385 steps at global batch size 128, with step 385 selected.
• The reward model uses Bradley-Terry modeling, a pairwise preference model, for up to 94 steps at global batch size 128; step 80 has the lowest validation loss. Each batch includes both bad-edit and unchanged-copy comparisons against good edits for the same contexts.
• Edit reinforcement learning uses REINFORCE Leave One Out, a reward-guided policy optimization method, for up to 47 steps at global/rollout batch size 64. Step 45 is selected.

4 Results

With the default Feedback + Edit system, Llama-3.1-Nemotron-70B-Instruct improves from 8.98 to 9.16 on MT Bench, 57.6 to 62.8 on AlpacaEval 2.0 Length Controlled, and 85.0 to 87.0 on Arena Hard. Llama-3.3-70B-Instruct improves from 8.29 to 9.07, 35.0 to 36.9, and 62.4 to 74.8 on the same evaluations.

• Mean MT Bench answer length rises from 2199.8 to 2614.4 characters for Nemotron and from 1827.6 to 2362.1 for Llama-3.3.
• The AlpacaEval gains persist after its length adjustment. The reported Arena Hard confidence intervals, ranges expressing uncertainty in the estimated scores, counsel caution when interpreting smaller differences.

Both starting models improve on all three quality evaluations after Feedback + Edit. The larger Arena Hard increase for Llama-3.3-70B-Instruct, from 62.4 to 74.8, shows that the size of the gain depends on the initial model.

5 Ablation Studies

The ablations show that component choices affect benchmarks differently. Prompted Self-Feedback + Self-Edit, using the initial model to critique and revise its own answer, raises Nemotron’s AlpacaEval score from 57.6 to 64.6 and MT Bench from 8.98 to 9.11, but lowers Arena Hard from 85.0 to 84.6 while producing substantially longer answers.

• Using trained feedback with prompted Self-Edit scores 85.4 on Arena Hard and 8.94 on MT Bench, below 87.0 and 9.16 for the dedicated Feedback + Edit system.
• The supervised-only Edit model leaves around 30% of responses unchanged despite actionable feedback. The authors report that this copying disappears after reinforcement learning; Arena Hard rises from 86.4 to 87.0, while AlpacaEval falls from 64.4 to 62.8.
• Training edits without feedback scores 67.4 on AlpacaEval but 84.5 on Arena Hard, compared with 64.4 and 86.4 for feedback-conditioned editing without reinforcement learning. These results favor feedback conditioning on the challenging evaluation, with different outcomes on simpler prompts.

The ablations produce different rankings across evaluations: editing without feedback has the highest AlpacaEval score shown, while the complete trained system has the highest Arena Hard and MT Bench scores. Feedback and dedicated editing therefore have task-dependent benefits in these experiments.

6 Scaling

The scaling study varies initial answers, effective feedback, meaning actionable suggestions, and candidate edits on Arena Hard. Best-of-N sampling, generating multiple answers and selecting the highest-scored one, supplies the baseline using Llama-3.1-Nemotron-70B-Reward as the selector.

• Increasing baseline candidates from 2 to 16 raises Arena Hard from 86.4 to 88.5. Applying Feedback-Edit to each candidate and selecting afterward raises the corresponding scores from 89.0 to 90.2.
• Increasing effective feedback from 2 to 16 raises Arena Hard from 87.0 to 89.0. The method generates 16, 32, or 64 critiques to obtain 4, 8, or 16 effective critiques, ranking them by criticism keywords: ‘However’, ‘improve’, ‘lack’, ‘benefit’, and ‘but’.
• Increasing edits per feedback set from 2 to 16 raises Arena Hard from 88.4 to 89.6. Initial-answer and edit sampling use temperature 0.7 and top p 0.9, followed by reward-based selection.

Each group increases a different candidate count from 2 to 16 and shows rising Arena Hard point estimates. The responses-per-prompt configuration reaches 90.2, above the Best-of-N baseline at 88.5, but the chart does not normalize every axis by computation or display confidence intervals.

Combining 8 initial responses with 16 effective feedback items each uses approximately 16 times the generated tokens of default greedy generation and scores 91.0, with reported 95% confidence-interval offsets of -1.0 and +1.2. Best-of-N 16 scores 88.5, with offsets of -1.2 and +1.5, at a similar generated-token budget.

• Replacing Llama-3.1-Nemotron-70B-Reward with Llama-3.3-Nemotron-70B-Select raises the combined system to 92.7, with 95% confidence-interval offsets of -1.2 and +0.9.
• OpenAI o1-preview-2024-09-12 scores 90.4, with offsets of -1.1 and +1.3. DeepSeek R1 scores 92.3 with no confidence interval reported, so its 0.4-point difference from the proposed system does not establish statistical superiority.
• At temperature 0.7, the authors observe little additional feedback diversity beyond 16 samples. For 32 critiques they combine 16 samples at 0.7 with 16 at 0.6; for 64 they use 16 each at 0.5, 0.6, 0.7, and 0.8.

Human Evaluation compares the initial Nemotron model against the optimally scaled system on 50 previously unused first-turn General prompts, with three independent judgments per prompt and eight in-house annotators overall. Six prompts are excluded because at least one annotator judged both answers invalid; among the remaining 44, the system produces 16 improvements, 21 ties, and 7 degradations.

• Answer order is randomly shuffled, and the overall preference is obtained by averaging the three judgments and rounding to the nearest preference rank.
• Improvement judgments comprise 13 slightly better, 2 better, and 1 much better; degradation judgments comprise 6 slightly worse and 1 worse.
• Preference explanations attribute improvements mainly to factuality, coherence, and conciseness. The small, General-only sample limits conclusions about broader human preference.

7 Distillation

Distillation, training a model to imitate the scaled system’s selected answers, uses 22,644 unique first-turn prompts, with 21,850 for training and 794 for validation. Three starting models receive 1 epoch of supervised fine-tuning, 170 steps at global batch size 128, so that inference can use a single model without the feedback-editing pipeline.

• Llama-3.1-8B-Instruct improves from 18.3 to 55.5 on Arena Hard, Llama-3.3-70B-Instruct from 62.4 to 88.8, and Llama-3.1-Nemotron-70B-Instruct from 85.0 to 88.4. Their selected checkpoints are at steps 120, 150, and 160, respectively.
• All three models also improve on MT Bench and AlpacaEval. For Llama-3.3-70B-Instruct, AlpacaEval rises from 35.0 to 61.6.
• Distillation removes the multi-stage inference procedure, although generated answers become longer. Its gains are generally smaller than those obtained by applying the optimally scaled system at inference time.

Distillation improves all three starting models across the quality evaluations while increasing answer length. The 8B model's Arena Hard gain from 18.3 to 55.5 shows that selected pipeline outputs can improve a smaller model without retaining the pipeline at inference time.

8 Conclusion

The paper concludes that dedicated Feedback and Edit models support inference-time scaling on open-ended general-domain tasks. Its strongest configuration reaches 92.7 on Arena Hard, while the default pipeline and distilled models improve point estimates across all three evaluations.

Ethical Considerations

Ethical Considerations describes adjustable inference computation, separately served model components, and parallel candidate generation as potential deployment advantages. The authors estimate approximately twice the latency of greedy generation when candidates run in parallel, but do not provide a controlled latency benchmark.

• HelpSteer3 is released under CC-BY-4.0 at https://huggingface.co/datasets/nvidia/HelpSteer3. The models remain subject to their respective Llama Community License Agreements.
• The authors report vendor ethical review, annotator consent for model training, and measures to prevent model-assisted annotations. They also describe vendor compensation, worker-support, and appeal policies.
• Selecting understandable feedback is proposed as a route to personalized editing; personalization is not experimentally evaluated.

Limitations

The authors identify critique oversampling, filtering, and keyword-based ranking as computationally inefficient. The data also omits prompts requiring responses longer than 2000 words, inputs exceeding 4,000 words across conversation turns, or knowledge after July 2024.

• ShareGPT and WildChat queries were collected before April 2023 and April 2024, respectively, limiting coverage of newer requests.
• The scaling experiments concentrate on Arena Hard, where more than 50% of prompts involve coding. This narrows the evidence for scaling across the full range of open-ended general-domain tasks.
• The paper presents Feedback-Edit as complementary to other inference-time scaling methods. It does not establish a universally optimal allocation of computation.

Appendix

• A Prompt Preprocessing: Nemotron 4 340B Instruct identifies programming and natural languages, followed by manual normalization and sampling within language groups. General and STEM curation excludes non-English requests, code-generation requests, prompts shorter than 10 characters, and identity-related prompts. BERTopic, a topic-modeling method, groups approximately 4,400 topics; sampling is uniform across topics, with prompts rated 4 or 5 out of 5 in complexity oversampled.
• B Prompt Templates: The appendix supplies templates for language identification, complexity scoring, checking whether feedback was applied, feedback generation, and editing with or without feedback. Editing normally receives up to three feedback items; when none is available, the literal filler <None> is used.
• C Bad Edit Details: Researchers inspecting disagreements found feedback annotators more often correct than edit annotators. The authors suggest that editors have an incentive to avoid issues requiring extensive revisions; this explanation is based on their inspection of disagreement samples.
• D Further Dataset Statistics: Language coverage is uneven despite spanning 14 programming languages and 13 natural languages. Python accounts for 39.0% of Coding feedback and 47.4% of edits, while Chinese accounts for 29.6% of Multilingual feedback and 32.4% of edits. Annotator geography also varies substantially across categories.
• E Illustrative Example: The prompt ‘Tourism marriage’ demonstrates how supplied feedback constrains a good edit. The good edit removes an unnecessary point identified by feedback, whereas the bad edit introduces additional changes beyond that feedback; a perfectly helpful critique is excluded from editing.
• F Aligned Model Evaluation Details: MT Bench contains 80 two-turn questions, or 160 prompts, uses GPT-4-0125-Preview to assign scores from 1 to 10, and limits greedy answers to 1024 tokens. AlpacaEval contains 805 first-turn instructions and uses GPT-4-1106-turbo with length correction; Arena Hard contains 500 first-turn instructions and compares against GPT-4-0314 using GPT-4-1106-preview. AlpacaEval and Arena Hard limit answers to 2048 tokens.
• G Training Details: Experiments use Nemo-Aligner, a model-alignment training framework, with nvcr.io/nvidia/nemo:24.05.llama3.1, a maximum sequence length of 4096, AdamW, a weight-decay optimization method, and 10 warmup steps; longer sequences account for less than 3% of each dataset and are discarded. Selected constant learning rates are 1e-6 for Feedback/Edit supervised fine-tuning and 8B distillation, 3e-6 for Edit Reward Model training and 70B distillation, and 5e-7 for Edit reinforcement learning, with a Kullback-Leibler divergence coefficient, constraining departure from the reference model, of 0.01. Reinforcement learning samples 4 responses per prompt with up to 2048 generated tokens at temperature 1 and top p 0.9; vLLM, an inference framework, uses the 0.6.6.post1 container.
• H Compute requirements: Training 70B models requires 80 H100-equivalent node-hours for Feedback supervised fine-tuning, 16 for Edit supervised fine-tuning, 5 for Edit Reward Model training, 48 for Edit reinforcement learning, and 16 for distillation. A node-hour measures one compute node running for one hour; each node contains 8 H100-80GB SXM graphics processing units.
• I Example Feedback by prompting Llama-3.1-Nemotron-70B-Instruct: A single prompted critique of a literature-review answer discusses strengths and weaknesses, including citations, depth, methodology, and implications. It illustrates the length and breadth of prompted feedback, rather than independently verifying the critique’s factual accuracy.
• J Example Feedback by Feedback Model: Three shorter critiques of the same literature-review answer identify redundancy, insufficient discussion of challenges, and references with limited relevance. They illustrate how sampled critiques can offer different actionable suggestions.
• K Additional Comparisons: Prompting ‘Let’s think step by step’ lowers Nemotron’s Arena Hard score from 85.0 to 81.2, and CritiqueLLM, a separate critique-generation model, with prompted Self-Edit scores 82.2. Applying the dedicated pipeline to Gemma-2-27B-IT raises Arena Hard from 47.5 to 76.2. In matched comparisons without reinforcement learning, 8B Feedback and Edit models generally perform worse than their 70B counterparts and fail to improve the starting models consistently.

Brief Thoughts

The component comparisons provide the clearest evidence for dedicated feedback and editing: both prompted self-editing and editing without feedback score lower on Arena Hard than the complete system. The headline ranking requires more caution because the final response selector changes, the gap to DeepSeek R1 is only 0.4 points with no reported interval for that comparator, and the human evaluation contains only 44 retained prompts. A deployment decision would require measuring the scaled pipeline and distilled models under the same latency and computation budget. Similar generated-token counts in one comparison do not establish equal end-to-end serving cost.

Python and JavaScript/HTML/CSS dominate the Coding subset, while Chinese is the largest Multilingual group. The distribution documents broad language coverage with uneven representation.

Annotator locations differ sharply by category and role: the US supplies 57.3% of General feedback, while Mexico supplies 43.9% of General edits. Geographic breadth coexists with concentrated contributions in individual subsets.

Feedback supervised fine-tuning is the largest listed training stage at 80 H100-equivalent node-hours, followed by Edit reinforcement learning at 48. These training costs on eight-GPU nodes do not quantify the scaled system's serving cost.

Additional comparisons extend the Arena Hard gains to Gemma-2-27B-IT, whose score rises from 47.5 to 76.2. The model-size comparison trains both 8B and 70B Feedback/Edit variants without reinforcement learning; 70B variants perform better, although this does not establish how 8B variants would behave after reinforcement learning.