Summary explanation: Progress Reward Modeling for Robotic Learning: A Comprehensive Survey

Contents

• Summary
• 1 Introduction
• 2 Interface: Input and Output of Progress Models
  • 2.1 Input Interface: How Is the Current Task State Represented?
  • 2.2 Input Interface: How Is the Task Goal Specified?
  • 2.3 Output Interface: What Does the Model Predict?
• 3 Methods: How Are Progress Rewards Constructed?
  • 3.1 Frozen Foundation Models as Semantic Reward Scorers
  • 3.2 Learning Rewards from Temporal and Relative Supervision
  • 3.3 Instruction-Tuned Progress Prediction
  • 3.4 Programmatic Reward Construction
• 4 Data and Benchmarks
  • 4.1 Data: How Is Progress Supervision Constructed?
  • 4.2 Benchmarking Progress Fidelity
  • 4.3 Benchmarking Robustness and Generalization
  • 4.4 Benchmarking Downstream Utility
• 5 Limitations and Future Directions
• Appendix
• Brief Thoughts

This article explains the key points of Progress Reward Modeling for Robotic Learning: A Comprehensive Survey.

• 2026-07-22 (arXiv)
• Zhang, Jianshu, Wu, Keliang, Lu, Haoran, Liu, Anbang, Zhang, Ce, Yin, Weijie, Qian, Chengxuan, Yang, Xiyuan, Pan, Zhenyu, Ye, Guo, et al.
• Northwestern University, Carnegie Mellon University, University of Wisconsin–Madison, University of California, Santa Barbara, University of Illinois Urbana-Champaign
• Paper

Read this article in Korean

Summary

• This survey organizes progress reward modeling for robotic learning, where dense feedback estimates whether an execution advances, stagnates, or reverses with respect to a task goal. It frames progress as a latent, task-conditioned, history-dependent state and connects model interfaces, reward construction, data construction, and evaluation. Its central argument is that heterogeneous outputs and benchmarks establish different properties and therefore should not be compared as equivalent evidence.

1 Introduction

Terminal success signals provide little intermediate guidance in long-horizon, high-dimensional robot tasks. Progress rewards can improve credit assignment and support monitoring, reranking, filtering, planning, action selection, and failure recovery, but a current observation alone may be insufficient because earlier subgoals may have been completed or undone.

The timeline places representative methods from 2017–2022 through 2026, illustrating the expansion from demonstration- and representation-based rewards to VLM critics, generated reward programs, and explicit progress models.

2 Interface: Input and Output of Progress Models

The survey treats a progress model as a task-conditioned black box specified by its current-state representation, task-goal specification, and output form. This interface-based taxonomy separates available evidence from the semantics of the produced signal, enabling comparison among systems with otherwise different architectures.

The diagram defines the interface taxonomy: a model receives a state representation and goal specification, then produces a state-wise score, progress delta, ranking, or executable reward function. It also distinguishes online recent-window context from offline full-trajectory context.

2.1 Input Interface: How Is the Current Task State Represented?

Current task state may be represented by a single observation, temporal context, relational comparison, or state-access interfaces such as APIs, logs, and proprioception. Single observations are efficient for online use but history-ambiguous; longer context provides stronger temporal evidence but can limit online use, while local comparisons and state-access assumptions have their own long-horizon and deployment constraints.

2.2 Input Interface: How Is the Task Goal Specified?

Task goals may be language-conditioned, vision-conditioned, or structured and programmatic. Language is convenient but can underspecify physical details; goal images and demonstrations provide richer physical grounding but can be sensitive to visual and embodiment variation; predicates, code, and APIs allow precise construction when relevant state variables are accessible.

2.3 Output Interface: What Does the Model Predict?

Progress models can emit state-wise scalar scores, transition-level progress deltas, rankings, or executable reward functions. Scalars are broadly reusable but require clear calibration, deltas capture local change and usually need temporal accumulation, rankings are relative rather than absolute, and program outputs depend on whether their logic and state features represent the intended objective.

3 Methods: How Are Progress Rewards Constructed?

The methods taxonomy identifies four ways to obtain usable progress rewards: frozen foundation-model scoring, learning from temporal or relative supervision, instruction-tuned progress prediction, and programmatic reward construction. The paradigms differ in their signal sources and in the assumptions they make about semantic priors, labels, temporal structure, and state observability.

The figure contrasts four construction paradigms by their signal source and conversion mechanism: frozen VLM or CLIP scoring, learned reward geometry from temporal or pairwise supervision, instruction tuning for explicit progress judgments, and generated executable reward logic.

3.1 Frozen Foundation Models as Semantic Reward Scorers

Frozen CLIP-style and vision-language-model methods derive reward-like signals from image-text alignment, task-completion token probabilities, or video temporal judgments without task-specific reward-model training. Their zero-shot utility comes at a calibration cost: the output is better regarded as a semantic prior than as a calibrated progress reward, so normalization, thresholding, differencing, or prompt engineering may be needed.

3.2 Learning Rewards from Temporal and Relative Supervision

Temporal supervision often assumes that later states in successful demonstrations are closer to completion, and can support goal-proximity, visual-distance, subtask, or reward-sketch approaches. Preference and ranking supervision compares states, clips, or trajectories and is often easier to collect than absolute scores, but learned comparisons still need conversion into a reward function and may not admit one consistent scalar ordering.

3.3 Instruction-Tuned Progress Prediction

Instruction-tuned progress models train VLMs or video-language models to answer explicit task-specific questions about absolute progress, success, relative improvement, preferences, or reasoning-grounded scores. Examples include rubric-based structured scoring, joint progress-success-preference prediction, before-after judgments, and generated progress reasoning; the defining feature is explicit instruction-following supervision for progress estimation.

3.4 Programmatic Reward Construction

Programmatic methods represent rewards as executable code, predicates, feature functions, or structured reward logic generated from task descriptions or visual reasoning. They are useful when tasks decompose into explicit subgoals, geometric relations, or API-exposed state, but generated rewards can remain misaligned when symbolic variables or available features omit important aspects of the task.

4 Data and Benchmarks

The survey groups progress-data construction into human-driven, human-in-the-loop, and fully automated pipelines, then separates evaluations of fidelity, robustness and generalization, and downstream utility. Human involvement supplies stronger semantic grounding, whereas automation offers scale but can import assumptions and systematic errors.

The figure relates the provenance of supervision to evaluation goals. It shows a continuum from human-driven to fully automated data construction and separates evidence for progress fidelity from robustness and generalization and downstream utility.

4.1 Data: How Is Progress Supervision Constructed?

Human-driven data include teleoperation, task videos, scalar scores, success cutoffs, keyframes, reward sketches, and behavioral comparisons, capturing semantics that are difficult to derive from time or state variables alone. Human-in-the-loop approaches use sparse anchors, verification, active queries, or designed prompts and APIs, while automated methods use temporal order, simulator state, foundation-model annotation, and trajectory augmentation; each source can create characteristic biases or artifacts.

4.2 Benchmarking Progress Fidelity

Progress fidelity should be evaluated separately through scalar calibration, temporal consistency, relative ordering, task grounding, and answerability or uncertainty. Counterfactual instructions and mismatched video-language pairs test goal conditioning rather than generic motion or elapsed-time cues, while selective prediction, abstention, and confidence calibration address ambiguity caused by occlusion, missing history, insufficient viewpoints, and underspecified goals.

4.3 Benchmarking Robustness and Generalization

Robustness and generalization evaluations should cover unseen tasks, instructions, objects, viewpoint and scene changes, and cross-embodiment transfer. The survey identifies non-monotonic execution—failure, regression, retrying, plateaus, recovery, and unusual subtask order—as especially important because successful monotonic demonstrations cannot show whether a model distinguishes progress from elapsed time.

4.4 Benchmarking Downstream Utility

Downstream utility includes online policy learning, offline reward relabeling, retrieval, filtering, reranking, planning, and action selection. Better policy success alone does not establish progress fidelity because it also depends on exploration, architecture, reset design, and environment dynamics; conversely, a calibrated estimator may be too sparse, noisy, brittle, expensive, or locally uninformative for control.

5 Limitations and Future Directions

The survey identifies four practical limitations: coarse spatial and temporal sensitivity, fixed-rate temporal assumptions, latency from multi-frame VLM inference, and inadequate long-horizon memory. It proposes finer multimodal physical sensing, temporally adaptive representations of plateaus and regressions, streaming encoders with cached task representations or hierarchical querying, and compact memories of completed, pending, and invalidated subgoals.

Appendix

• The PDF contains no appendix section: after Section 5, it proceeds directly to References. The paper provides an associated resource repository at https://github.com/sterzhang/Awesome-Progress-Models.

Brief Thoughts

The survey’s key methodological distinction is between constructing a reward, showing that it faithfully represents progress, and showing that it improves a downstream decision-making system. Its framework suggests that dense-reward claims should report the state-history and goal interfaces, output semantics, supervision source, calibration evidence, robustness to non-monotonic behavior, and control-time cost.