Towards Agentic Robots: Model, Data, Environment, and Harness | Summary

This article explains the key points of Towards Agentic Robots: Model, Data, Environment, and Harness.

• 2026-10-14 (arXiv)
• Qin, Chonghan, Wu, Dongming, Feng, Kaituo, Wang, Jiaxu, Yu, Xinlei, Sun, Peiwen, Fan, Yijia, He, Xilin, Chen, Bingao, Lian, Weitong, et al.
• MMLab, The Chinese University of Hong Kong, Fudan University, NovaXBot, Tsinghua University, Zhejiang University, Sun Yat-sen University
• Paper
• Github

Read this article in Korean

Summary

• Robots need to coordinate actions, track progress, and recover from failures when predicting the next movement alone cannot complete a task reliably. As motivating evidence, the survey cites GPT-6 Astra achieving 22.48% average success without task-specific fine-tuning across 42 RoboDojo simulation tasks, compared with 19.34% for the trained robot policy DM0.5, a model that maps observations to actions. This comparison comes from a cited study, rather than an experiment conducted by the survey.
• Towards Agentic Robots: Model, Data, Environment, and Harness organizes existing research into seventeen overlapping categories across four functional domains. It defines agentic robots as systems that plan and coordinate execution using observations and feedback, and the harness as software that exposes capabilities, manages their invocation, checks execution, and supports reuse. For harness, see An Empirical Study of Harness Design for Coding Agents | Summary.
• The survey argues that reported performance depends on the complete execution setup, including available controllers, observation access, feedback, and interaction budgets—the allowed time, calls, or attempts. Its contribution is a functional taxonomy and comparative synthesis, without a new controlled performance ranking. It identifies fair benchmarking, physical deployment, efficiency, and transfer across robot platforms as unresolved challenges.

The progression separates learning executable actions, connecting knowledge to action, predicting consequences, and coordinating decisions beyond a policy. The text describes overlapping capabilities, so the mountain illustration does not establish that each paradigm replaces its predecessor.

1 Introduction

Improved action prediction does not ensure reliable execution of tasks requiring many dependent steps. The introduction connects agent contributions across learning, data acquisition, environment construction, and execution software, as illustrated in Figure 2.

• The cited RoboDojo comparison also reports GalaxeaVLA G0.5 at 14.88%, Xiaomi-Robotics-1 at 13.93%, OpenWAM-α at 11.92%, and π0.5 at 6.91%. These figures motivate the survey but do not isolate the contribution of agent coordination.
• The stated contributions are a unified taxonomy, an analysis of representative mechanisms and their dependencies, and a research agenda for practical deployment.

The arrows connect environmental interaction to data, experience to model learning, harness decisions to planning and tool use, and execution back to the environment. These dependencies explain why an account of system behavior must include more than the model.

2 Evolution of Robot Learning

The paper describes four connected research traditions: conventional policies; vision-language-action models (VLAs), which connect visual and linguistic knowledge to robot actions; world models, which predict environmental changes; and agentic systems, which coordinate execution and improvement. Figure 1 illustrates these traditions as a progression, while the text explicitly allows their capabilities to coexist within one system.

2.1 Conventional Robot Policies

Conventional robot policies learn actions from demonstrations or reward-based experience. The survey separates image-based policies from 3D-aware policies, which incorporate scene geometry, according to their scene representations rather than their action dimensionality.

• Image-based examples include RoboMimic for learning from recorded demonstrations, ACT for modeling action sequences, and Diffusion Policy for generating actions through iterative denoising, which progressively removes noise from candidate actions.
• Geometric examples include PerAct’s voxels, a grid of 3D cells; RVT’s rendered virtual views; and DP3’s features from sparse point clouds, sets of sampled 3D locations. Scene representation, learning objective, and action decoder—the component producing actions—remain independently combinable choices.

2.2 Vision-Language-Action Models

VLAs adapt pretrained vision-language backbones, the models supplying visual and linguistic features, through training on robot trajectories, sequences of robot states and actions. Their action representations include discrete tokens, continuous diffusion or flow-based decoders that transform noise into action sequences, compact action tokens, and motion curves.

• Memory supplies earlier observations and task progress: NativeMEM compresses each historical frame from each camera view into a single memory token, a compact representation available to later action predictions.
• Intermediate reasoning supplies language subtasks, spatial guidance, or visual goals before action generation.
• Future prediction can supervise policy representations without requiring future images during execution; BagelVLA instead interleaves language reasoning, visual prediction, and action generation.

2.3 World Models and World-Action Models

World models predict the effects of actions and can serve as simulated environments or sources of visual plans, predicted observations describing a desired future. World-action models (WAMs) combine future visual prediction with action learning, linking generated behavior to its anticipated environmental consequences. For World-action models (WAMs), see Long-WAM: Scaling the Context of World-Action Models | Summary.

• 4D world modeling represents how 3D geometry changes over time. PointWorld predicts scene-point displacements, while TesserAct and MVISTA-4D connect predicted geometric futures to robot actions.
• Inverse dynamics, the mapping from predicted observations or internal representations to actions, makes visual plans executable but depends on both feasible predictions and robot-specific control.
• Fast-WAM learns through video co-training without future imagination at inference; ImageWAM uses pretrained image-editing features without decoding future images.

2.4 Agentic Robots

Agentic robots select and coordinate executable capabilities, interpret outcomes, and revise subsequent operations toward a goal. This perspective also covers collecting experience, organizing learning experiments, and using evaluation feedback to improve capabilities for later tasks.

3 Foundations of Agentic Robots

The foundations section defines agentic robots, introduces the functional taxonomy, and compares its scope with related concepts. Figure 3 maps representative systems onto the four domains, allowing contributions to occupy multiple categories.

Representative works populate the seventeen categories, providing examples of the functions covered by the taxonomy. Placement identifies functional contributions rather than mutually exclusive system types or a ranking of effectiveness.

3.1 Definition and Scope

An agentic robot autonomously plans and coordinates execution from environmental observations and execution feedback, without a human specifying every step. It can continue, seek information, revise a plan, or recover from failure, whether implemented as one integrated model or a combination of models, policies, programs, and tools.

• The drawer example makes the definition operational: select an opening skill, a reusable procedure or executable behavior; verify that the drawer opened; and decide whether to retrieve the object or revise the attempt.
• The scope includes physical robots and simulation. Action prediction or low-level feedback control alone does not establish the defined coordination capabilities.

3.2 Taxonomy

Table 1 defines seventeen non-exclusive categories: five in Model, three in Data, three in Environment, and six in Harness. Classification follows the function shaped by agent decisions, allowing the same work to appear wherever it makes a relevant contribution.

• Zero2Skill connects data collection and preparation to policy training; Harness VLA combines several harness functions around fixed executors, the components that carry out selected operations.
• ENPIRE connects experiment management with policy improvement, illustrating why capability development and task execution belong in the same framework.

The definitions distinguish intervention points: training changes the learning process, adaptation changes parameters or conditioning through experience, and harness functions govern invocation, retained information, recovery, and reuse. A system can therefore contribute to several categories.

3.3 Comparison with Related Concepts

Embodied artificial intelligence (AI), intelligence studied through environmental interaction, overlaps with agentic robotics across perception, reasoning, and action. Autonomous robotics already includes planning and monitoring, while robot foundation models acquire reusable capabilities through broad training; this survey concentrates on how those capabilities participate in ongoing decisions.

• The paper explicitly states that “agentic” does not imply complete independence from human guidance or a higher level of autonomy.
• Hi Robot illustrates compatibility between these perspectives by pairing a vision-language model (VLM), a model interpreting visual and linguistic information, with a VLA that executes updated intermediate instructions.

4 Model: What Should the Agent Learn, Understand, Decide, and Control?

The Model domain separates interventions in training, perception, planning, robot control, and adaptation. Figure 4 uses a drawer-retrieval task to show why missing handle evidence, an unsuitable action order, an infeasible grasp, and repeated policy failures require different remedies.

The shared drawer task holds the goal constant while separating five intervention points. Missing handle evidence, an incorrect action sequence, and an unsuitable learned policy require different diagnostic evidence and repairs.

4.1 Training

Training methods let agents revise reward signals, policy code, training configurations, or experimental workflows. Eureka generates reward code and revises it using training feedback, while ENPIRE exposes physical reset, verification, and safety interfaces to coding agents that modify policy or training code.

• Full policy training makes reward evaluation expensive. Cheaper alternatives use short proxy training runs, offline causal information, labeled demonstrations, or predefined reward components, each constraining the evidence available for judging a candidate.
• HARBOR structures simulation reinforcement learning, learning through reward-based interaction, into typed artifacts and checked stages. Its reliability depends on compatible task, reward, and algorithm interfaces; ENPIRE depends on reliable physical resets.
• The section recommends keeping measurements stable when agents revise rewards, prompts, tools, data, or training recipes, so that a change does not also alter its judge.

4.2 Perception

Perception converts observations into task-relevant concepts, relations, geometry, or beliefs and can include deciding what evidence to obtain next. Active perception, taking actions to reveal missing information, includes camera movement and manipulation that exposes occluded objects.

• SCOPE makes camera orientation and zoom explicit actions; MG-VQA can invoke grasp-and-remove or pushing tools before answering a visual question.
• Useful outputs must match the downstream executor: a semantic description such as “upright” needs a compatible geometric representation before it can guide motion.
• Evaluation conditions limit interpretation: CoRelNav uses oracle instance masks and identifiers, ground-truth object regions and identities supplied by simulation; INSPECT evaluates annotated replay and a fixed set of robot images rather than closed-loop physical inspection, where actions produce new observations.

4.3 Planning

Planning comprises generating operations or subgoals, checking proposed plans before execution, and updating them from execution outcomes. The section distinguishes choosing the next action from defining dependencies across multiple operations or revising the model used to plan.

• Hi Robot supplies language subgoals to a trained action policy; EmboTeam converts symbolic search results into behavior trees, executable structures containing conditions, fallbacks, and synchronization.
• SayPlan and GAVEL check candidate plans against graph-based scene models, representations of objects and their relationships; ReflectVLM evaluates imagined visual consequences.
• HUME treats additions to its planning model as uncertain hypotheses that perception can confirm or reject. The appropriate planning horizon, how many steps are planned ahead, depends on how reliably action outcomes can be predicted.

4.4 Robot Control

Robot control translates selected operations into physical movement through action or language commands, target poses and programs, or motion constraints and cost functions that specify acceptable or preferred movement. The required output detail depends on what the execution layer already handles, including inverse kinematics, computing joint configurations for a target position and orientation, and collision checking.

• GPT-Policy delegates pose interpolation, motion timing, and kinematic checks to execution software; these checks do not independently establish collision avoidance or task completion.
• ReKep specifies relationships between keypoints, selected spatial locations on objects, and VoxPoser constructs spatial value maps that score possible movements for an optimizer. Their solvers can respond to geometric changes without a language-model call at every control step.
• FAEA’s program-revision evaluation supplies ground-truth object and gripper states. Deployment from camera images would additionally require perception to provide the state expected by its interfaces.

4.5 Adaptation

Adaptation can change retained context, planner configurations, or learned model parameters, with different persistence and scope. The survey distinguishes immediate recovery within an attempt from improvements after retraining and generalization to other tasks.

• TidyBot and DROC condition decisions on examples or corrections without necessarily updating model weights; SERP adjusts planning parameters.
• OLAF converts verbal corrections into revised action labels, while Yell At Your Robot updates the high-level policy with the low-level controller fixed.
• Self-Improving Embodied Foundation Models uses a frozen checkpoint for rewards and success detection while training a separate policy. Its physical experiments use LanguageTable; ALOHA self-improvement is evaluated in simulation.

5 Data: How Should the Agent Collect, Annotate, and Refine Robot Data?

The Data domain separates acquiring new experience, adding structured supervision, and improving existing records. Figure 5 follows the same drawer interaction through collection, annotation, and refinement, including failure and recovery segments.

The same experience becomes a new record, a labeled record, and then a selected or corrected training example. Retaining useful recovery segments while discarding idle behavior illustrates why an unsuccessful attempt can still contain useful training data.

5.1 Collection

Collection methods acquire physical robot experience, execute tasks in simulation, or coordinate generative models to synthesize observations and motions. Agents can select underrepresented tasks, prepare or reset scenes, judge outcomes, and direct further collection toward observed weaknesses.

• Reported scales use different units and scopes: SOAR releases approximately 30,500 trajectories, Tether reports 1,085 successes in 1,946 attempts, and FIND collects 456 episodes in six hours.
• Zero2Skill invokes human assistance after a predefined retry limit; RoboClaw calls a paired reset policy after confirming task completion.
• RoboCurate checks consistency between generated videos and simulator replays of predicted actions. Its 10,000 generated trajectories are counted before filtering.

5.2 Annotation

Annotation adds text, spatial labels, interaction targets, or outcome labels to recorded experience. The surveyed pipelines combine semantic interpretation with perception tools, geometric calculations, simulator records, or execution evidence to produce and validate supervision.

• NILS segments demonstrations using object motion, state changes, and gripper interactions before generating task descriptions.
• Interaction labels describe contact or affordances, object regions suitable for a desired manipulation; outcome labels describe progress, success, failure, or safety violations.
• Reasoning traces add explanations of subtasks and next actions, while geometry-derived question-answer pairs provide spatial supervision with answers computed from scene geometry.

5.3 Refinement

Refinement selects useful trajectories or segments and revises unreliable supervision. HELP and ARS retain productive portions of unsuccessful attempts, while Zero2Skill applies human-corrected success criteria to previously collected demonstrations.

• Reflective Test-Time Planning uses later execution outcomes to revise earlier action scores and reflections, retaining the revised records for subsequent learning.
• The reliability of selection depends on the progress critic, a model estimating task progress, or the event-verification process used to distinguish useful progress and recovery from idle or harmful behavior.

6 Environment: What Worlds, Tasks, and Tests Should the Agent Build?

The Environment domain distinguishes the executable world, the task specification, and the evaluator. Figure 6 shows that an unstable scene, an infeasible goal, and an invalid success signal are separate failure sources that require different repairs.

The panels separate constructing a scene that supports execution, defining what the robot must achieve, and fixing how performance is measured. A failed test can consequently require repairing geometry, revising a task, or correcting an evaluator.

6.1 Reconstruction

Reconstruction turns observations, demonstrations, or requested layouts into scenes that support execution and interaction. Validation must address static support, dynamic response, articulated motion through connected joints, and task-relevant physical properties rather than visual similarity alone.

• Articulate-Anything revises link and joint code using rendered comparisons; DiagGen routes simulation diagnostics to segmentation, material, or mesh repair.
• Structured scene interfaces constrain the available edits, while generated code permits broader changes that require software, geometry, and physics checks.
• Matching an observed motion does not establish the correct response to a new action. Simulated stability and feasibility also do not establish real-world validity.

6.2 Task Generation

Task generation specifies goals, initial states, constraints, and success conditions, and can adjust the distribution of practice tasks as the learner changes. The paper separates generating a new task from generating a new solution to an unchanged task.

• FATE separates static semantic and geometric checks from dynamic execution checks and repairs the failing component.
• CurricuLLM generates curricula, sequences of practice subtasks, together with reward or goal-distribution code; RATs selects practice goals from scene information, available skills, and failure memory.
• Repair can alter difficulty or meaning, and a curriculum optimized using the learner’s own success signal can become easier without improving the target ability. Evaluation on independently specified tasks is therefore needed.

6.3 Benchmarking

Benchmarking defines the task, interface, reset procedure, and scoring rule used to measure an ability. The same task suite can test different capabilities when observation access, controller assistance, feedback timing, or interaction budgets change.

• RoboDojo combines 42 simulation tasks with 18 real-world tasks; RobotWorld covers 84 simulation tasks across manipulation, mobile manipulation, locomotion, driving, and aerial control.
• KinDER’s open-loop skill composition, execution without feedback-driven replanning, does not measure that replanning ability. ReactHuman pauses simulation during inference, measuring a hazard-response decision rather than real-time reaction.
• Agent-generated evaluations require validated scoring and suitable execution evidence. Improving an evaluator is a separate outcome from improving the robot it assesses.

7 Harness: How Should Robot Capabilities Be Orchestrated, Recovered, and Evolved?

Harness software organizes capabilities around execution through orchestration, memory, monitoring and recovery, skills, infrastructure, and layered systems. Figure 7 connects task goals and evidence to dispatch, the assignment of operations to executable components, physical execution, and returned feedback.

The central execution path is surrounded by functions that supply context, choose executors, check outcomes, preserve behavior, and enforce interfaces. It shows where dispatch, evidence, and feedback handling enter execution, alongside the policy itself.

7.1 Orchestration

Orchestration determines which component runs, what it receives and returns, whether its starting conditions hold, and when control returns to the decision process. Reliable selection requires both capability knowledge and compatibility between successive execution states.

• VoLo can interrupt a failing VLA and switch to a hand-designed skill; RoboHarness moves the robot toward the starting state expected by the next policy.
• Function protocols define valid arguments, returned evidence, and success conditions. Independent checks remain useful because even detailed execution results are the executor’s own reports.
• Handoffs, transfers of control between components, can follow progress events, execution budgets, or periodic schedules. Returning too late prolongs failure, while returning too early can interrupt an action that would succeed.

7.2 Memory

Memory retains earlier observations, attempts, and task progress for later decisions without changing model weights. Its functions are writing useful records, updating them as the scene changes, and retrieving the subset relevant to the next decision.

• Mimir separates task memory from world memory; RoboStream refreshes evidence at each planning step; OVAL verifies remembered candidates before relying on them.
• BATON retrieves transition conditions for the next skill and checks them against a live wrist-camera view. 2AM supplies compact task-state summaries to an action model without its own memory.
• Reusing earlier results can reduce computation but requires attention to validity: MeCo validates transferred solutions, while APEX continues exploration using spatial evidence that may lag behind the scene.

7.3 Monitoring and Recovery

Monitoring checks progress, completion, failure, or constraints before, during, or after execution; recovery changes behavior after a problem is detected. The survey separates detecting a problem, selecting and validating a response, and executing recovery.

• Checks before execution rank or reject candidates; checks during motion can stop or adjust execution; checks afterward can use the full record to explain failure.
• Recovery can revise policy inputs, switch executors, follow a different task branch, or adjust motion parameters. Available recovery actions constrain what the system can repair.
• Continuing from a disturbed scene differs from resetting and retrying, and LIBERO-RECOVER evaluates these conditions separately. Retry budgets and escalation rules are also part of the execution protocol.

7.4 Skills

Skills are reusable packages of procedural knowledge that a harness can load as guidance or call as executable behavior. The section traces fixed skill selection, generated control programs, persistent libraries, validated revisions, and training of executors or code-generating models.

• Code as Policies composes perception and control interfaces into programs; Teach and Grow admits reusable behavior only after scoped tests on held-out states, states excluded from the skill’s development.
• ASPIRE promotes validated repairs, RATs can retire repeatedly failing skills, and RoboRSI checks revisions through review, functional tests, and replay without regression.
• Improvement need not accumulate: Explore, Execute, Evolve reports non-monotonic progress, and a cited 123-round study found that accepted changes did not produce success on its target task.

7.5 Infrastructure

Infrastructure exposes robot capabilities through interfaces, perception and manipulation toolchains, policy-serving systems, and runtime gates that check commands before hardware execution. The survey cites comparisons showing how these choices affect measured performance under particular evaluation conditions.

• Replacing NavMCP’s complete three-channel protocol with an episodic interface lowers question-answering accuracy from 74.0% to 59.1%; halving Show-Harness’s action step size raises zero-shot success, success without task-specific training, from 60% to 82%.
• A cited study reports 65.0% success with full simulator observations and 22.5% with RGB alone, color images without privileged simulator state. Adding ASPIRE’s execution engine raises macro-average success, the mean across evaluated perturbations, on the evaluated LIBERO-Pro perturbations from 14% to 62%.
• SmolVLA reduces task time by about 30% through asynchronous execution, overlapping inference with queued actions, at a small cost in success. RoboGuard reduces jailbreak success, the success of attacks inducing prohibited robot behavior, from 92.3% to 2.3% in its reported evaluation.

7.6 Layered Systems

Layered systems divide authority and operating rates among System 2 planners, System 1 learned visuomotor policies, and System 0 analytic controllers. These are the survey’s functional labels for high-level reasoning, learned action generation from visual observations, and explicit control.

• Helix passes a latent vector, an internal learned representation, from a 7–9 Hz vision-language model to a 200 Hz policy. GR00T N1 uses a 10 Hz reasoning module and 120 Hz action chunks, sequences of actions generated together; Ground Slow, Move Fast combines a 2 Hz planner, a 30 Hz policy, and model-predictive control, which repeatedly optimizes motion using predicted outcomes.
• Calling a frozen VLA as one tool among analytic primitives, predefined control operations, makes components replaceable and calls inspectable, but every handoff must restore compatible state and intent.
• Decision application programming interfaces (APIs), interfaces selecting among supplied options, can support a faster intermediate layer. Their option probabilities require calibration on robot decisions and do not directly represent the probability of physical success.

8 Outlook: What Comes Next for Agentic Robots?

The outlook identifies four unresolved questions: isolating an agent’s contribution, executing its decisions reliably on physical robots, operating within resource limits, and adapting to unfamiliar platforms. Combining capable models and tools does not by itself establish improvement on these dimensions.

8.1 Fair Benchmarking

Fair benchmarking requires comparisons under matched executors, observation access, and interaction budgets, together with component removals such as disabling memory or replanning. Complete-system evaluations should additionally report computation, execution time, observations, retries, and human assistance.

• Tasks involving incomplete information, unexpected scene changes, and coordination across skills can expose agent contributions that a single fixed policy might otherwise conceal.
• For systems modifying data, models, or skills, evaluation should test unseen tasks beyond the feedback used for optimization.

8.2 Real-robot Agentic Robotics

Physical deployment requires interfaces that describe when skills apply, what conditions they require, and how completion is confirmed. Valid commands can still fail because of noisy perception, calibration errors, unexpected contact, or a changing scene.

• A stop request does not establish that motion has stopped. Replanning needs feedback confirming the current state and the effects of actions already taken.
• Real-robot evaluations should disclose information supplied to the agent, responsibilities delegated to lower-level controllers, and recovery behavior when assumptions fail.

8.3 Efficient Robot Use

Efficient robot use depends on when reasoning is invoked as well as model inference speed. HarnessWAM and VLAs-as-Tools illustrate using lightweight progress feedback to request higher-level decisions when needed.

• Evaluation should include reasoning, tool calls, communication, and physical completion time. Reducing calls can worsen overall efficiency if it increases failures, retries, or movement time.
• The appropriate timing depends on progress, uncertainty, and the consequences of delayed intervention; cloud and local computation introduce different latency and resource tradeoffs.

8.4 Cross-Embodiment Generalization

Cross-Embodiment Generalization, transferring capabilities across robots with different bodies, sensors, and control interfaces, remains unresolved. Similar goals can require different plans for wheeled robots, humanoids, and drones, beyond translating the same commands into another interface.

• The paper proposes identifying unfamiliar robots’ capabilities before planning and retaining general strategies without assuming that robot-specific motion details transfer.
• Suggested evaluations measure success on unseen robots, additional configuration or training, and whether experience from other platforms improves performance.

9 Conclusion

The conclusion connects the seventeen categories through the observations, executable capabilities, and feedback available to an agent. It emphasizes testing changes to data, models, skills, and software on new tasks while checking preservation of previously acquired capabilities and accounting for time and resource costs.

Appendix

• The supplied PDF contains no appendix; References occupies pages 49–79. The authors provide the accompanying literature repository at https://github.com/chonghanqin/Awesome-Agentic-Robots.

Brief Thoughts

The taxonomy is useful for separating an agent’s decisions from the execution conditions that make them effective. The cited gaps between privileged and perceived observations, and between alternative interfaces or execution engines, support evaluating the complete robot system. The overlapping categories help locate interventions and failure sources, but the survey does not provide a controlled causal comparison of the methods it reviews. Its discussion of unsuccessful cumulative skill improvement provides a useful limit on the evidence: validated local edits and additional iterations do not automatically establish persistent gains on a target task.