The Landscape of Agentic Reinforcement Learning for LLMs: A Survey | Summary

Contents

Summary
1 Introduction
2 Preliminary: From LLM RL to Agentic RL
3 Agentic RL: The model capability perspective
4 Agentic RL: The Task Perspective
5 Enviroment and Frameworks
- 5.1 Environment Simulator
- 5.2 RL Framework
6 Open Challenges and Future Directions
7 Conclusion
Appendix
Brief Thoughts

This article explains the key points of The Landscape of Agentic Reinforcement Learning for LLMs: A Survey.

2025-09-02 (arXiv), Transactions on Machine Learning Research (01/2026)
Zhang, Guibin, Geng, Hejia, Yu, Xiaohang, Yin, Zhenfei, Zhang, Zaibin, Tan, Zelin, Zhou, Heng, Li, Zhongzhi, Xue, Xiangyuan, Li, Yijiang, et al.
University of Oxford, Shanghai AI Laboratory, National University of Singapore, University of Illinois Urbana-Champaign, Brown University, University College London, University of Science and Technology of China, Imperial College London, Dalian University of Technology, Chinese Academy of Sciences, The Chinese University of Hong Kong, University of Georgia, University of California, San Diego, University of California, Santa Barbara, University of Bristol
Paper
Github

Summary

Training large language models (LLMs), models that generate language, to complete interactive tasks requires learning how earlier decisions affect later outcomes. Synthesizing over five hundred works, this survey distinguishes Agentic Reinforcement Learning (Agentic RL), reward-based learning of interactive decision rules, from single-response optimization by comparing multi-step, partially observed environments with a single-step prompt–response setting.
The survey organizes the literature by capabilities—planning, tool use, memory, self-improvement, reasoning, and perception—and by application domain. It connects these taxonomies to training algorithms, reward design, environments, benchmarks, and open-source frameworks.
The paper compiles results from cited studies without conducting a common experimental comparison: rStar2-Agent, for example, reports 80.6% and 69.8% average pass@1, success with one candidate attempt, on the AIME24 and AIME25 mathematics benchmarks after 510 RL steps. Unresolved questions include credit assignment, attributing delayed outcomes to earlier actions, reward reliability, training costs, and whether RL expands reasoning capabilities or increases the probability of existing successful solutions.

1 Introduction

The introduction identifies fragmented terminology, isolated capability studies, and inconsistent evaluation protocols as obstacles to comparing agent training methods. It defines Agentic RL through learnable policies, rules for selecting actions from available information, operating in sequential decision loops with planning, reasoning, tools, memory, and reflection.

The primary focus is reward-based improvement of LLM agents in dynamic environments. Human-value alignment alone, traditional RL without LLMs, and improvements restricted to static language benchmarks fall outside that focus.
The survey connects formal foundations, capability and task taxonomies, practical infrastructure, and open challenges; Figure 1 provides its organizational map.

The primary organizational structure of the survey.

2 Preliminary: From LLM RL to Agentic RL

The preliminary section contrasts supervised fine-tuning (SFT), learning from prompt–response demonstrations, with reinforcement fine-tuning (RFT), optimizing models through rewards. Preference-based RFT (PBRFT), optimization using preference data or preference-derived rewards, supplies the paper’s baseline for explaining interactive task execution.

The comparison concerns states, actions, transitions, rewards, objectives, and decision granularity. Figure 2 depicts the expansion from text responses to tool-mediated interaction.

The transition from LLM RL to Agentic RL through broader actions and dynamic interaction.

2.1 Markov Decision Processes

A Markov decision process (MDP), a model of states, actions, transitions, and rewards, provides the common formal vocabulary. The paper models PBRFT as a degenerate MDP with \(T=1\) and \(\gamma=1\), where T is the task horizon and γ weights delayed rewards, whereas Agentic RL uses a partially observable Markov decision process (POMDP), where observations reveal only part of the underlying world state.

Table 1 compares a single prompt and terminal response with multiple states, text and external actions, dynamic transitions, and accumulated rewards. These are the survey’s organizing abstractions.

Formal comparison between traditional PBRFT and Agentic RL.

2.2 Environment State

PBRFT begins from one static prompt and ends when the model emits its response. In Agentic RL, the agent acts over multiple time steps using observations \(o_t=O(s_t)\), where O maps world state s_t to the observation at time t; tool results, user messages, and environmental feedback inform subsequent decisions.

2.3 Action Space

The action space combines language generation and environment-interactive behavior: A_agent = A_text ∪ A_action, the union of text outputs and external actions. External actions can acquire information or modify state and may contain sequences of primitive actions.

The paper illustrates structured actions with <action_start> and <action_end>, call(“search”, “Einstein”), and move(“north”). These examples describe the formulation without prescribing an interface for every surveyed system.

2.4 Transition Dynamics

The baseline formulation transitions deterministically to a terminal state after one response. Agentic RL models evolving states through s_{t+1} ∼ P(s_{t+1} | s_t, a_t), meaning that the next state is drawn from a transition distribution conditioned on the current state and action, supporting repeated communication, information acquisition, and environment manipulation.

2.5 Reward Function

PBRFT assigns a scalar score to a completed response without intermediate feedback. The agentic formulation permits a completion reward, a reward for step-level progress, or zero otherwise, with signals from sources such as unit tests and symbolic verification.

Outcome rewards score final success; process rewards score intermediate steps. The latter provide more direct signals for credit assignment, but their usefulness depends on the evaluator or rules.

2.6 Learning Objective

The baseline maximizes expected response reward, while Agentic RL maximizes J_agent(θ) = E_{τ∼π_θ}[Σ_{t=0}^{T−1} γ^t R_agent(s_t,a_t)], with 0 < γ < 1 in the stated formulation. This is the expected discounted reward over a trajectory τ, a sequence of interactions sampled from policy π_θ with parameters θ.

Policy-gradient methods update action probabilities using reward estimates; value-based methods learn estimates of future reward. The survey connects both to exploration and long-term credit assignment.

2.7 RL Algorithms

The algorithm review covers REINFORCE, a sampled policy-gradient estimator; Proximal Policy Optimization (PPO), which clips the ratio between updated and previous action probabilities; Direct Preference Optimization (DPO), which learns from preferred and dispreferred responses without a separate explicit reward model; and Group Relative Policy Optimization (GRPO), which estimates action quality from relative rewards within groups of sampled outputs. Table 2 catalogs variants by objective and mechanism.

REINFORCE has high estimator variance and sample-efficiency limitations. PPO constrains updates but typically incurs the cost of a critic, a model estimating expected reward.
DPO depends on preference-data quality and coverage. GRPO removes the absolute-value critic, but its group-relative advantage estimates, measures of action quality relative to a baseline, can be noisy or inaccurate.
Figure 3 connects optimization methods to agent actions and environmental rewards. The survey provides no common experiment ranking the four algorithms.

Agent–environment interaction and the reinforcement-learning loop for agentic LLMs.

Representative variants of the PPO, DPO, and GRPO algorithm families.

3 Agentic RL: The model capability perspective

The capability taxonomy treats planning, tool use, memory, self-improvement, reasoning, and perception as interdependent components whose behavior can be optimized through feedback. Figure 5 connects these components to representative learning mechanisms and interaction patterns.

3.1 Planning

Planning, choosing a sequence of actions to achieve a goal, is organized around RL as an external guide and as an internal driver. External guidance trains an auxiliary value or heuristic function to evaluate proposed plans, often supporting Monte Carlo Tree Search (MCTS), search that samples and evaluates branching action sequences; internal optimization learns plan-generation behavior from interaction.

The survey places RAP and LATS in the external-guidance category and discusses ETO’s preference learning from successful and failed trajectories and RLTR’s tool-use rewards for evaluating action sequences under internal planning.
Learning When to Plan addresses deliberation cost. Planner-R1 is cited for improved learning efficiency with process-level reward shaping, including competitive planning by smaller models.
A policy controlling search depth, branching, and deliberation is proposed as a synthesis of the two approaches.

3.2 Tool Using

Tool use progresses from ReAct, a Thought-Action-Observation prompting cycle, through demonstration-based training to Tool-Integrated Reasoning (TIR), interleaving natural-language reasoning with operations such as code execution or image editing. Methods such as ToolRL, ReTool, and ToRL use task feedback to optimize which tools to invoke and when.

Figure 4 illustrates representative stages rather than an exhaustive chronology. ToolRL is discussed as learning tool behavior from base models without imitation traces.
Sparse final rewards make it difficult to identify individual tool calls’ contributions in long-horizon TIR. GiGPO and SPA-RL illustrate early approaches to finer temporal attribution.
More granular supervision must avoid penalizing useful exploration or encouraging reward hacking, exploiting a scoring mechanism without achieving the intended goal.

Development of agentic tool use through representative methods.

3.3 Memory

The memory review distinguishes retrieval-augmented generation (RAG), generating responses with retrieved external information, from token-level and structured memory. Memory-R1 learns ADD/UPDATE/DELETE/NOOP operations—adding, updating, deleting, or leaving entries unchanged—from downstream question-answering performance, while MemAgent learns which natural-language memory tokens to retain or overwrite.

Token-level memory can use readable text or latent embeddings, numerical representations without directly readable text. MemoryLLM and M+ illustrate latent memory representations, while MemGen explores RL-based latent memory; Table 3 identifies methods that directly employ RL.
Zep, A-MEM, G-Memory, and Mem0 organize relational, temporal, or hierarchical information. The survey identifies learned control of structured memory construction and revision as underexplored.

Agent memory categories and representative methods, with direct use of RL indicated.

3.4 Self-Improvement

Self-improvement is divided into verbal self-correction without parameter updates, training that internalizes correction, and iterative self-training. Examples span Reflexion and Self-Refine, preference-trained reflective agents, execution-verified task generation, and shared repositories of successful trajectories.

Absolute Zero generates tasks and learns from execution outcomes. SiriuS illustrates collective bootstrapping, using shared multi-agent experience to build training data, while MALT’s cited implementation uses three agents.
Adaptive reflection is a proposed direction: learning when a brief check suffices and when search or execution-based verification is needed. The reviewed methods do not establish unlimited self-improvement.

3.5 Reasoning

Fast reasoning uses few intermediate steps, while slow reasoning produces deliberate, structured reasoning traces. Chain-of-thought (CoT), explicit intermediate reasoning text, verification, search-generated training data, and RL support the latter, with longer inference time as a practical cost.

The paper highlights overthinking, unnecessarily extended reasoning that increases latency, and training instability across heterogeneous environments.
Adaptive test-time scaling, varying inference computation with task difficulty, is discussed as a way to balance speed and accuracy.

3.6 Perception

Perception extends RL to vision–language models, models combining visual inputs and language, and audio-related tasks. The section distinguishes answering from a fixed observation from active perception, which revisits input regions, invokes visual tools, or generates intermediate images.

Grounding links reasoning to specific input regions. GRIT interleaves bounding-box tokens with reasoning, while Pixel Reasoner introduces crop, erase, and paint actions.
Intersection over Union (IoU), overlap divided by the union of predicted and reference regions, supplies a visual reward in several methods. Generation-driven approaches such as GoT-R1 and T2I-R1 optimize intermediate planning and image production.
Audio applications include question answering and speech-synthesis component optimization using speaker similarity and word error rate, the rate of word-level transcription errors.

Conceptual overview of RL mechanisms across six core agentic capabilities.

3.7 Others

Maintaining coherence across multiple turns requires learning signals connecting intermediate decisions to delayed outcomes. The survey groups approaches into combining process supervision with final rewards and extending preference comparisons from individual responses to multi-step segments.

EPO, ThinkRM, SPO, and AgentPRM provide intermediate evaluations through reward models, whereas RLVMR uses programmatic rules. Segment-level DPO compares conversational snippets or action sequences.

4 Agentic RL: The Task Perspective

The task taxonomy covers search and research, code and software engineering, mathematics, graphical user interfaces (GUIs), visual interfaces operated through screen-based actions, vision, embodied interaction, multi-agent systems, and other applications. Figure 6 maps representative methods across domains; Table 9 connects applications to qualitative capability dependencies.

Evolution tree of reinforcement learning for domain-specific agents.

4.1 Search & Research Agent

Search and research agents optimize query generation, retrieval, reasoning, and synthesis across multiple steps. The section separates open-source methods using live search from simulated or internal retrieval, then discusses closed-source research systems; Table 4 records representative methods, base models, and resources.

Search-R1 and DeepResearcher combine outcome rewards with retrieved-token masking, excluding retrieved text from the model’s learning signal. StepSearch adds intermediate rewards, and ASearcher supports 40+ tool calls through asynchronous RL, collecting interactions and updating policies without requiring every task to finish together.
ZeroSearch and SSRL reduce dependence on live search during training, addressing uncontrolled document quality and application programming interface (API) costs; APIs provide programmatic access to external services.
OpenAI Deep Research is reported at 51.5% pass@1 on BrowseComp, a benchmark for finding difficult-to-locate information. Its case study combines subgoal planning, web search, scratchpad memory, and cross-source verification; the survey provides no matched experiment isolating RL’s contribution.

RL-based methods for search and research agents.

4.2 Code Agent

Code agents are organized into code generation, iterative refinement, automated software engineering (SWE), and executable world models. Compilation, tests, and runtime traces supply outcome and process feedback; Figure 7 compiles benchmark scores, and Table 5 records methods, reward types, and base models.

For generation, DeepCoder-14B uses GRPO-based training, while PRLCoder scores partial snippets through compiler and test feedback. The survey reports PRLCoder’s +10.5% pass-rate improvement over its base model without specifying a common evaluation across all methods.
For refinement, RLEF, Reinforcement Learning from Execution Feedback, learns correction from execution errors, IterPref targets faulty regions, and CTRL trains a separate critic with additional inference cost. For SWE, Qwen3-Coder trains across 20,000 parallel environments, and SWEET-RL uses privileged information, training feedback unavailable to the deployed agent, on ColBench.
Code World Models (CWMs), programs representing transition and reward dynamics, enable planning through executable simulation. GIF-MCTS and WorldCoder illustrate this approach, whose repeated program synthesis, repair, and simulation can incur substantial inference cost.

Reported benchmark performance of RL-enhanced code and software engineering methods.

RL methods for code and software engineering agents.

4.3 Mathematical Agent

Mathematical agents are divided into informal reasoning in natural language, including numerical or symbolic code execution, and formal reasoning producing mechanically checkable proofs. Both use outcome and process rewards; formal methods also use expert iteration, repeatedly training a policy on verified trajectories found through search.

The survey reports that 1-shot RLVR, reinforcement learning with verifiable rewards, performs close to a 1.2k-example training condition with 1 example and close to the 7.5k MATH training dataset with 2 examples. These comparisons come from a cited study and concern its training and evaluation settings.
rStar2-Agent uses a 14B-parameter model, a high-throughput Python execution environment, GRPO-RoC, a GRPO variant resampling correct rollouts amid tool noise, and a multi-stage recipe. After 510 RL steps, its reported average pass@1 is 80.6% on AIME24 and 69.8% on AIME25.
Proof assistants such as Lean, Isabelle, Coq, and HOL Light mechanically verify proofs. DeepSeek-Prover-v2 adds subgoal feedback; Kimina-Prover requires at least one tactic block, a command sequence refining proof goals, and reuse of no less than 60% of Lean 4 snippets in the final proof. Figure 8 and Table 6 summarize results and reward regimes.

Reported benchmark performance of RL-enhanced mathematical methods.

RL methods for mathematical reasoning agents.

4.4 GUI Agent

GUI agents progress from pretrained models used without task-specific training, through SFT on screen–action pairs, to RL on static datasets and interactive environments. Table 7 separates training paradigm from environment complexity.

RL-free Methods include MM-Navigator, SeeAct, and TRISHUL. Demonstration-based approaches improve grounding but depend on human interaction traces; TongUI constructs the 143K-trajectory GUI-Net from multimodal web tutorials.
RL in Static GUI Environments uses pre-collected traces and rule-based action evaluation. GUI-R1, UI-R1, InFiGUI-R1, and AgentCPM-GUI illustrate format rewards, argument accuracy, subgoal supervision, and compact action interfaces.
RL in Interactive GUI Environments learns from online rollouts, sequences collected through agent interaction. DiGiRL combines offline and online learning on Android, ZeroGUI generates tasks and estimates rewards, and ComputerRL combines API-GUI actions with asynchronous training and alternating RL and SFT.

GUI-agent methods categorized by training paradigm and environment complexity.

4.5 Vision Agents

Vision applications span images, videos, and 3D understanding and generation. The survey selectively reviews rewards for localization, segmentation, scene graphs, representations of objects and their relations, temporal reasoning, and generation quality.

Image methods use region overlap, precision, recall, and structured-output matching. Active approaches revisit visual evidence or use tools and generated sketches.
Video methods include token-weighted rewards, keyframe supervision, and a regression formulation of GRPO in DeepVideo-R1. Generation methods optimize temporal coherence, controllability, and semantic alignment.
3D methods include MetaSpatial’s physics-aware and rendered-image rewards, Scene-R1 without point-wise 3D supervision, and generation approaches using human preferences or 2D reward signals.

4.6 Embodied Agents

Embodied agents perceive and act in physical or simulated environments; vision-language-action (VLA) models combine visual observations, language instructions, and action prediction. The survey separates navigation, which emphasizes planning movement, from manipulation, which requires precise object control.

VLN-R1 rewards predicted paths against reference paths, while RLVLA and VLA-RL use vision–language evaluators for trajectory-level manipulation feedback.
The simulation-to-reality gap, differences between training simulations and physical deployment, remains a limitation. The paper notes the cost and maintenance burden of physical experiments and millions of interaction steps.
The Voyager case study describes Minecraft exploration, reusable skills, memory, and curriculum selection. The survey places it within an RL-oriented account of embodied learning without independently measuring the contribution of each component.

4.7 Multi-Agent Systems

Multi-agent systems (MAS), multiple agents solving tasks through coordinated interaction, are classified by whether learning changes external coordination, selected agent policies, or all agents jointly. Table 8 distinguishes task-dependent configuration from training the agents’ LLM backbones.

RL-Driven Optimization of Non-Parametric Coordination Modules keeps agent parameters frozen and optimizes communication or workflow structures, as in GPTSwarm, MaAS, and G-Designer. Credit assignment operates at the topology or message-routing level.
RL-Driven Optimization of Selected Agent Policies updates a subset of agents, balancing specialization, stability, and sample cost. MALT combines supervised training with preferences from successful and failed reasoning paths.
End-to-End Multi-Agent Reinforcement Learning jointly optimizes collaboration, often using a decentralized POMDP, a cooperative decision model with separate agent observations and actions. MAGRPO, ReMA, and SPIRAL illustrate joint learning, hierarchical roles, and self-play, learning through interactions among agents; attributing rewards across both time and agents remains difficult.

Learning and evolution paradigms in LLM-based multi-agent systems.

4.8 Other Tasks

Other applications include text games, database querying, chart-to-code generation, time series, general question answering, and social interaction. Recurring mechanisms include finer reward attribution, learned action sequences, and adaptive reasoning depth.

The survey reports SkyRL-SQL-7B surpassing GPT-4o and o4-mini on SQL generation benchmarks using 653 training examples; SQL is the language used to query relational databases. The section also discusses rewards at multiple levels of detail for chart-to-code generation.
Time-R1 and TimeMaster address temporal reasoning, while Sotopia-RL uses utterance-level multidimensional rewards. Adaptive Mode Policy Optimization switches reasoning modes and is reported to improve accuracy and shorten reasoning relative to fixed-depth methods such as GRPO.
Table 9 assigns core, supporting, or minimal dependencies between task domains and capabilities. These labels serve as qualitative navigation aids.

Qualitative application–capability dependency matrix.

5 Enviroment and Frameworks

The infrastructure review connects agent training to environments returning observations and rewards and to frameworks executing learning and collecting experience. It separates task environments and benchmarks from software systems used to train policies.

5.1 Environment Simulator

The environment inventory spans web and GUI interaction, coding and SWE, specialized scientific and engineering domains, games, and general-purpose evaluation. Table 10 records capabilities, task domains, modalities, and resources, while the text distinguishes interactive environments from datasets and evaluation suites.

Web Environments include WebShop, WebArena, VisualwebArena, and AppWorld. AppWorld contains 9 applications, 457 invokable APIs, and approximately 100 virtual characters; GUI Environments include AndroidWorld with 116 hand-crafted tasks across 20 applications and OSWorld across Ubuntu, Windows, and macOS.
Coding & Software Engineering Environments include interactive tools such as Debug-Gym and R2E-Gym, with over 8K tasks, alongside evaluation suites. HumanEval has 164 tasks, MBPP has 974, BigCodeBench has 1,140, and SWE-rebench provides >21k tasks; executable-world-model benchmarks assess transition fidelity and planning utility.
Domain-specific Environments include PaperBench, reproducing 20 ICML 2024 papers, MLE-Bench with 75 Kaggle competitions, MedAgentGym, and SecRepoBench with 27 repositories and 15 weakness categories. Simulated & Game Environments include TextWorld, Crafter, Craftax, and Factorio; General-Purpose Environments include AgentGym, Agentbench, and InternBootcamp with over 1000 verifiable reasoning tasks.

Environments and benchmarks for agentic reinforcement learning.

5.2 RL Framework

The framework review separates Agentic RL systems, reinforcement learning from human feedback (RLHF), learning from human-derived preference signals, and LLM fine-tuning toolkits from general-purpose RL libraries. Table 11 summarizes their stated features.

Agentic systems include Verifiers, SkyRL-v0, AREAL, MARTI, AgentFly, Agent Lightning, AWorld, ROLL, AgentRL, and VerlTool. Features include verifiable environments, asynchronous experience collection, hierarchical learning, multi-agent training, and tool-integrated rollouts.
AWorld is reported to achieve a 14.6× speedup over single-node execution through parallel rollouts across clusters. The survey does not compare all frameworks under the same workload.
OpenRLHF, TRL, trlX, HybridFlow, SLiMe, and Oat support model post-training. RLlib, Acme, Tianshou, Stable Baselines3, and PFRL provide broader learning algorithms and execution components.

Reinforcement-learning frameworks categorized by type and key features.

6 Open Challenges and Future Directions

The challenges section centers on trustworthy behavior, scaling training, and scaling environments. It also examines the mechanisms of RL improvements, deployment architecture, and social effects.

6.1 Trustworthiness

Trustworthiness covers security, hallucination, confident outputs or actions unsupported by evidence, and sycophancy, agreement with user beliefs even when they are incorrect. Tools, memory, and agent communication create additional paths through which malicious external content and flawed rewards can affect behavior.

Security proposals include sandboxing, execution within restricted environments, penalties for unsafe intermediate actions, adversarial training against manipulation, and monitoring tool and memory activity.
Outcome-only optimization can reward correct final answers despite unsupported intermediate reasoning or reduce abstention on unanswerable questions. The paper discusses evidence-checked process rewards and training on both solvable and unsolvable tasks, while emphasizing that effects depend on the training pipeline.
For sycophancy, the survey discusses rewards penalizing uncritical agreement and Cooper’s joint adaptation of policy and reward models to close scoring loopholes.

6.2 Scaling up Agentic Training

Training scale depends jointly on computation, model capacity, data composition, and algorithmic efficiency. The survey cites prolonged-training gains from ZeroTIR and ProRL while discussing reduced output diversity, sensitivity to training settings, and interference between domains.

A cited web-agent study reports that teacher-generated SFT followed by on-policy RL, learning from interactions produced by the current policy, outperforms either alone and cuts compute by half compared with pure SFT. This comparison applies to that study’s setting.
Cross-domain benefits depend on initialization and pretraining exposure; the Guru dataset spans six reasoning domains. POLARIS and other recipes emphasize difficulty calibration, diverse sampling, and stable training.

6.3 Scaling up Agentic Environments

Scaling environments involves learning or generating training conditions as well as optimizing the agent. The paper discusses automated reward construction from exploratory trajectories and adaptive curricula, task sequences adjusted to observed agent weaknesses.

EnvGen uses an environment-generator LLM to alter configurations based on agent performance. The survey presents automated rewards and task generation as approaches to scarce interactive environments and costly manual reward specification.

6.4 The Mechanistic Debate on RL in LLMs

The mechanistic debate contrasts an amplifier view, where RL increases the likelihood of existing successful trajectories, with a new-knowledge view, where learning changes computational strategies or expands reachable solutions. Pass@k, success with at least one correct candidate among k attempts, helps distinguish improved single-attempt reliability from broader solution coverage.

The survey estimates that approximately 2/3 of cited mathematical and code-reasoning studies primarily emphasize pass@1 improvements, while about 1/3 report expanded pass@k frontiers. It provides no controlled pooled experiment establishing the underlying mechanisms.
Reported capability expansion is associated with reliable executable or formal rewards, compositional tasks with verifiable intermediate states, and base models between near-random and near-ceiling performance.
Post-saturation generalization, improving test performance after training accuracy approaches its ceiling, and cross-category transfer suggest changes beyond probability reweighting in some cases. Whether RL produces abilities beyond pretraining remains unresolved.

6.5 Architectural Patterns for Real-World Agent Deployment

Deployment architecture is organized around guardrails, checks and constraints on inputs, outputs, and execution; human-in-the-loop verification, human review or correction of agent decisions; hierarchical orchestration; and communication protocols. These mechanisms address unreliable or delayed feedback and coordinate complex workflows.

Guardrails include input constraints, output checks, and sandboxed execution. Human intervention can be triggered by uncertainty, while supervisor–worker structures coordinate subtasks and enforce global constraints.
Fixed communication interfaces support predictable integration; learned channels allow adaptation but require regularization, constraints discouraging unstable behavior. The section synthesizes design principles without a deployment ablation comparing their individual contributions.

The social-impact discussion covers misuse, resource consumption, labor-market effects, bias amplification, and evaluation contamination, overlap between training information and evaluation material that can inflate scores. These concerns come from cited work rather than a new impact study.

The paper discusses concealed harmful behavior activated during deployment and possible effects on entry-level or routine cognitive work. These risks and labor-market implications are not measured directly by the survey.
Rollout generation and extended reasoning increase resource costs; quantization, representing model values with fewer bits, is discussed as a mitigation. Preference learning can reinforce annotator biases or reduce minority preference coverage, motivating fairness-aware rewards, diverse annotators, and diversity-preserving objectives.
Dynamic benchmarks such as LiveCodeBench, LiveSearchBench, LiveTradeBench, and LiveBench aim to reduce contamination and improve evaluation beyond fixed suites.

7 Conclusion

The conclusion restates the formal distinction between single-step response optimization and extended agent–environment interaction. The survey contributes capability and application taxonomies, connects them to RL mechanisms, and consolidates environments, benchmarks, and frameworks.

Appendix

The supplied 95-page PDF contains no separate appendix: the main discussion ends on page 49, followed by References through page 95. The reviewed source is arXiv:2509.02547v5, dated 17 Apr 2026, and identifies its venue as Transactions on Machine Learning Research (01/2026); its review link is https://openreview.net/forum?id=RY19y2RI1O.

Brief Thoughts

The survey supports engineering decisions by separating what is optimized, how feedback is produced, and whether training includes live interaction; Tables 1, 7, 8, 10, and 11 make those distinctions concrete. Its benchmark plots map reported progress, but heterogeneous base models, rewards, environments, and candidate budgets limit conclusions about the best method or the contribution of RL alone. A recurring design question is whether intermediate feedback reliably captures task progress. The discussions of long-horizon tools, code execution, formal proofs, and trustworthy behavior connect this choice to credit assignment and scoring loopholes. The MDP/POMDP distinction is an organizing abstraction. Individual systems still require inspection of their actual observations, transitions, and learning procedures because the taxonomy also includes prompting, preference optimization, and external-module learning.