Verifiable Hidden Dynamics Play: Generating Agentic RL Environments from Solved Mechanisms | Summary

Contents

Summary
1 Introduction
2 Related Work
3 Formulation
4 Producing environments at scale
5 Experiments
6 Conclusion
Appendix
Brief Thoughts

This article explains the key points of Verifiable Hidden Dynamics Play: Generating Agentic RL Environments from Solved Mechanisms.

2026-09-23 (arXiv)
Shen, Xinjie, Fan, Wei, Guo, Xudong, Tu, Jianhong, Su, Yang, Kuang, Chuqiao, Zhang, Yinger, Liu, Dayiheng.
Georgia Institute of Technology, Alibaba Token Foundry, Alibaba Group
Paper

Read this article in Korean

Summary

Language-model agents need inexpensive training environments in which actions change persistent state and final outcomes can be scored reliably. Verifiable Hidden Dynamics Play (VHD-Play), which generates environments from solved mathematical decision models, raises Qwen3.6-35B-A3B’s mean interactive score from 0.204 to 0.815 across five optimization families after training on three families, compared with the starting checkpoint under the same evaluation protocol.
VHD-Play computes reference outcomes before generating an environment, deriving executable dynamics, the rules governing state changes and outcomes, and the scoring standard from the same mathematical instance. A frozen setter, the language model constructing the scenario and tools, uses real-world document passages to diversify settings while the player, the agent being trained, obtains permitted views of hidden parameters through interaction. The pipeline produces 3,300 admitted environments, meaning records that pass the applicable generation checks, at a reconstructed public-rate cost of approximately $0.01–$0.03 each.
Reinforcement learning, training from outcome rewards, improves all eleven evaluated mechanism families, groups of decision problems sharing a mathematical structure, and transfers to function calling, travel planning, and a 365-day simulated business. In the business benchmark, mean ending balance rises from 54,294 to 182,844 across five runs per model, while comparisons with fully specified problems indicate that stateful execution accounts for most of the starting model’s performance gap. The main generated-family results use one training run and one evaluation seed, and reference replay, executing reference policies to check generated behavior against precomputed outcomes, covers eight families.

1 Introduction

The introduction treats environment generation and dependable outcome scoring as a joint construction problem: executable actions determine possible trajectories, sequences of observations and actions, while evaluation determines their value. Existing approaches offer different combinations of exact checks, interactive dynamics, varied settings, and low authoring cost. VHD-Play samples and solves a mechanism first, uses document passages to render its setting, and withholds the complete parameter draw and reference outcomes from the player; Figure 1 motivates this design through the gap between solving a written-out problem and executing it interactively.

The paper connects formal reasoning with generated interactive environments. Solver-assisted reasoning and manually specified simulators provide computable outcomes, while generated tools and environments reduce repeated domain authoring; learned or generated evaluators require separate validation. VHD-Play retains a precomputed reference from the mechanism supplying the interactive dynamics, providing a standard against which the generated realization can be checked.

3 Formulation

An agentic environment is a process whose observations depend on state and whose actions change subsequent possibilities. The formulation begins with a mathematical decision model and renders its transitions, information structure, constraints, and objective as an environment. That model supplies both the interactive process and its evaluation standard; the player need not reconstruct it symbolically.

3.1 Construction order

Environment-first construction fixes an environment before defining its outcome rule or annotating trajectories. Mechanism-first construction instead solves M(θ), the mathematical model with sampled parameters θ, to obtain reference outcomes zθ and fix the outcome rule before realizing executable dynamics D(θ) and wrapping them as the player-facing environment E(θ). Figure 2 shows when these artifacts become available; an environment-first system may also have known, sophisticated dynamics.

Environment-first and mechanism-first construction orders.

3.2 Model, dynamics, and interface

A mechanism family defines a distribution over multi-period decision problems. Each environment combines executable state transitions and utility, the numerical value accumulated through interaction, with an observation map specifying what the player can see, an action set, and a shared turn budget. A turn is one answered interface call; probes, actions revealing designated hidden information, and decisions changing state both consume this budget, while an optional computation tool neither reads nor changes environment state.

The builder receives the complete parameter draw and document seed; the player receives only observations exposed by the interface. This creates partial observability: relevant state or parameters remain hidden except for the views permitted by interaction.
Separating the model from its interface permits written-out and interactive presentations of the same instance under the same objective. In inventory control, stock persists, early orders occupy later capacity, and lost sales cannot be recovered, requiring information gathering and decisions with lasting effects.
Figure 3 connects this information boundary to generation: solved instances and document seeds produce environments that undergo execution and supported reference checks before training.

A solved mechanism, its player-facing interface, and the checked environment-generation pipeline.

3.3 Outcome evaluation

A family-specific solver computes a reference pair: the full-information upper reference u*(θ) and the utility u0(θ) of a fixed default policy, a prescribed baseline strategy. Equation (4) scores realized utility u(π;θ), where π is the player’s action-selection policy, as $r(\pi;\theta)=\operatorname{clip}_{[0,1]}\!\left(\frac{u(\pi;\theta)-u_0(\theta)}{u^*(\theta)-u_0(\theta)}\right)$; clip[0,1] restricts the result to the interval from 0 to 1. The default maps to 0 and the upper reference to 1 without a language-model judge; under partial information, the upper reference need not be attainable by an online policy that acts using only information available so far. For language-model judge, see J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learning | Summary.

4 Producing environments at scale

The pipeline samples parameters, computes reference outcomes using arithmetic or solvers, and maps decisions, constraints, and objectives into executable state updates and terminal utility, the accumulated value at an episode’s end. A frozen setter generates a document-grounded scenario, relational database, tool schemas and implementations, and player instructions, using reusable family dynamics where available. Admission rejects or regenerates candidates that fail execution or applicable reference-agreement checks.

Table 1 reports 3,300 admitted environments, at least ten tools per environment, three training families, eleven evaluated families, and scenario seeds from 28 topical domains. The approximately $0.01–$0.03 marginal cost is reconstructed from token use and public rates after reusable family components exist.
Normalized topical entropy, a measure of how evenly domain labels are distributed, is 0.997, close to 1 for a uniform distribution. These labels describe settings and language; mechanism families determine decision dynamics.
Policy training uses GRPO, group relative policy optimization, which compares rollouts, sampled interaction trajectories, from the same instance using their outcome rewards. The trained checkpoint’s generated-family evaluation episodes average 59.7 assistant turns.

Scale and structure of the generated environment corpus.

5 Experiments

The experiments test transfer beyond training mechanisms, the gap between problem solving and stateful execution, and whether environment size and policy learning can increase together. Qwen3.6-35B-A3B is the starting policy, with the trained checkpoint evaluated against it under the same protocol. The dataset contains 2,200 training environments, 300 held-out environments from the three training families, and 800 environments from eight unseen families, with 100 environments per evaluation family.

Generated-family evaluation uses the normalized solver-derived score, while external benchmarks retain their native metrics. Qwen3.7-Max is an additional reference evaluated under its application programming interface protocol, the settings used to query the hosted model.
Training families are inventory DP, routing, and negotiation. The main reported checkpoint follows 34 optimizer steps.

5.1 Training gains and transfer

Training improves every evaluated generated family. Held-out training families gain +0.56 on average; near-OOD, out-of-distribution optimization families unseen during training, gain +0.63; far-OOD groups with distinct decision structures or restricted probing gain +0.24 and +0.16, respectively. Figure 4 reports transfer to three external benchmarks with different instances and interfaces.

BFCL V4, the Berkeley Function Calling Leaderboard’s tool-use evaluation, improves from 61.25 to 64.08 across ten interaction-focused cells, individual evaluation subsets, with a reported gain of 2.84 points computed before rounding. Eight cells improve and two remain unchanged.
TravelBench, a benchmark for constructing multi-day itineraries under coupled timing, budget, commonsense, and personalized constraints, improves plan quality from 0.700 to 0.794 on 20 English tasks. Qwen3.7-Max scores 0.891.
On E-Commerce Bench’s 365-day business simulation, the trained model completes all five runs without bankruptcy over 1,147–1,825 assistant turns. Mean ending balance rises from 54,294 to 182,844, or 3.4×, exceeding Qwen3.7-Max’s 165,224; Base completes four runs and goes bankrupt once.

External transfer to BFCL V4, a 365-day storefront, and TravelBench.

5.2 A learnable gap under stateful interaction

Table 2 compares Written-Out F, a complete instance answered with a Python tool available; Informed-Agentic I, the same stateful environment with parameters revealed upfront; and Agentic A, the environment with parameters acquired through interaction. Across five optimization families, Base scores 0.962, 0.231, and 0.204, respectively, showing that most of its initial shortfall persists when hidden parameters are disclosed. Training raises these scores to 0.992, 0.875, and 0.815, reducing the written-out-to-informed-agentic gap by 84% and the written-out-to-agentic gap by 77%.

Illustrative paired trajectories show improvements in planning across a shared multi-period budget, completing observation before commitment, and stopping costly exploration. These examples describe behavioral changes without estimating failure prevalence.
A descriptive decomposition across eight optimization families attributes +0.030 of a +0.443 mean score gain to increased code-tool adoption, leaving +0.413 within fixed tool-use strata, groups defined by whether an episode used the code tool. Linear programming contributes no adoption-shift term because both models use the tool in every episode.
Forward Kullback–Leibler (KL) divergence, measuring differences between the trained and starting models’ token distributions, is 0.089 nats per assistant token on held-out trajectory states. This is consistent with capability elicitation, improved deployment of existing problem-solving ability, but does not establish that the player reconstructs the underlying mechanism.

Base-to-trained scores under written-out, informed-agentic, and agentic presentations.

5.3 Co-scaling environment and policy frontiers

The co-scaling experiment increases knapsack size and horizon through 6×5, 8×7, 9×9, and 11×11 items × decision periods; knapsack allocation selects item subsets under capacity and budget constraints. A frozen Qwen3.6-35B-A3B constructs admitted environments, while a separate copy starts from Base and trains at each scale under a matched budget before evaluation on fresh environments at that scale. Figure 5 reports gains of +0.412, +0.551, +0.675, and +0.861; at 11×11, Base scores 0.085 and the scale-trained checkpoint scores 0.946.

Each scale uses a separately trained checkpoint. The experiment establishes learnability under scale-matched training; it does not test one checkpoint’s transfer across scales.
The fixed setter receives complete mechanism structure and reference outcomes, while the player must recover relevant information through the interface. This information asymmetry permits the same starting model to construct tasks that its initial policy handles poorly.

Environment–policy co-scaling across four knapsack sizes and decision horizons.

6 Conclusion

The conclusion presents mechanism-first construction as a way to generate stateful environments and solver-derived trajectory scores from a shared solved model. Across the evaluated mechanisms and external benchmarks, training improves interactive execution while preserving written-out problem solving. The setter and size experiments support generation and learning at the tested scales within the evaluated mechanism families.

Appendix

A Discussion interprets the results through the asymmetry between complete structure available during construction and information recoverable by the player. Mathematical mechanisms supply hidden quantities, coupled constraints, and delayed consequences, while interfaces control their disclosure. The experiments establish feasibility and learnability over the tested families without establishing that players explicitly reconstruct the mathematical model or that operations research, mathematical modeling of constrained decisions, suffices for general agentic competence.
B Admission and verification specifies C1, executability; C2, a valid default region appropriate to the objective; and C3, reward agreement at the optimum where compatible replay exists. An audit samples ten held-out environments from each of eight replay-supported families and runs oracle, default, and idle policies twice: respectively, a solver-derived reference strategy, the prescribed baseline, and a strategy making no commitments. All 480 executions finish without exceptions, all 240 repeated terminal outcomes agree, and no raw utility exceeds its precomputed upper reference; negotiation, sequential stopping, and sequential search have computable references but fall outside this replay coverage.
C Implementation and record structure describes reusable family components and environment records. C.1 Reference computation uses Bellman recursion, solving inventory decisions backward over states; Held–Karp recursion, solving routes over customer subsets; enumeration; HiGHS, the linear-programming solver; and SLSQP, the numerical optimization routine used through scipy.optimize for the separable quadratic program. C.2 Setter passes generates entity relationships, a database, at least ten executable tools, player instructions, a default policy, and an output format; reusable dynamics are supplied where available, other dynamics are instantiated by the setter, and reference outcomes always remain sampler-computed.
C.3 Setter capacity compares thirty requested environments across inventory DP, routing, and tardiness scheduling, which penalizes jobs completed after their due dates. Frozen Qwen3.6-35B-A3B emits 22 records and passes compatible replay on 21, giving 70% effective admission, while Qwen3.7-Max emits and passes all thirty; mean attempts per emitted record fall from 2.50 to 1.03 and interface refinement from 45.5% to zero. Parameter draws are unpaired, so the comparison measures feasibility and realization burden; reconstructed costs are roughly $0.01–$0.03 per admission for both setters, using public rates accessed August 28, 2026, including a limited 50%-off Max promotion and conversion at ¥7 per US dollar.
C.4 Environment and interpreter isolation keeps hidden parameters and references inside environment dynamics and exposes only declared interface operations. All presentations share a state-independent computation utility that cannot access environment internals. C.5 Training implementation uses 64 prompts per step and 16 retained rollouts per prompt, oversampled to 18 attempts, with return-minus-group-mean advantages, the reward differences used for updates, no standard-deviation normalization, no KL penalty, and no value model estimating expected returns; one epoch per batch uses learning rate 2×10−6 and clip ratio 4×10−3, limiting the policy update, producing 34,816 scored rollouts from 2,176 environments over 34 steps.
D Representative generated environments traces three mechanisms into operational scenarios with positive default-to-optimum gaps and successful reference replay. D.1 Inventory control as hardware replenishment uses three products over six days, shared truck capacity six, and joint setup cost 37.69, with raw utility 22.74→45.01. D.2 Routing as education logistics uses seven sites over three days and capacity twenty, with raw utility −117.81→36.40; D.3 Knapsack allocation as adaptive-sports planning uses eleven items over nine days, shelf capacity seventeen, and shared budget 76, with raw utility 371.45→1,340.96.
E Construction, feedback, and optimization separates environment source, outcome provenance, where the scoring standard comes from, feedback placement, and policy optimization. E.1 Construction order and method coverage locates learned evaluators, rubrics, trajectory annotations, and executable verifiers along these independent axes. E.2 A released environment-first comparison contrasts VHD-Play with EnvScaler, which generates stateful program skeletons, reusable code defining state, rules, and tools, and tasks before generating checklist validators that score the final state.
The matched EnvScaler control in E.2 uses the same starting model, GRPO settings, 2,200 training records, and 34-step budget, drawing scenarios from 51 program skeletons. Its native training reward reaches 0.854, but external transfer is mixed: BFCL averages 58.56 against Base’s 61.25 and VHD-Play’s 64.08; TravelBench scores 0.753 against 0.700 and 0.794; mean storefront balance is 52,661 against 54,294 and 182,844, with both trained models completing all five runs without bankruptcy. This compares complete training pipelines without isolating construction order as a causal factor; E.3 Feedback and optimization separately explains that mechanism-first construction permits terminal or process feedback, rewards attached during a trajectory, and different optimizers, although this implementation uses a terminal scalar with GRPO.
F Evaluation settings specifies a maximum generated-family episode length of 1,000 turns, a per-turn generation cap of 16,384 tokens, cumulative generation cap of 120,000 tokens, prompt cap of 8,192 tokens, and managed context window of 65,536 tokens. Trained generated-family episodes have median 31, mean 59.7, and 95th percentile 211 assistant turns. The 3,300 environments comprise 2,200 training records and 1,100 evaluation records across eleven families.
G Mechanism families catalogues inventory DP, routing, negotiation, knapsack, linear programming, quadratic programming, sequential stopping, sequential search, facility location, scheduling, and scheduling with tardiness. These respectively include bargaining, selecting when to accept a sequentially revealed option, paying to inspect options before choosing, selecting service facilities, and ordering jobs; optimization defaults act on visible sticker values, split shared budgets evenly, and never probe, while other defaults are family-specific. Sequential stopping and sequential search use hindsight upper bounds computed with information unavailable during online action, so their scores compare models against a common reference without measuring distance to an attainable online optimum.
H Additional experimental results supplies training curves, full score grids, behavioral cases, external results, and tool-use controls. H.1 Training dynamics reports inventory reward 0.408→0.904, routing 0.150→0.898, and negotiation reaching 0.470 from near zero; these curves document optimization on training instances, while held-out evaluation supplies transfer evidence. H.2 Generated-family results reports positive agentic gains in every evaluated family, from +0.066 for sequential search to +0.726 for knapsack, and estimates forward KL at 0.089 nats per token on 96 held-out states, with a task-bootstrap 95% interval [0.082, 0.097] obtained by resampling tasks.
H.3 Trajectory evidence presents three illustrative paired cases. H.3.1 Horizon-wide planning contrasts Base spending 27 of 29 budget units in three periods and scoring 0 with a trained seven-period plan scoring 1; H.3.2 Information acquisition before commitment contrasts revealing four of eight trajectories with revealing all eight before planning. H.3.3 Stopping costly information acquisition contrasts inspecting fourteen candidates at cost 134 and net value −40.71 with eight inspections at cost 27 and net value 66.29, yielding trained score 0.799.
H.4 External-benchmark details provides all ten BFCL cell scores, the three benchmark aggregates, and a mean of 1,359 assistant turns for completed trained storefront runs. H.5 Code-tool adoption control uses an Oaxaca–Blinder decomposition, a descriptive separation of adoption changes from within-group score changes: code-tool adoption rises from 50.4% to 79.2% across eleven families, while +0.030 of the +0.443 mean gain across eight optimization families is attributed to the adoption shift. Most of that difference remains within fixed tool-use groups; this decomposition does not establish causal mediation.
I Admitted-environment statistics reports mean, median, and maximum setter attempts of 1.41, 1, and 5; first-attempt admission of 65.6%; and interface rewriting for 31.6% of admitted environments. Aggregate reconstructed token use is approximately 900 input and 6,300 output tokens per admission, supporting the approximately $0.01–$0.03 public-rate estimate. The three-family generation pool spans 28 domain labels with overall normalized entropy 0.997 and at least 0.995 within each family, describing topical balance.
J Environment-complexity sweep gives the exact scale-matched results: 6×5, 0.432→0.843; 8×7, 0.321→0.872; 9×9, 0.237→0.912; and 11×11, 0.085→0.946. Reported gains are +0.412, +0.551, +0.675, and +0.861, respectively. Every trained score comes from a separate checkpoint evaluated on fresh environments at its training scale.
K Limitations and scope restricts the evidence to mechanisms with computable reference outcomes, three training families, eleven evaluated families, and reference replay on eight families along oracle, default, and idle paths. The main generated-family experiment uses one training run and one evaluation seed, and training receives only terminal supervision. Presentation, tool-use, and scale analyses characterize behavior, while training on the complete construction does not separately attribute gains to the outcome rule and the interface.

Brief Thoughts

The central methodological contribution is the shared origin of executable dynamics and a reference standard computed before generation. The F/I/A comparison provides direct evidence for a substantial execution gap: performance falls from 0.962 to 0.231 when fully revealed parameters must be used through stateful interaction, before information gathering is required.

External transfer and the matched EnvScaler control support the practical value of this training corpus under the tested starting model and update budget. Generality remains constrained by one main training run, five storefront runs per model, twenty TravelBench tasks, and replay coverage limited to eight families. Multiple training seeds and controlled changes to the interface and outcome rule would help assess reproducibility and identify which components produce the gains.