Long-WAM: Scaling the Context of World-Action Models | Summary

This article explains the key points of Long-WAM: Scaling the Context of World-Action Models.

• 2026-10-07 (arXiv)
• Huang, Wei, Zhang, Bohan, Liu, Chenzhi, Liu, Isabella, Yang, Shuai, Mao, Weian, Wang, Luozhou, Xiao, Yicheng, Lin, Weifeng, Hu, Qixin, et al.
• NVIDIA, MIT, HKU, UCSD
• Paper
• Github
• Project Page

Read this article in Korean

Summary

• Robot controllers need visual history to infer motion and task progress, but processing that history can delay their response. On the RoboCasa GR-1 tabletop simulation benchmark, extending Long-WAM’s observed history from 0.0 to 19.2 seconds raises success from 63.3% to 78.7%, whereas an initialization pretrained with attention in both temporal directions shows no net gain over that interval.
• Long-WAM first learns autoregressive (AR) video prediction, which predicts future video from preceding observations, using robot and egocentric videos, recorded from a person’s viewpoint, without action labels. It preserves this temporal ordering when adapting the video foundation into a world-action model (WAM), which connects video prediction to executable robot actions.
• The preferred inference strategy predicts future visual latents, compressed video representations, before generating an action chunk, a sequence of robot commands. Long-WAM achieves 99.5% success on LIBERO-Long, a suite of longer manipulation tasks, 94.4% average success on the bimanual RoboTwin 2.0 benchmark, and 34.9% success on the dynamic-manipulation benchmark DOMINO after dynamic-data fine-tuning, leading the methods compared in the paper.
• Hardware and software optimizations reduce four-step-per-expert RTX 5090 inference from 356.0 to 107.4 ms per action chunk, including full observation encoding; streaming encoding separately moves preparation before the inference trigger while inference overlaps robot execution. Physical experiments report 19/20 successes on dynamic cup stacking, compared with 0/20 for both π₀.₅, a vision-language-action policy baseline, and Fast-WAM, a world-action baseline, and 81.7% average success across three tasks on the YAM bimanual manipulator lasting over 40 seconds on average.
• The context study trains a separate policy for each history window, so it does not demonstrate a single policy that adjusts its history at inference time. Longer history also increases computation: RTX 5090 chunk latency rises from 107.4 ms at 2.4 seconds to 341.0 ms at 19.2 seconds, and success declines at the longest tested window, where 80.4% of sampled history frames are padding.

1. Introduction

A single image can show object position while leaving motion and interaction progress ambiguous. The paper studies whether additional observed history improves control and whether those benefits can be retained under real-time computation constraints.

• Retaining observations does not necessarily teach a model to predict from them. The robot-domain AR initialization’s advantage over the bidirectionally pretrained initialization grows from 3.3 percentage points without history to 17.1 points at 19.2 seconds.
• Figure 1 connects video-only predictive pretraining, action adaptation that preserves temporal ordering, context-length evaluation, and accelerated execution.

The overview connects predictive pretraining to action generation and deployment. Its context curves show that useful history duration differs between LIBERO-Long and RoboCasa GR-1, so the results do not establish one optimal window for every task.

2. Related Work

DreamZero and LingBot-VA adapt bidirectionally pretrained video generators to causal control, where visual predictions depend only on their own and preceding temporal blocks. LingBot-VA 2.0 jointly pretrains causal video and learned latent actions; Long-WAM instead separates action-free predictive pretraining from learning the robot’s action representation.

• Memory compression, retrieval, and persistent caches address how history is retained. Long-WAM examines how the duration of the robot’s own observed history interacts with predictive pretraining.
• Earlier deployment approaches use caching, fewer generation steps, action-only decoding, or constraints on transitions between action chunks. Long-WAM retains future prediction and evaluates asynchronous execution, which runs inference while the robot executes an earlier chunk.

3. Scaling the Context of WAMs from AR Video Generation

The method learns robot motion and object-interaction dynamics through long-sequence AR video pretraining, then transfers that foundation to action generation while preserving its causal temporal structure. Actions depend on both observed history and predicted futures, making prediction an intermediate representation for control.

3.1. LongLive2.0-Robot: Robot-Domain AR Video Pretraining

LongLive2.0-Robot continues training from LongLive-2.0’s 16-second AR checkpoint on approximately 10,000 window-equivalent hours from RoVid-X, AgiBot World, EgoDex, EgoVerse, and VITRA. This estimate assigns a fixed duration to each training sample rather than measuring unique footage; video-only supervision allows sources to contribute without sharing action coordinates or actuator dimensions.

• Training uses sequences up to 30 seconds long and sequence parallelism, which distributes long sequences across graphics processing units (GPUs). Teacher forcing conditions each noisy target chunk on its ground-truth prefix, while block-causal attention prevents access to later chunks.
• Error recycling introduces buffered model errors into training inputs while retaining the original clean target. This teaches recovery from rollout-like perturbations; the conditioning image remains clean and is excluded from the loss.

3.2. Causal-to-Causal World-Action Adaptation

The video and action experts, the model components responsible for each modality, use an asymmetric attention interface. Video queries read their own and earlier visual blocks without reading actions; action queries read observed history, partially denoised future video, and the entire noisy action chunk.

• The paper calls its predict-then-act strategy inverse dynamics modeling (IDM): predict future visual latents first, then infer actions conditioned on those predictions. Figure 2 contrasts IDM with current-observation-only actions, history-conditioned actions without future prediction, and co-denoising (CoD), which updates video and actions jointly.
• Training uses two flow-matching passes, which learn noise-to-data generation through velocity prediction: a video pass conditioned on clean history and an action pass conditioned on history plus a forward-noised ground-truth future at noise level σ⋆ = 0.9. The action pass detaches the visual cache, so its loss updates the action expert and proprioceptive adapter, which incorporates robot state, without propagating through that cache.
• At inference, four video denoising steps stop at σ⋆ = 0.9. Future latents are never decoded into pixels, and their key–value (KV) cache, stored attention keys and values, is reused throughout action denoising.

The attention masks distinguish exposing actions to history from conditioning them on an explicit predicted future. Sequential video-then-action inference makes future visual features reusable during action denoising while introducing a sequential inference cost.

3.3. Context Scaling of World-Action Models

The context experiment varies the duration of real observations while holding the forecast and action horizons fixed within each benchmark. Each window uses a separately trained policy evaluated at its training context length; zero history retains the current observation, and unavailable early history repeats the initial frame.

• The initializations are Wan2.2 with bidirectional pretraining, LongLive-2.0 with AR pretraining, and LongLive2.0-Robot with robot-domain AR pretraining. All undergo causal WAM adaptation, so bidirectional describes pretraining rather than the adapted policy’s temporal mask.
• RoboCasa GR-1 uses a 0–38.4-second sweep, with a complementary study on LIBERO-Long. The protocol tests whether additional evidence improves the same near-term decision, while longer prefixes increase attention, cache storage, and prefill, the computation that builds the conditioning cache.

4. Long-WAM Infrastructure: Real-Time Edge Deployment

The deployment system combines asynchronous execution with accelerated local inference on RTX 5090, DGX Spark, and Jetson AGX Thor. A streaming causal variational autoencoder (VAE), the video encoder that produces visual latents, processes incoming observations before the next inference trigger to reduce preparation work after that trigger.

4.1. Asynchronous Execution

The controller overlaps inference with robot execution without action blending, which mixes consecutive predictions, or prefix guidance, which constrains a new chunk using preceding actions. Of each H-step predicted chunk, the first R steps are eligible for execution; inference has nominal stride S control steps, with R/2 ≤ S < R ≤ H, leaving overlap O = R − S.

• At handoff, elapsed actions are discarded and the aligned suffix of the new chunk is executed. Avoiding a wait requires T_ready ≤ OΔt = (R − S)Δt, where Δt is the control interval and T_ready includes transfer, queueing, and computation after the trigger observation becomes available.
• Figure 3 shows how encoding observations as they arrive can meet this deadline when full-window encoding causes repeated waits. At fixed R, shorter overlap allows later inference triggers but leaves less preparation time.

At the same nominal trigger stride and overlap, processing observations before the trigger reduces the work remaining before action handoff. The lower schedule illustrates how full-window encoding can accumulate idle periods and delay subsequent triggers.

4.2. Efficient Edge Deployment

Video-expert linear layers use W4A4 NVFP4, four-bit weights and activations, while action computation and KV storage retain BF16, a 16-bit floating-point format. CUDA Graph replay, which replays captured operations, and PyTorch compilation reduce the launch overhead introduced by small quantization kernels.

• Within each inference call, the implementation reuses quantities unchanged across denoising steps, including text/state KV, observed-video KV, and rotary position embedding tables, which encode token positions. Shared-input quantization processes the common input to query, key, and value projections once.
• Online softmax combines attention over separate video and action buffers with shared normalization, avoiding buffer concatenation. Device-specific tuning changes matrix-multiplication tiles, execution configurations, and attention or VAE backends to address each platform’s hardware constraints.

5. Experiments

Evaluation spans four simulation benchmarks and eight physical task configurations. It covers single-arm and bimanual manipulation, moving-object interception, household task sequences, and longer tabletop tasks, with separate analyses of context, prediction strategy, planning, and execution latency.

5.1. Implementation Details

LongLive2.0-Robot pretraining uses approximately 30,720 GPU-hours on 64 NVIDIA H100 GPUs; downstream WAM training uses 16 GB200 GPUs. Action adaptation uses the AdamW optimizer with a peak learning rate of 10⁻⁴, cosine decay, 5% warmup, BF16 precision, and gradient clipping at 1.0.

• Physical evaluations use a Unitree G1 humanoid and a YAM bimanual manipulator. Main simulation results on LIBERO, RoboTwin 2.0, and DOMINO use up to 2.4 seconds of context.

5.2. Results on Simulation Benchmarks

Long-WAM (IDM) reaches 99.5% average success across LIBERO suites and 99.5% on LIBERO-Long. Its LIBERO-Long result exceeds LingBot-VA by 1.0 percentage point and Fast-WAM by 4.3 points.

• On RoboTwin 2.0, it achieves 94.7% on Clean and 94.2% on Randomized, the benchmark’s two evaluation conditions, averaging 94.4%. This average exceeds ABot-M0.5 by 0.3 points and LingBot-VA 2.0 by 0.8 points.
• After dynamic-data fine-tuning on DOMINO, Long-WAM achieves 34.9% success and a manipulation score, the benchmark’s additional performance measure, of 45.1. These exceed Fast-WAM’s success by 15.0 points and PUMA’s manipulation score by 10.1 points.
• The 19.2-second RoboCasa GR-1 policy reaches 78.7%, exceeding the strongest listed baseline, Cosmos Policy at 67.1%, by 11.6 points.

5.3. Ablation Studies

Figure 6 shows task-dependent context benefits: LIBERO-Long peaks at 2.4 seconds, while RoboCasa GR-1 benefits through 19.2 seconds. On GR-1, increasing context from 2.4 to 19.2 seconds raises success from 66.3% to 78.7% for robot-domain AR pretraining and from 65.7% to 76.7% for the general AR initialization.

• The bidirectional initialization rises from 61.7% at 2.4 seconds to 64.1% at 9.6 seconds, then falls to 61.6% at 19.2 seconds. Access to causal history alone does not reproduce the AR variants’ gains in this comparison.
• At 38.4 seconds, the two AR variants decline to 75.2% and 74.2%. This window is about three times the 12.1-second average training trajectory and contains 80.4% padded history frames; the proposed data-coverage explanation remains a hypothesis.
• Table 1 reports LIBERO-Long success of 94.5% without future-video denoising, 95.8% with CoD, and 99.5% with IDM. The accompanying prose instead gives 97.8% for CoD, which is its tabled overall average, creating an internal discrepancy.

Both AR initializations benefit from longer history on RoboCasa GR-1, while the bidirectional initialization peaks earlier and loses its gain. LIBERO-Long largely saturates after 2.4 seconds; the evenly spaced horizontal positions represent unequal durations and do not form a linear time axis.

6. Atomic Execution and Compositional Planning

On RoboCasa365, a benchmark separating individual household skills from multi-stage compositions, the authors add GPT-6 Astra as a high-level planner to a Human300-trained Long-WAM checkpoint with 2.4 seconds of visual context, without further policy training. The planner decomposes goals into atomic instructions, selects how many actions from each chunk to execute, and can issue bounded corrections to the robot’s end effector, its manipulation tool.

• Across 50 tasks, overall success rises from 31.4% to 54.4%. Composite-Seen, compositions represented during training, improves from 15.8% to 38.8%, and Composite-Unseen, unseen compositions of skills, from 6.1% to 35.0%; the planner alone achieves 25.2% overall.
• Querying Long-WAM every 15 steps without a planner reaches 84.4% Atomic-Seen success on individual seen skills, close to the hierarchy’s 85.6%, but only 11.2% and 5.0% on the composite splits. This control supports a planning contribution beyond shortening execution intervals.
• Planner augmentation increases Long-WAM’s overall result by 23.0 points, compared with 13.6 points for π₀.₅. These system-level comparisons support complementary roles for learned execution and task decomposition.

Adding the planner improves Long-WAM on composite tasks, including an increase from 6.1% to 35.0% on Composite-Unseen. The 54.4% overall result averages 18 Atomic-Seen, 16 Composite-Seen, and 16 Composite-Unseen tasks, making the split results necessary for interpreting the aggregate.

6.1. Real-world Deployment

Physical policy evaluations use 20 trials per policy and condition. Long-WAM’s cup-grasping success at conveyor speeds of 3.0, 4.5, 6.0, and 7.5 cm/s is 100%, 100%, 95%, and 90%, respectively; Fast-WAM falls from 75% to 0%, and π₀.₅ from 15% to 0%.

• Dynamic stacking at 3 cm/s requires intercepting a moving green cup and placing it inside a blue cup. Long-WAM succeeds in 19/20 trials, while both compared policies achieve 0/20.
• On YAM, brick sorting, dumpling placement, and bowl stacking achieve 16/20, 16/20, and 17/20 successes. These tasks last over 40 seconds on average and yield 81.7% mean success; the reported YAM results do not include a policy baseline comparison.

Increasing conveyor speed separates Long-WAM from the two policy baselines: its observed grasping success remains 90–100%, while both baselines reach 0% at the highest speed. Stacking additionally requires coordinated placement after interception, with 19/20 Long-WAM successes and none for either baseline.

6.2. Efficiency

On RTX 5090, optimized V4/A4, four denoising steps for each expert, reduces inference latency from 356.0 to 107.4 ms while RoboTwin 2.0 success changes from 94.4% to 93.5%. V2/A2, two steps per expert, reaches 81.8 ms and 92.5% success.

• Optimized V4/A4 latency is 328.2 ms on DGX Spark and 378.7 ms on Thor, corresponding to 4.1× and 3.2× speedups over their BF16 eager baselines, which execute without the listed optimizations. Table 8 shows cumulative gains, so its rows do not measure each optimization independently.
• With execution horizon R = 24 and trigger stride S = 12, IDM success changes from 94.4% synchronously to 94.2% asynchronously. Fast-WAM falls from 91.8% to 76.4%, and LingBot-VA from 92.2% to 46.7%.
• Latency comparisons use the baselines’ native configurations and denoising budgets. The timing protocol includes full observation VAE computation but excludes preprocessing, text encoding, controller overhead, and interprocess communication, so these values do not measure the complete observation-to-actuation delay.

The measurements show cumulative gains from shared optimizations followed by platform-specific tuning. Final latencies differ across devices, and every row retains preceding changes, so the table does not measure independent additive effects.

Appendix

• A. Discussion and Conclusion argues that useful history duration depends on task requirements and available computation. Each window is trained separately; adaptive windows within one policy and real-robot context-length sweeps are future work, and the observed trends do not establish a power law or universal memory optimum.
• B. Data and Model Training Details, including B.1. Pretraining Data, specifies 2,294,889 model-ready samples across five dataset families and six source subsets. Training, validation, and test splits contain 2,251,021, 21,931, and 21,937 samples; assigning 16 seconds to each training sample yields 10,004.5 window-equivalent hours rather than a measurement of unique raw footage. Action-free pretraining does not itself demonstrate improved transfer to unseen robot embodiments.
• B.2. Training Objectives and Inference Interface clarifies that padded future latents and actions are masked out of the adaptation loss. Training supplies a forward-noised ground-truth future, whereas inference generates its future from Gaussian noise; matching σ⋆ = 0.9 does not make these conditioning distributions identical. Robot-state changes, context eviction, and temporal-position changes can invalidate visual KV across decisions, so within-call reuse does not establish unrestricted persistent-cache reuse.
• C. LongLive2.0-Robot Video Predictions presents selected image-and-instruction-conditioned predictions for lid placement, shoe transfer, drawer opening, pouring, and T-shirt folding. These examples illustrate coordinated predicted motion but provide no aggregate video-prediction metric. D. Baseline Details identifies comparison sources, including external RoboCasa365 leaderboard scores and Qwen-RobotManip scores from its technical report.
• E. Discussion of Asynchronous Overlap Length fixes R = 24 and tests trigger strides of 12, 16, and 20, producing overlaps of 12, 8, and 4 control steps. Success is 94.2%, 95.0%, and 94.3%, while overlap root-mean-square error, which measures disagreement between temporally aligned action predictions, and a discrete jerk proxy both decrease. The jerk proxy measures abrupt changes using third differences of non-gripper joint states without division by Δt³; smoother transitions do not monotonically increase task success.
• F. IDM versus Co-Denoising: Capability and Inference Latency reports CoD latencies of 90.9 ms for V4/A4 and 65.7 ms for V2/A2, compared with IDM’s 107.4 and 81.8 ms. Timing averages the medians from two independent processes, each with five excluded warmups and 30 steady-state samples; the variants use their own trained schedules and cache implementations, so the comparison does not isolate execution order alone. Inference rates are reciprocals of these latencies, not robot control frequencies; observed-video KV reuse also has device- and denoising-budget-dependent effects.
• G. Context-Dependent Inference Latency reports RTX 5090 chunk latencies of 74.6, 107.4, 138.3, 204.5, and 341.0 ms for histories of 0.0, 2.4, 4.8, 9.6, and 19.2 seconds. Increasing history eightfold from 2.4 to 19.2 seconds raises latency approximately 3.2×; streaming encoding separately reduces the preparation performed after the inference trigger.
• H. Additional Real-World Deployment Visualizations supplies selected overview and wrist-camera sequences. H.1. Dynamic Composite Manipulation shows baseline interception failures and Long-WAM’s completed cup nesting; H.2. Moving-Object Grasping at Different Speeds shows selected grasps across four conveyor speeds; H.3. Long-Horizon Manipulation shows successive object interactions in the three YAM tasks. These montages illustrate behavior without providing additional aggregate success estimates.

Brief Thoughts

The clearest evidence concerns the interaction between initialization and context length: under causal action adaptation, both AR foundations improve with longer GR-1 history while the bidirectional foundation does not sustain its improvement. This supports predictive pretraining as a useful design choice, although comparing different pretrained checkpoints does not isolate temporal ordering from all other training differences.

Measured latency reductions and near-preserved asynchronous success support the systems contribution, but the 107.4 ms headline applies to 2.4 seconds of history; 19.2 seconds requires 341.0 ms and produces peak GR-1 success. Physical results demonstrate responsive execution, while a real-robot context sweep remains necessary to establish whether the simulation’s context-scaling trend transfers.

The latency sweep quantifies the cost of longer context: 19.2 seconds of history requires 341.0 ms per chunk compared with 107.4 ms at 2.4 seconds. Peak GR-1 success and the 107.4 ms headline therefore belong to different context settings.

Table 4 labels the 63.3% GR-1 result as 2.4 seconds, while Figure 6 and the ablation prose associate 63.3% with zero history and 66.3% with 2.4 seconds. The LIBERO-Long CoD prose also conflicts with Table 1, so those individual comparisons require qualification even though the peak results are consistently reported. Each 20-trial physical condition has a success-rate resolution of five percentage points, limiting comparisons between nearby rates.