WorldPlay: Towards Long-Term Geometric Consistency for Real-Time Interactive World Modeling | Summary

Contents

• Summary
• 1. Introduction
• 2. Related Work
• 3. Method
  • 3.1. Preliminaries
  • 3.2. Dual Action Representation for Control
  • 3.3. Reconstituted Context Memory for Consistency
  • 3.4. Context Forcing
  • 3.5. Streaming Generation with Real-Time Latency
• 4. Experiments
  • 4.1. Main Result
  • 4.2. Ablation
  • 4.3. Application
• 5. Conclusion
• Impact Statement
• Appendix
• Brief Thoughts

This article explains the key points of WorldPlay: Towards Long-Term Geometric Consistency for Real-Time Interactive World Modeling.

• 2025-12-16 (arXiv), Proceedings of the 43rd International Conference on Machine Learning, Seoul, South Korea. PMLR 306, 2026.
• Sun, Wenqiang, Zhang, Haiyu, Wang, Haoyuan, Wu, Junta, Wang, Zehan, Wang, Zhenwei, Wang, Yunhong, Zhang, Jun, Wang, Tengfei, Guo, Chunchao.
• Hong Kong University of Science and Technology, Beihang University, Tencent Hunyuan
• Paper
• Github
• Project Page

Read this article in Korean

Summary

• WorldPlay: Towards Long-Term Geometric Consistency for Real-Time Interactive World Modeling addresses fast interactive video generation with stable scene geometry when users revisit locations. The model combines Dual Action Representation, Reconstituted Context Memory, and Context Forcing to generate streaming 720p video at 24 FPS on 8×H800 GPUs.
• Discrete commands and continuous camera poses jointly condition a chunk-wise autoregressive diffusion transformer. Retrieved spatial memories complement recent temporal context, while temporal reframing assigns nearby positional indices to historical frames to reduce the weakening of their influence with temporal distance.
• Context Forcing aligns teacher and student memory conditioning during distribution matching, reducing inference to 4 denoising steps. On the 600-case evaluation, the full model achieves long-term PSNR 18.94, SSIM 0.585, LPIPS 0.371, rotation distance 0.332, and translation distance 0.797, outperforming the compared baselines across these metrics.
• The experiments support improved revisit consistency over the tested sequences, with examples spanning real, stylized, first-person, and third-person scenes. The authors describe generation lasting approximately 30 seconds and identify residual error accumulation, occlusion-sensitive retrieval, and broader physical interactions as unresolved limitations.

1. Introduction

Interactive world modeling requires prompt visual responses to user actions and stable scene content across revisits. The introduction distinguishes fast models that neglect memory from memory-based models whose conditioning complicates distillation; WorldPlay addresses both requirements through next-chunk prediction, with each chunk representing 16 video frames.

• Dual Action Representation combines scale-adaptive discrete controls with camera poses that support accurate spatial indexing.
• Reconstituted Context Memory retrieves relevant history and rewrites temporal positions; Context Forcing aligns teacher and student memory conditioning during distillation.
• The project page is https://3d-models.hunyuan.tencent.com/world/ and the online demo is https://3d.hunyuan.tencent.com/sceneTo3D.

Repeated views marked by red boxes illustrate revisit consistency across real, stylized, and third-person scenes. Reconstruction and smoke examples demonstrate additional applications, but the montage does not quantify consistency or response latency.

2. Related Work

The related work covers latent video diffusion, autoregressive video generation, interactive world models, and distillation. Explicit reconstruction-based memory depends on reconstruction quality, whereas implicit memory retrieves historical observations using geometric relevance; the paper proposes making such memory compatible with few-step interactive generation.

• CausVid distills a causal student from a bidirectional teacher, and Self-Forcing improves rollout training to reduce exposure bias.
• WorldPlay extends this distillation setting by aligning teacher and student memory contexts.

The capability comparison assigns WorldPlay continuous and discrete control, 720p output, real-time operation, long-term consistency, and long-horizon generation in a general domain. These categorical entries summarize the authors' claims; later experiments provide quantitative comparisons.

3. Method

WorldPlay predicts the next chunk conditioned on past observations, action history, the current action, and a text prompt or image describing the world. Its pipeline encodes retrieved context, injects both action representations, generates a new chunk with an autoregressive diffusion transformer, and adds the output to the memory cache.

Each generation block receives reconstituted spatial and temporal context, while generated outputs update the cache. This feedback structure shows how selected historical observations condition successive chunks during streaming.

3.1. Preliminaries

The backbone uses a causal 3D VAE and a Diffusion Transformer trained with flow matching. A clean video latent and Gaussian noise are linearly interpolated, and the network learns to predict their difference using the loss L_FM(θ) = E[k,z₀,z₁] ||N_θ(z_k,k) − v_k||², where \(v_k=z_0-z_1\).

• Four temporal latents form one chunk, which can decode into 16 frames; block causal attention replaces full-sequence attention.
• Training assigns different noise levels to different chunks, allowing the model to predict subsequent chunks autoregressively.

3.2. Dual Action Representation for Control

Discrete inputs provide plausible movement across scenes with different scales, but they do not precisely identify previous camera locations. Continuous poses supply those locations but introduce training difficulties from scale variation; WorldPlay combines the two representations to support control and spatial retrieval.

• Positional encoding and a zero-initialized MLP encode discrete keys into the timestep embedding that modulates DiT blocks.
• PRoPE injects camera-frustum relationships into causal self-attention using intrinsic and extrinsic camera parameters.
• The camera-conditioned attention branch is added through a zero-initialized projection to the original attention output.

Discrete key tokens join the time embedding, whereas camera information enters causal self-attention through a separate PRoPE branch. The two action representations therefore condition different parts of the transformer.

3.3. Reconstituted Context Memory for Consistency

Reconstituted Context Memory selects recent chunks for temporal continuity and non-adjacent historical observations for spatial consistency. Spatial selection uses both field-of-view overlap and camera distance, avoiding the computation and redundancy of conditioning on every past frame.

• Standard absolute RoPE indices make old memories increasingly distant from the current prediction, potentially exceeding the trained positional range and weakening their influence.
• Temporal Reframing dynamically reassigns context positions to maintain a fixed, small relative distance to the current chunk, regardless of the actual elapsed time.

Full context grows with history, selected memory with absolute indices retains large temporal gaps, and relative reindexing keeps retrieved observations near the current chunk. The comparison separates memory selection from the positional treatment of selected memories.

3.4. Context Forcing

Context Forcing addresses the conditional distribution mismatch between a memory-aware causal student and a bidirectional teacher. The student self-rolls out 4 chunks using reconstituted memory, while the teacher receives the corresponding memory context with those rollout chunks excluded.

• Aligning memory conditioning supports distribution matching without requiring the teacher to process an entire long video.
• Progressive distillation increases the rollout length during training and produces a student using 4 denoising steps.
• Historical context is sampled from real videos. The ablation shows artifacts when historical context is self-generated, which the authors attribute to the teacher having been trained with clean memory.

The autoregressive rollout produces samples evaluated by frozen real-score and trainable fake-score bidirectional models. Aligned memory conditioning makes their score comparison applicable to the student's memory-conditioned generation.

3.5. Streaming Generation with Real-Time Latency

The reported 24 FPS at 720p requires Context Forcing together with deployment optimizations on 8×H800 GPUs. Sequence and attention parallelism distribute chunk tokens across devices, while progressive VAE decoding streams smaller frame batches through the NVIDIA Triton Inference Framework.

• Sage Attention, float quantization, matrix multiplication quantization, and KV caching reduce inference cost.
• The paper reports throughput under this hardware configuration but gives no numerical end-to-end input latency or time-to-first-frame.

4. Experiments

Training uses approximately 320K real and synthetic video samples. The main test set contains 600 cases from DL3DV, game videos, and AI-generated images, with short-term evaluation over 61 frames and long-term evaluation over at least 250 frames.

• Short-term videos follow reference camera trajectories and are compared with ground-truth frames.
• Long-term evaluation uses cycle trajectories and compares return-path frames with corresponding frames generated during the initial pass.
• PSNR, SSIM, and LPIPS measure visual agreement. ViPE estimates generated camera poses for rotation and translation errors, with the first pose set to identity and translation scale normalized using the furthest frame.
• Baselines are CameraCtrl, SEVA, ViewCrafter, Matrix-Game-2.0, GameCraft, Gen3C, and VMem.

4.1. Main Result

The full model achieves short-term PSNR 21.92, SSIM 0.702, LPIPS 0.247, rotation distance 0.031, and translation distance 0.121. It leads the compared methods on visual metrics and translation error, while Gen3C achieves the lowest short-term rotation distance, 0.024, followed by ViewCrafter at 0.029.

• In the long-term setting, WorldPlay scores 18.94, 0.585, 0.371, 0.332, and 0.797 respectively, leading all compared baselines across the five metrics.
• Without Context Forcing, long-term PSNR is 16.27, SSIM 0.425, LPIPS 0.495, rotation distance 0.611, and translation distance 0.991; the distilled model improves each metric.
• The MEt3R structural consistency score is 0.133 for WorldPlay, versus 0.187 for Gen3C, 0.367 for Matrix-Game-2.0, and 0.305 for GameCraft; lower scores indicate better consistency.
• Acceleration measurements increase throughput from 2.80 FPS to 3.80 FPS with DiT parallelism and quantization, to 16.0 FPS with corresponding VAE optimizations, and to 24.0 FPS with streaming decoding.

WorldPlay leads all compared methods on the five long-term metrics and improves each metric over its version without Context Forcing. Its short-term rotation distance of 0.031 exceeds Gen3C's 0.024 and ViewCrafter's 0.029.

The examples compare WorldPlay with Gen3C, GameCraft, and Matrix-Game 2.0 under illustrated action sequences. Red boxes identify scene elements preserved on return in WorldPlay, providing selected qualitative examples alongside the quantitative revisit results.

WorldPlay's MEt3R score of 0.133 is lower than all three listed baselines, including Gen3C at 0.187. The metric evaluates structural agreement between matched initial-pass and revisit frames.

The staged measurements show that VAE parallelism and quantization increase throughput from 3.80 FPS to 16.0 FPS, and streaming decoding raises it to 24.0 FPS. The complete configuration achieves an 8.57× improvement over 2.80 FPS; the table does not isolate parallelism from quantization.

4.2. Ablation

The ablations support combining discrete and continuous actions, reframing temporal positions, and aligning memory during distillation. Dual actions achieve rotation distance 0.028 and translation distance 0.113, compared with 0.103 and 0.615 for discrete actions alone and 0.038 and 0.287 for continuous actions alone.

• On long-term data, Reframed RoPE improves PSNR from 14.03 to 16.27, SSIM from 0.358 to 0.425, LPIPS from 0.534 to 0.495, rotation distance from 0.805 to 0.611, and translation distance from 1.341 to 0.991.
• Latent-level teacher memory selection misaligns context with the chunk-based student and produces collapsed outputs in the qualitative experiment.
• A memory-less teacher weakens revisit consistency, while self-generated historical context introduces artifacts. These context alternatives are illustrated qualitatively without a numerical ablation table for each alternative.

Combined action conditioning achieves the best value in every reported column, including PSNR 22.09 and translation distance 0.113. The larger pose errors for discrete-only conditioning support adding continuous spatial information.

Reframed RoPE improves all five long-term metrics relative to standard RoPE, including PSNR from 14.03 to 16.27 and translation distance from 1.341 to 0.991. Both visual agreement and estimated pose accuracy improve under temporal reframing.

The upper example shows visual degradation under standard RoPE, while the lower example highlights changed scene details after a revisit. Reframed RoPE preserves these selected details more closely, complementing the numerical positional-encoding ablation.

Misaligned context produces collapsed outputs, while the memory-less teacher and self-generated historical context show consistency or appearance failures. The final row illustrates the selected training design, without numerical effect sizes for these alternatives.

4.3. Application

Generated videos can be passed to a separate feed-forward 3D reconstruction model to produce point clouds. Text prompts can also change during streaming to trigger environmental transitions or object appearances, including smoke, balloons, and rainbows.

• The reconstruction examples demonstrate a downstream use of generated multiview observations; the point clouds are produced by the separate reconstruction model.
• Promptable events are demonstrated visually, without a dedicated quantitative evaluation of event correctness.

Point clouds reconstructed from generated indoor and stylized outdoor views show that the videos can supply observations to a separate reconstruction model. The examples demonstrate this application without measuring reconstruction accuracy.

The sequences show a balloon appearing and a rainbow forming while navigation continues. They illustrate how text changes affect subsequent streamed content, without measuring event reliability across a benchmark.

5. Conclusion

The conclusion brings together action control, memory retrieval, and distillation for interactive video world modeling. Its stated limitations include generation lasting approximately 30 seconds, unresolved autoregressive error accumulation, restricted interaction types, and field-of-view retrieval failures under significant occlusion.

• Efficient scaling to minutes or hours remains a challenge.
• Multi-agent interaction and complex physical dynamics are proposed extensions rather than experimentally established capabilities.

Impact Statement

The Impact Statement says the work aims to advance Machine Learning and acknowledges potential societal consequences without identifying any that the authors believe require specific discussion. It provides no application-specific risk analysis or mitigation evaluation.

Appendix

• A. Training and Inference Details: The model uses pretrained DiT-based video diffusion backbones, 4 latents per chunk, 3 temporal memory chunks, 1 spatial memory chunk, and the first chunk as an attention sink. Stage One: Action Control trains a bidirectional model for 30K iterations and an autoregressive model for another 30K, using 61 frames, Adam, learning rate 1e−5, and batch size 64.
• Stage Two: Memory trains both models on sequences of up to 160 latents, or 637 frames. Generation noise is sampled from [0,1], memory noise from [0,0.2], and loss is computed only on the generation sequence. Stage Three: Context Forcing uses progressive rollout lengths for 2K iterations with batch size 64, student learning rate 1e−6, and fake-score model learning rate 2e−7; Algorithms 1 and 2 specify training and inference with KV Cache. At inference, pose-only inputs are converted into discrete actions by thresholding relative transformations, while discrete-only inputs are converted into poses using predefined relative transformations.
• B. Dataset: The 320K clips comprise 40K Sekai clips (12.5%), 60K DL3DV-derived clips (18.75%), 50K UE-rendered clips (15.625%), and 170K game recordings (53.125%). Processing includes filtering dense moving objects, reconstruction and custom revisit rendering, Difix3D+ repair, descriptive text annotation, and ViPE pose estimation; missing discrete actions are derived by thresholding camera-pose components. Videos are segmented into 30 to 40 second clips; pose estimates are subsampled every four frames, and clips with an inter-frame motion Peak-to-Median Ratio above 5.0 are discarded.
• C. Additional Experimental Results, C.1. More Qualitative Results, and C.2. Long Video Generation: Additional examples show composite movements, alternating rotations and translations, and control of human and animal agents. Revisit examples compare frame 1 with frame 252, and further examples span 637 frames. The appendix states that generation time per chunk remains constant as video length grows, without presenting a numerical latency-versus-length curve.
• C.3. Comparison of Models under Context Forcing: The student and bidirectional teacher each use 100 function evaluations, compared with 4 for the distilled model. The teacher achieves PSNR 19.31, SSIM 0.599, LPIPS 0.383, rotation distance 0.209, and translation distance 0.717; the distilled model scores 18.94, 0.585, 0.371, 0.332, and 0.797. Distillation improves every listed metric over the original student, while the teacher retains better PSNR, SSIM, and pose accuracy.
• C.4. Ablation for Memory Size: Using 3 spatial and 1 temporal memory chunks gives PSNR 16.41, versus 16.27 with 1 spatial and 3 temporal chunks. The latter configuration has better SSIM, LPIPS, rotation distance, and translation distance. The authors explain that larger spatial memory can enlarge teacher context because spatial selections differ between adjacent chunks, whereas temporal context overlaps.
• C.5. Evaluation on VBench compares videos generated from the same initial image and actions using a radar plot; the authors report advantages in consistency, motion smoothness, and scene generalization. C.6. Evaluation on WorldScore uses 2,000 static-setting cases with an image, text prompt, and camera trajectory. WorldPlay obtains the highest average, 79.74 versus Voyager’s 77.62, although WonderWorld has higher camera control, content alignment, and 3D consistency scores, and Voyager also has higher content alignment.
• D. User Study: Thirty assessors compare paired videos generated from identical initial images and actions, using 300 benchmark cases and 300 customized trajectories. Preference for the distilled model is 88.5% against GameCraft, 78.4% against Matrix-Game-2.0, 92.1% against ViewCrafter, and 72.9% against Gen3C. Against the bidirectional teacher, preference is 48.1% for the distilled model and 51.9% for the teacher, so the reported comparison favors the teacher.
• E. Additional Applications and E.1. Promptable Event: Changing a text prompt refreshes cached key–value states using KV-recache, intended to remove previous prompt information while preserving motion and visual continuity. Demonstrations include weather changes, fire eruption, and new objects or characters. E.2. Video Continuation shows follow-up content consistent with an initial clip in motion, appearance, and lighting, without a separate numerical continuation benchmark.

Brief Thoughts

Aligning memory conditioning during distillation is a well-supported contribution: the student comparison shows improvements across all five long-term metrics, and qualitative ablations show failures under alternative context designs. The reported 24 FPS requires 8×H800 GPUs, and generation lasting approximately 30 seconds leaves persistence over much longer interactive sessions unverified. Cycle-path comparisons test agreement with earlier generated views, and MEt3R adds structural evidence beyond pixel similarity. These measurements do not independently establish the correctness of a persistent 3D world. Pose errors depend on ViPE estimates. The data pipeline’s discussion of pose-estimation failures warrants caution when interpreting these errors as exact camera-control accuracy. The distilled model improves the original student and wins human comparisons against several baselines, but the bidirectional teacher retains advantages in four quantitative metrics and receives 51.9% of paired preferences.