ReVision: Refining Video Diffusion with Explicit 3D Motion Modeling | Summary

This article explains the key points of ReVision: Refining Video Diffusion with Explicit 3D Motion Modeling.

• 2025-04-30 (arXiv), Transactions on Machine Learning Research (01/2026)
• Liu, Qihao, He, Ju, Yu, Qihang, Chen, Liang-Chieh, Yuille, Alan.
• Johns Hopkins University, Independent Researcher
• Paper
• Project Page

Source version. This retrospective W18 article reviews arXiv 2504.21855v2, dated 8 January 2026 and published in TMLR (01/2026). The 2025-04-30 date identifies the original arXiv release; the benchmarks and tables include the later v2 revisions.

Read this article in Korean

Summary

• Video generators often produce plausible images while failing to maintain coherent complex movements and object interactions. Applied to Stable Video Diffusion (SVD), a pretrained video generator, ReVision raises Dynamic Degree, the motion-activity score in the VBench++ video-generation benchmark, from 43.17% to 83.15% relative to the SVD-XT-1.1 baseline, while also improving its reported consistency, smoothness, and imaging-quality scores.
• ReVision generates a coarse video, extracts and corrects an explicit three-dimensional motion representation, and regenerates the video using that corrected motion. Its Parameterized Motion Prior model (PMP), a learned network that repairs motion sequences, supports humans, animals, and general objects through category-appropriate representations.
• The reported 1.5B-parameter ReVision-SVD also receives higher motion preferences than 13B-parameter HunyuanVideo, another video generator, in a study without target-pose inputs, which specify a desired final configuration. These results require selective generator fine-tuning, an annotated 20K-video dataset, and external reconstruction models; generating 32 frames takes 49 seconds versus 36 seconds for SVD in the reported timing comparison.

1 Introduction

ReVision: Refining Video Diffusion with Explicit 3D Motion Modeling addresses the absence of reliable motion guidance in general video generation. Human image animation, which transfers motion to a pictured person using a supplied pose sequence, suggests that explicit guidance can improve complex movement, but complete sequences are usually unavailable for ordinary generation.

• The proposed Extract–Optimize–Reinforce pipeline obtains its initial motion from the generator itself, then corrects that motion before a second generation pass.
• Users may optionally specify final positions of objects or body parts through rough sketches or dragging instead of supplying a complete trajectory.

2 Related Work

The related work covers generation from text or an initial image, motion control through text and optical flow, and human image animation. Optical flow describes image-plane movement between frames; ReVision extracts object-specific motion and refines it in a parameterized three-dimensional representation.

• Existing animation methods use dense guidance such as keypoints, which mark selected body locations, or flow and body-part masks, which label the pixels occupied by each part.
• ReVision constructs and repairs its guidance from an initially generated video when reference motion is unavailable.

3 Preliminary

Latent diffusion generates data by reversing noise corruption in a compressed representation produced by a Variational Autoencoder, an encoder–decoder model. SVD extends an image diffusion model with temporal convolution and attention layers, which process relationships across video frames.

• SMPL-X and SMAL are parametric mesh models for humans and animals, respectively: pose and shape parameters determine their three-dimensional surfaces, and predefined vertex labels provide body-part masks.
• SMPL-X produces 10,475 vertices and uses 165 pose, 10 shape, and 10 expression parameters; SMAL produces 3,889 vertices and uses 105 pose and 41 shape parameters.
• General objects use a 2.5D representation: two-dimensional bounding boxes, which enclose objects, and segmentation masks augmented with estimated depth, rather than complete category-specific surface models.

4 Method

The method first adapts SVD to accept motion guidance, then reuses the adapted generator for coarse and refined video generation. Between these passes, PMP corrects the extracted motion and converts it into image-aligned conditioning, meaning additional inputs that guide generation.

The diagram places motion correction between two uses of the video generator: extracted motion is refined before it becomes dense guidance. It explains why ReVision requires motion reconstruction and a second synthesis pass.

4.1 Motion-Conditioned Video Generation

Motion conditioning adds two channels: a part segmentation mask and a confidence map, a spatial marker of the available guidance. Only the initial convolutional block and temporal layers are fine-tuned, allowing the model to accept different guidance levels without retraining the entire generator.

• Full motion sequences occupy 40% of training examples and receive confidence 1.
• Final-frame polygon target masks occupy 30% and receive confidence 0.5; polygons approximate the rough sketches supplied during inference.
• The remaining 30% use empty masks with confidence 0, preserving generation without explicit motion inputs.

The added mask and confidence inputs let one adapted generator accept complete trajectories, rough final poses, or absent motion guidance. The confidence channel is supplied as a marker during training rather than predicted by a separate reliability model.

4.2 Proposed Method: ReVision

S1: Coarse Video Generation uses the first frame and an optional final-frame target pose to propose motion and scene structure. Because only motion features are retained, this pass can use one-quarter of the original resolution, one-half of the frames, and 32 rather than 50 denoising steps, reducing its reported runtime from 36 seconds to 8 seconds.

S2: Object-Centric 3D Optimization fits human and animal meshes or constructs a compact representation for general objects. For each general object, 16 mask-contour vertices, four bounding-box corners, and the box center form 21 geometric points, which estimated depth lifts into three-dimensional space.

• PMP receives the extracted motion, text features from a pretrained CLIP text encoder, which maps the description into feature vectors, and motion strength, a speed measure computed from changes in adjacent-frame parameters.
• Transformer blocks combine relationships within the motion sequence through self-attention and incorporate text and motion-strength conditions through cross-attention.
• Training corrupts annotated motion by adding noise, shuffling temporal order, or dropping and repeating segments, then teaches PMP to recover the original sequence.

S3: Fine-grained Video Generation projects the corrected three-dimensional motion into two-dimensional part masks and associated confidence maps. The same adapted SVD then regenerates the video with a complete motion sequence as guidance, replacing the sparse or absent guidance used in S1.

5 Experimental Results

The experiments evaluate image-to-video generation, which starts from a supplied image, motion transfer, which reproduces reference movement, human preferences, occlusion handling, runtime, dance generation, and PMP’s contribution. They also test whether the framework transfers to another base video generator and whether PMP improves a separate human-motion generator.

5.1 Image-to-Video Generation

Training uses 20K automatically annotated videos from Panda-70M, a captioned video dataset, with frame-level boxes, semantic masks, depth maps, and reconstructed human or animal meshes. SVD is fine-tuned for 300K iterations with batch size 64 and constant learning rate 2 × 10−5, sampling 16-frame clips at stride 4 and resolution 1024 × 576.

On VBench++, ReVision-SVD improves Dynamic Degree from 43.17% to 83.15%, subject consistency from 95.42% to 96.13%, and imaging quality from 70.23% to 71.48%. ReVision-Wan2.1, trained with the same dataset and settings, raises Dynamic Degree from 51.38% to 73.67% over the Wan2.1-I2V-14B-720P base generator, although motion smoothness decreases slightly from 97.90% to 97.74%.

• The second base-generator comparison supports applicability beyond SVD, with a small tradeoff in motion smoothness.
• Dynamic Degree should be interpreted alongside consistency, quality, and human assessments; increased movement alone does not establish accurate motion.

Both base-generator comparisons show a large increase in Dynamic Degree alongside mostly improved preservation and quality scores. Wan2.1's motion-smoothness score decreases from 97.90% to 97.74%, so the gains are not uniform across metrics.

On DAVIS, a video-object segmentation benchmark used here for motion transfer, ReVision-SVD improves all six reported metrics over MotionClone, ImageConductor, and Go-with-the-Flow, three motion-conditioned generation methods. Its CoTracker mIoU, a mean intersection-over-union overlap score for tracked motion, is 0.80 versus 0.74 for Go-with-the-Flow; optical-flow error falls from 0.36 to 0.33 and pixel mean squared error, the average squared pixel difference, falls from 0.053 to 0.046.

• The main preference study uses 500 activity descriptions generated with GPT-4o, a model used here to create prompts and input images, and five inputs per description for each baseline comparison, totaling 5,000 video pairs.
• No model receives a target pose. Each pair uses seed 42 and three randomly selected raters from Amazon Mechanical Turk (MTurk), a crowdsourcing platform, producing 15,000 comparisons with randomized display order.
• For SVD comparisons, inputs are generated images resized to 1024 × 576; for HunyuanVideo comparisons, ReVision receives the first frames of HunyuanVideo outputs generated at 1280 × 720. These are distinct comparison protocols.

ReVision is preferred on motion consistency, amount of motion, and motion realism against both baselines, with a smaller advantage over HunyuanVideo. No model receives a target pose in these comparisons.

ReVision-SVD leads the listed methods on tracked-motion overlap, flow error, pixel error, subject consistency, background consistency, and smoothness. The table evaluates transfer accuracy and appearance preservation as well as motion.

Generating a 32-frame video takes 49 seconds for ReVision-SVD: 8 seconds for coarse generation, 5 seconds for detection and motion refinement, and 36 seconds for final generation; the paper reports a single A100 GPU for the added detection and refinement stage. SVD takes 36 seconds and HunyuanVideo takes 411 seconds in the timing table, making ReVision approximately 8.4-fold faster than HunyuanVideo while adding 13 seconds over SVD.

• For occlusion, where one object hides another, and interaction preferences, raters assess the same 5,000 pairs; 10,207 evaluations remain after excluding 4,793 responses marked as having no interaction or occlusion.
• Among the valid comparisons, ReVision receives 97.63% preference against SVD and 63.99% against HunyuanVideo.
• Depth-aware projection provides a mechanism for ordering overlapping objects, but these preference results do not establish general physical correctness.

ReVision adds 13 seconds to SVD while remaining approximately 8.4-fold faster than HunyuanVideo in the reported comparison. Despite the source caption's claim of matching SVD's speed, the numerical entries show 49 seconds versus 36 seconds.

For longer generation, PMP expands a 32-frame motion sequence to 64 frames by interpolation, which inserts intermediate motion states, and then to 128 frames by extrapolation, which extends motion beyond the original sequence, and refinement. Video synthesis uses overlapping 32-frame windows with stride 24 and overlap 8, aligns motion conditions across overlaps, and blends the overlapping frames while sharing appearance and identity conditioning.

• The demonstration supports motion extension through parameter-space resampling and clip stitching; the paper does not report a dedicated quantitative long-video benchmark.

5.2 Complex Motion Generation

Dance experiments use the TikTok Dancing dataset with the train/test split used by the Disco dance-generation method, crop videos to 576 × 1024, and fine-tune SVD on the training split for 30K iterations with batch size 8 and learning rate 1 × 10−5. With full motion guidance, ReVision achieves Structural Similarity Index (SSIM), an image-structure similarity score, 0.864; Peak Signal-to-Noise Ratio (PSNR), an image reconstruction-quality measure, 30.08; Learned Perceptual Image Patch Similarity (LPIPS), a learned perceptual image-difference measure, 0.210; and Fréchet Video Distance (FVD), a video-distribution fidelity metric, 121.26.

• Higher SSIM and PSNR and lower LPIPS and FVD indicate better results under the reported evaluation; the full-motion variant leads all listed baselines on these metrics.
• Image-only and target-pose variants obtain FVD 136.43 and 130.14, respectively, versus 152.97 for VividPose, a human image animation baseline. Their paired-frame SSIM, PSNR, and LPIPS values are not reported.
• The full-motion comparison uses the guidance supplied to animation baselines; the sparse-input variants support a separate claim about generation with less motion guidance.

The table separates full-motion conditioning from image-only and target-pose generation. All three ReVision settings improve FVD over the listed baselines, but paired-frame quality metrics are reported only for the full-motion setting.

5.3 Parameterized Motion Prior Model (PMP)

A PMP ablation, which compares the pipeline with and without motion refinement, evaluates 500 video pairs with three raters each, totaling 1,500 evaluations. Adding PMP raises reported object-consistency preference from 12.4 to 87.6 and motion-consistency preference from 4.0 to 96.0, while reducing the reported morphological failure rate, the percentage of videos with incorrect human or animal structure, from 83.5 to 14.3.

• The qualitative dance comparison shows correction of broken motion using the ground-truth first frame and target pose; this evidence concerns those recovery conditions.
• As a separate motion denoiser, PMP improves MoMask, a human-motion generation model, using the same training dataset as MoMask.
• On HumanML3D and KIT-ML, human motion–language benchmarks, Top-1 R-Precision, a ranked text–motion matching score, increases from 0.521 to 0.544 and from 0.433 to 0.471, respectively; Top-2 and Top-3 scores also increase.

Adding PMP to MoMask improves all six reported ranked text–motion matching scores across HumanML3D and KIT-ML. This supports using the refinement network outside video regeneration, although these scores do not evaluate video appearance.

Users favor outputs with PMP for object and motion consistency, and the reported structural failure rate falls from 83.5 to 14.3. This comparison supports PMP's contribution under the tested conditions without isolating each reconstruction component.

6 Conclusion

The paper concludes that correcting explicit three-dimensional motion before video regeneration improves motion fidelity and coherence. The benchmark gains, human preferences, and PMP ablations support this conclusion within the tested datasets, base generators, and conditioning protocols.

Appendix

• The Appendix supplies architecture, dataset, ablation, limitation, and broader-impact details, and points to supplementary videos containing the main-paper examples and additional motion-control demonstrations. A Architectural details for PMP describes a single model shared across object categories, built from self-attention, cross-attention, and feedforward layers. It directly predicts corrected poses from perturbed input and is supervised with mean squared error between predicted and reference poses.
• B Dataset reports approximately 55% human, 15% animal, and 30% general-object videos in the 20K-video subset. Caption-based motion filtering and segmentation select suitable clips; human annotations combine body, hand, and facial estimates into SMPL-X, while animal annotations combine segmentation, camera estimation, and SMAL fitting. Projection-overlap and reconstruction-quality checks filter unreliable annotations.
• C Ablation Study compares full and efficient coarse-generation settings on 500 video pairs. The full model receives 52.60% overall preference, 50.27% visual-quality preference, 51.87% motion-consistency preference, and 54.27% amount-of-motion preference, showing near-even visual-quality and motion-consistency preferences with a larger preference for the full setting’s amount of motion. Additional examples illustrate how parametric meshes supply body-structure priors, text selects motion type, and motion strengths 0.3 and 0.7 produce different trajectories for the same target pose.
• C Ablation Study also tests confidence values (1, 0.5, 0), (0.8, 0.5, 0.2), and (3, 2, 1) for full motion, polygon target pose, and empty guidance. Across these settings, SSIM ranges from 0.851 to 0.873, PSNR from 30.07 to 30.10, and LPIPS from 0.210 to 0.217. These results support the paper’s use of the confidence channel as a conditioning-type marker rather than a calibrated estimate of prediction reliability.
• D Limitations identifies dependence on multiple category-specific reconstruction models, persistent difficulty with fingers and hands, and memory constraints stated as 80GB RAM for the vanilla-SVD implementation. The 128-frame demonstration uses stitched windows rather than one uninterrupted generation pass. E Statement of Broader Impact discusses possible creative and synthetic-data applications alongside risks of deepfakes and malicious content, without reporting measured downstream benefits or a tested safeguard system.

Brief Thoughts

The main engineering contribution is an explicit motion representation between two generation passes that can be corrected and reused. The PMP ablation and improvements on both SVD and Wan2.1 provide more direct evidence for this design than model-size comparisons alone.

Practical use requires accounting for annotation construction, category-specific reconstruction dependencies, and the measured 49-second runtime. The results establish better motion under the reported evaluations; general physical correctness and quantitatively validated long-video consistency remain unestablished, and the authors identify fine hand detail as a limitation. Project materials are linked at https://revision-video.github.io/.