Video Creation by Demonstration | Summary

This article explains the key points of Video Creation by Demonstration.

• 2024-12-12 (arXiv)
• Sun, Yihong, Zhou, Hao, Yuan, Liangzhe, Sun, Jennifer J., Li, Yandong, Jia, Xuhui, Adam, Hartwig, Hariharan, Bharath, Zhao, Long, Liu, Ting.
• Google DeepMind, Cornell University
• Paper
• Project Page

Read this article in Korean

Summary

• Video Creation by Demonstration addresses how to reproduce an action shown in a video within a different scene supplied as an initial image. On curated, action-compatible examples from Something-Something v2, Epic Kitchens 100, and Fractal, human raters preferred its δ-Diffusion model for action transfer in 77%, 83%, and 80% of comparisons, respectively, against WALT, an image-and-text-conditioned video generator fine-tuned on each dataset and given ground-truth action captions.
• The method extracts action latents, learned representations used to control the generated action, from a demonstration video. An appearance bottleneck, a module that limits scene-appearance information in those controls, subtracts features predictable from individual frames from video features aggregated across time, reducing the transfer of the demonstration’s objects and background into the new scene.
• Training is self-supervised: input conditions and reconstruction targets come from the videos themselves, without action labels supervising generation. The experiments support improved action transfer under curated compatible inputs, but the generator produces 16-frame, 128×128 segments, and reported failures include incorrect spatial relations, changing object identity, and extra hands.

1. Introduction

The central challenge is transferring an action concept, the intended interaction rather than its exact movement trajectory, across differences in actors, objects, viewpoints, and environments while continuing naturally from the supplied context image. Explicit controls such as text, tracked points, depth maps representing scene distances, and segmentation masks identifying image regions can omit action details or require additional effort to obtain.

• The proposed solution combines a pretrained video representation, an appearance bottleneck, and conditional future-frame prediction, which generates subsequent frames from the supplied image and action controls.
• Figure 1 illustrates transfer across everyday object interactions, first-person kitchen activities, and robotic manipulation.

The examples distinguish action transfer from copying a source scene: demonstrations and context images contain different objects or environments, while the generated sequences attempt to retain the context and reproduce the demonstrated action.

2. Related Works

The related work covers conditional video generation, methods that separate motion from appearance, and pretrained video encoders that capture actions and scene context.

2.1. Video Generation

The task belongs to image+video-to-video generation (I+V2V), where an initial image is animated using another video or signals extracted from it. Earlier approaches often use explicit controls, whereas δ-Diffusion conditions generation on learned representations extracted directly from the demonstration.

• MotionDirector adapts a video diffusion model, a generator that iteratively removes noise to produce video, separately for each demonstration at inference time; δ-Diffusion requires no inference-time optimization.
• Genie learns discrete action codes, reusable action representations selected from a finite set, from video games. This paper instead targets action concepts in real-world videos.

2.2. Modeling Motion and Action

Conventional action transfer and re-targeting methods, which apply motion from one video to another scene or subject, often assume sufficiently aligned subjects and viewpoints for low-level motion to represent the desired action. Here, the same action can require different movements: closing a top drawer and closing a bottom drawer share an action concept but differ in trajectory.

2.3. Video Foundation Models

Video foundation models, pretrained encoders intended to support multiple video-understanding tasks, provide representations containing both action and contextual information. The paper uses VideoPrism and motivates its choice through prior results on captioning, question answering, and action localization, which identifies where or when actions occur, using frozen features computed without updating the encoder.

3. Methodology

δ-Diffusion separates learning into action-latent extraction and conditional video generation. Its design addresses the risk that a generator given unrestricted demonstration features will copy the demonstration’s appearance and ignore the context image.

3.1. Task Formulation

Given a context image I and demonstration video V, the objective is to generate a video that continues from I and performs the action concepts shown in V. The task explicitly assumes that the demonstrated action is compatible with the context; generating a cutting action from an image without hands is outside the stated setting.

3.2. Method Overview

A vision encoder extracts spatiotemporal features, representations combining information across image locations and time, from the demonstration. The appearance bottleneck transforms these into action controls, which the generator combines with the context image to predict future frames, as illustrated in Figure 2.

• During training, the context frame and demonstration are taken from the same video, and the following clip also serves as the reconstruction target.
• At inference, the demonstration and context may come from different scenes.

The diagram shows how action controls are constructed: a single-frame predictor estimates the component of temporally aggregated features predictable from appearance, and subtraction leaves the residual supplied to the generator alongside the context image.

3.3. Extracting Action Latents

Let z be the video encoder’s temporally aggregated representation with shape T×N×D, where T, N, and D are the temporal, spatial, and feature dimensions. A per-frame predictor P estimates each temporal slice zₜ from features of only the corresponding frame, producing hₜ and minimizing the summed L1 reconstruction error ‖zₜ−hₜ‖₁, the sum of absolute feature differences.

• The action representation is \(\delta_v = z - [h_0, h_1, …, h_{T-1}]\), with the bracketed features concatenated over time.
• The authors describe this residual as “temporal surprisal”: information available from the frame sequence that the single-frame predictor cannot reconstruct.
• Reconstruction aligns the appearance estimate with the video-feature space, allowing subtraction to serve as the removal operator. This is a design for reducing appearance information, not a demonstrated guarantee of complete separation.

3.4. Training 𝛿-Diffusion

Training samples T+1 frames, uses the first as the context image, and uses the remaining T frames as both the demonstration and prediction target. Action-preserving augmentations, including random spatial cropping, alter the demonstration before extracting controls so that directly copying its appearance becomes harder.

• The generator is a latent diffusion model, which produces video by iteratively removing noise in a compressed representation rather than directly in pixel space.
• A tokenizer, the component that compresses frames into that representation, encodes the context and target frames. The model uses a denoising objective with the velocity-prediction target adopted from prior diffusion work (Gupta et al., 2024; Salimans and Ho, 2022).

3.5. Implementation Details

The frozen VideoPrism Base encoder has 0.1B parameters, accepts 16 frames at 288×288, and produces a 16×16×16×768 feature tensor. Its factorized architecture processes space and time separately, exposing per-frame spatial features before temporal aggregation so that the appearance predictor can reuse them.

• The predictor contains 4 Transformer encoder blocks, attention-based feature-processing layers, with 1024 hidden dimensions and 8 attention heads that compute separate feature interactions. It trains for 30k iterations with L1 loss, Adam, a stochastic optimizer, a 10⁻⁴ base learning rate, and 4.5×10⁻² weight decay.
• The generator is WALT Large with 0.3B parameters, initialized from an image+text-to-video checkpoint and fine-tuned for 500k iterations per dataset to generate 16 frames at 128×128.
• Action features replace the original text embeddings through a projection to 2048-dimensional vectors. Inference uses classifier-free guidance, a conditioning-strength mechanism combining conditional and unconditional predictions, with scale 1.25.

4. Experiment

The experiments test generation fidelity, action transfer, and continuity from the context image across human-object interactions and robotic manipulation. They combine machine evaluation of bottleneck variants with human comparisons against WALT and MotionDirector.

4.1. Datasets

The three principal datasets are Something-Something v2 (SSv2), which contains 174 labeled everyday action categories; Epic Kitchens 100, described in the paper as 55 hours of first-person kitchen videos; and Fractal, which contains around 130k robotic manipulation episodes over 700 tasks. A shared appearance predictor trains on their training splits together with Ego4D, an additional source of first-person video, while separate generators train on each principal dataset.

• The machine-evaluation protocol specifies selecting the top-10 most frequent labels in each dataset and sampling 1k video pairs per label, implying 10k pairs per dataset.
• For each pair, one video supplies the demonstration and the other supplies its first frame as context and its clip as reference. This assumes that matching labels identify transferable action concepts.

4.2. Evaluation Setup

Fréchet Video Distance (FVD) measures the difference between generated and reference video distributions, with lower values indicating closer distributions. Because it does not establish instance-level action fidelity, the paper also uses I3D, a pretrained video feature extractor, to compute embedding cosine similarity (ES), which compares feature-vector directions, and reference-video retrieval hit rates.

• Hit@100, Hit@500, and Hit@1k measure whether the corresponding reference appears among the highest-ranked 100, 500, or 1000 candidates, respectively, representing 1%, 5%, and 10% of the retrieval set.
• Five human raters compare methods on 20 examples per dataset, 60 examples overall, assessing visual quality (VQ), action transferability (AT), and context consistency (CC).
• Table 3 names the continuity rubric “Frame Continuity” and asks whether generation naturally continues from the initial image; its action rubric also includes camera movement.

4.3. Ablation Study

Table 1 compares the learned appearance bottleneck with unrestricted video features and temporal normalization, which subtracts the temporal mean from those features. The learned bottleneck achieves the highest ES and all reported retrieval hit rates on every dataset, but it does not consistently achieve the lowest FVD.

• On SSv2, unrestricted features, temporal normalization, and the proposed method obtain ES values of 0.838, 0.846, and 0.853, and Hit@100 values of 57.7%, 63.4%, and 66.9%, respectively.
• The proposed method obtains FVD 38.0 on SSv2, compared with 47.3 and 38.4; on Epic Kitchens 100, its 42.3 exceeds temporal normalization’s 41.7; on Fractal, its 44.9 exceeds unrestricted features’ 41.9.
• Figure 3 provides a corresponding visual example: the other variants import the demonstration’s keyboard appearance, whereas the learned bottleneck preserves the target scene.

The learned bottleneck improves embedding similarity and all retrieval hit rates across the three datasets. FVD varies by dataset, so the table supports improved instance-level alignment more consistently than improved distribution-level generation fidelity.

4.4. Main Results

Table 2 reports that human raters prefer δ-Diffusion over both baselines on every dataset and rubric. Against WALT, preference rates for VQ/AT/CC are 0.74/0.77/0.70 on SSv2, 0.70/0.83/0.65 on Epic Kitchens 100, and 0.82/0.80/0.74 on Fractal.

• Against MotionDirector, the corresponding VQ/AT/CC rates are 0.86/0.98/0.91, 0.96/0.98/0.98, and 0.96/1.00/0.97.
• Both baselines receive ground-truth action captions. WALT uses the context image and text; MotionDirector additionally uses the demonstration and per-instance LoRA, low-rank adaptation that updates a restricted set of model parameters to learn appearance and motion.
• Figure 4 illustrates baseline differences across all three datasets. Figure 5 demonstrates autoregressive generation, where generated frames condition the next segment, by chaining picking-and-placing, closing a drawer, and opening a drawer.
• Figures 6–9 hold the context image fixed while changing demonstrations, providing qualitative evidence that the demonstrated action influences the resulting motion.

Every reported preference rate exceeds 0.5. Action-transfer preference ranges from 0.77 to 0.83 against WALT and from 0.98 to 1.00 against MotionDirector, under the curated human-evaluation conditions.

The demonstration contains a keyboard, while the target context contains a different handheld object. Keyboard content appears in generations using unrestricted features or temporal normalization, illustrating appearance leakage that the learned bottleneck reduces in this example.

The selected examples show different failure patterns: MotionDirector can produce little useful action or inconsistent appearance, and WALT can retain the scene while missing the demonstrated action. δ-Diffusion more closely combines the intended action with the supplied context in these examples.

Three demonstrations from different scenes guide a continuing robotic sequence in one generated environment. The example illustrates action control through segment chaining, although it does not quantify long-horizon reliability.

5. Conclusion

The paper combines action controls extracted from pretrained video features with generation trained without paired cross-scene examples. Physical realism remains unreliable in complex scenes: Figure 10 illustrates incorrect spatial relations, appearance leakage, meaning transfer of the demonstration’s visual content into the output, during rapid rotation, and an extra hand associated with mismatched viewpoints.

• The authors suggest that scaling the generator could alleviate these limitations, but the paper does not experimentally establish that remedy.

Broader Impact

The authors present demonstration control as a way to make detailed video generation easier and to simulate interactions beyond those directly shown in the demonstration. They identify potential risks shared with other controlled video-generation methods: dataset bias, misleading or inappropriate content, and privacy concerns involving people without their explicit consent.

Appendix

• A Author Contributions describes the contributors’ roles in research, infrastructure, data curation, evaluation, demonstrations, and technical or strategic advice.
• B Improving Generation Quality explains how conditioning on previously generated frames extends videos beyond a single segment. It also describes self-conditioning and guidance for improving generation quality.
• C Human Evaluation Setup describes compatible demonstration-context selection, randomized side-by-side comparisons, and rating rubrics. The SSv2 wording about 20 pairs per class is not reconciled with the reported final total of 20 examples per dataset.
• D Details on Baselines documents the MotionDirector and WALT implementations. WALT is fine-tuned for 200k steps per dataset, versus 500k iterations for the proposed generator, so the reported training durations differ.
• E Qualitative Results provides links to the videos underlying the main-paper examples.
• F Controllability by Demonstration shows that different demonstration actions can guide generation from a fixed context image and provides additional video samples.
• G Failure Cases describes errors in spatial placement, loss of object permanence during rapid appearance changes, and inconsistent content when demonstration and context differ substantially.

Brief Thoughts

The clearest evidence concerns the appearance bottleneck: its controlled ablation improves every reported instance-level metric across three datasets, and the visual examples illustrate the appearance-copying failure it addresses. Interpretation of the baseline comparisons is limited by 60 curated human-evaluation examples, different reported generator fine-tuning durations, and machine metrics that do not directly verify physical correctness. The autoregressive example demonstrates that several action controls can be chained, but the paper provides no quantitative long-horizon evaluation or evidence of real-time interactive operation. Compatibility filtering is part of the task’s operating conditions, so these results do not establish reliable transfer from arbitrary demonstration-image pairs.