Mixture of Contexts for Long Video Generation | Summary

This article explains the key points of Mixture of Contexts for Long Video Generation.

• 2025-08-28 (arXiv)
• Cai, Shengqu, Yang, Ceyuan, Zhang, Lvmin, Guo, Yuwei, Xiao, Junfei, Yang, Ziyan, Xu, Yinghao, Yang, Zhenheng, Yuille, Alan, Guibas, Leonidas, et al.
• Stanford University, ByteDance Seed, Johns Hopkins University, CUHK, ByteDance
• Paper
• Project Page

Read this article in Korean

Summary

• Long-video generators must preserve character identities, scene layouts, and actions without the quadratic cost of dense attention, which compares every pair of token features representing text or video content. Compared with dense Long-Context Tuning (LCT), a video generator with a scene-level context window, Mixture of Contexts (MoC) achieves a reported 2.2× end-to-end speedup at 85% attention sparsity—the fraction of token-pair interactions omitted—on eight-shot, 64-second scenes at 480p and 12 frames per second, while improving five of six reported quality metrics.
• MoC replaces dense attention in a diffusion transformer (DiT), a Transformer backbone that generates content through denoising, with retrieval over chunks aligned to frames, shots, and captions; a shot is a continuous video segment between cuts. Each query, the feature seeking relevant context, selects the top-k highest-scoring chunks by similarity to their mean-pooled keys, the averages of features used to match context. Mandatory access to caption tokens and the current shot preserves prompt conditioning and local detail, while causal routing restricts optional retrieval to earlier chunks.
• The main comparison reduces reported attention floating-point operations (FLOPs), a measure of arithmetic work, from 1.7 × 10^13 to 2.3 × 10^12. Dynamic Degree, the benchmark measure of motion extent, rises from 0.4583 to 0.5625, but Image Quality, a measure of frame-level technical quality, falls from 0.5140 to 0.5003.
• The appendix examines routing choices through ablations, experiments that vary individual design components, and tests another video-generation backbone, Wan-2.1-1.3B. The main long-video quality comparison remains confined to the LCT setup; short single-shot videos require fewer FLOPs with MoC yet run more slowly end to end because pooling and token-gathering overhead outweigh the attention savings.

1 Introduction

The Introduction frames long-video generation as selective recall: a model must recover relevant earlier events while preserving identity and narrative coherence. Temporal redundancy makes dense attention expensive, but compressing history can discard details, and fixed sparse patterns cannot adapt to which historical events matter for a particular query.

• MoC retrieves selected original context tokens rather than replacing the entire history with a fixed-size summary.
• The method combines learned retrieval, content-aligned chunking, mandatory caption and local connections, and progressive sparsification, which gradually increases the fraction of omitted interactions during training. The reported speedup concerns approximately 180k-token scenes.

2 Related Work

Related Work groups prior approaches into Long Video Generation, Sparse Attention for Video Generation, and Context Learning in Visual Generation. Recurrent states carry information between generation steps, hierarchical keyframes establish coarse video structure, and compressed history summarizes earlier content; sparse attention reduces token interactions, while external visual memories retrieve stored frames or views.

• LCT is the closest baseline because it expands a pretrained single-shot model to a joint window of up to eight shots while retaining dense attention.
• MoC learns retrieval within attention layers. Some external-memory approaches instead select historical views using camera-view overlap, and visual context-learning methods condition generation on reference images.
• The section discusses several sparse-attention methods, but the main long-video experiment compares MoC only with LCT.

3 Method

The Method replaces the backbone’s dense attention with a MoC layer that selects relevant context, partitions tokens along frame, shot, and caption boundaries, and constrains optional retrieval by temporal order. Figure 1 shows how dynamically retrieved chunks are combined with mandatory text and intra-shot connections before attention is computed.

The diagram separates optional retrieval from mandatory connections: a query scores pooled chunks, retrieves selected video tokens, and retains caption and current-shot access. Dashed routes identify connections removed by causal masking, showing how sparse context selection preserves local attention while restricting optional retrieval to earlier context.

3.1 Mixture of Contexts

For each query, MoC scores candidate chunks using the dot product between the query feature and each chunk’s mean-pooled key feature, then performs ordinary attention over the tokens in the selected chunks. The router adds no dedicated parameters: its inputs come from the model’s existing query and key projections, which change during fine-tuning.

• With Q, K, and V denoting query, key, and value features—the values supply the information combined by attention—and d the feature dimension, dense attention is Attn(Q,K,V) = Softmax(QKᵀ/√d) · V. Softmax normalizes the matching scores into attention weights.
• For routed indices Ω(q_i), the positions selected for query q_i, MoC computes Attn(q_i,K,V) = Softmax(q_i Kᵀ_{Ω(q_i)}/√d) · V_{Ω(q_i)}.
• The top-k choice is discrete and non-differentiable. The paper explains adaptation through gradients in attention over selected keys and values, rather than through a differentiable top-k selection.

Context Drop-off randomly removes selected chunks to train robustness to missing context, while Context Drop-in adds random chunks to expose underused routes to learning signals. Routing operates independently for every attention head, a parallel feature-specific attention computation, in every layer.

• Drop-off samples p_drop from Uniform(0,p_max), a uniform distribution over the stated interval, and removes floor(p_drop · k) selected chunks.
• Drop-in samples the number of added chunks as \(m\sim\operatorname{Poisson}(\lambda)\), a count distribution controlled by λ.
• Independent routing across heads and layers broadens the union of accessed context even when each individual route remains sparse.

3.2 Attention Chunking and Routing

Attention Chunking and Routing aligns chunks with video structure because adjacent entries in a flattened text-video stream can cross spatial, temporal, or shot boundaries. Every visual query retains access to all text tokens and its own shot, leaving the optional retrieval budget available for context beyond those mandatory connections.

• Text tokens serve as an attention sink, a persistent set of tokens available to every query, and typically constitute less than 1% of the sequence.
• Intra-shot links preserve access to object trajectories, lighting continuity, and other local cues during fine-tuning.
• Content alignment keeps pooled chunk descriptors from combining unrelated material across a cut.

Without causal routing, two chunks can select one another and form an isolated feedback pair that limits access to earlier identity information. MoC masks optional edges \((i\to j)\quad\text{whenever }j\ge i\) before top-k selection, producing a directed acyclic graph (DAG), a graph without directed cycles, for the routed connections.

• Figure 2 shows shots 9 and 11 preserving the same newly generated woman while failing to preserve the woman’s identity from the earlier café scene.
• The temporal restriction applies to optional routing; mandatory intra-shot attention remains part of the method.

The riverbank shots agree with each other on the woman's appearance but disagree with her earlier café identity. Strong reciprocal routing between shots 9 and 11 in the heatmap illustrates how local agreement can coexist with failed historical recall when optional routes form a feedback pair.

3.3 Computation Efficiency

Computation Efficiency implements variable-length chunk routing entirely on the graphics processing unit (GPU), using torch.bucketize and prefix-sum boundary tables, on-the-fly segment_reduce pooling, and token gathering in head-major order, which groups tokens by attention head. Selected query-key groups are packed into one variable-length Flash-Attention call, an efficient attention kernel that accommodates unequal chunk sizes.

• For sequence length L, number of chunks C, retrieved chunk count k, average selected chunk length m̄, and head dimension d, Equation (4) gives FLOPs_MoC ≈ Ld + 2LCd + 4Lkm̄d.
• Equation (5) gives FLOPs_dense = 4L²d. The routing term depends on the number of candidate chunks, so these expressions do not establish unconditional linear scaling.
• The illustrative calculation uses approximately 180k tokens, m̄ ≈ 1024, k = 5, C = 36, and d = 128, reporting 2.32 × 10^12 versus 1.66 × 10^13 FLOPs. The stated MoC estimate does not follow from those inputs in Equation (4), and Equation (6) does not follow algebraically from Equations (4)–(5); the reported arithmetic savings therefore need clarification and remain distinct from the measured 2.2× end-to-end speedup.

4 Experiments

The main Experiments initialize MoC from pretrained LCT, whose backbone is a 3B-parameter multimodal diffusion transformer (MMDiT), a diffusion Transformer processing text and visual tokens. The dense LCT baseline and MoC are tested on eight shots per scene, each lasting 8 seconds at 480p and 12 frames per second, for approximately 180k tokens over 64 seconds.

• The paper reports fine-tuning under the same training scheme as LCT after replacing its attention module.
• VBench, a benchmark suite for generated video, measures subject and background consistency, motion smoothness, motion extent, aesthetic quality, and image quality. These assess preservation of subjects and backgrounds, continuity of movement, amount of motion, visual appeal, and technical frame quality, respectively.
• The main results table does not provide an evaluation sample count, confidence intervals, or run-to-run variation.

Table 1 reports 85% sparsity, a reduction from 1.7 × 10^13 to 2.3 × 10^12 attention FLOPs, and improvements in five of six VBench metrics. The text reports a 2.2× end-to-end speedup; Dynamic Degree increases from 0.4583 to 0.5625, while Image Quality decreases from 0.5140 to 0.5003.

• Subject Consistency: 0.9378 → 0.9421; Background Consistency: 0.9526 → 0.9535.
• Motion Smoothness: 0.9859 → 0.9920; Dynamic Degree: 0.4583 → 0.5625.
• Aesthetic Quality: 0.5436 → 0.5454; Image Quality: 0.5140 → 0.5003.

At 85% sparsity, MoC reduces reported attention FLOPs from 1.7 × 10^13 to 2.3 × 10^12 and improves subject consistency, background consistency, motion smoothness, motion extent, and aesthetic quality. Image Quality decreases from 0.5140 to 0.5003, so the quality advantage does not extend to every metric.

Figure 3 compares dense LCT and MoC on café and device-interaction sequences, showing plausible shot-level compositions under sparse attention. Figure 4 provides selected examples of persistent landmarks, drawing-to-building correspondence, neon signage, computer details, and distinct people across cuts; these examples illustrate coherence without establishing its frequency across an evaluation set.

Paired dense and sparse outputs follow the same café and device-interaction shot descriptions while producing different visual realizations. The selected frames show plausible cross-shot characters and settings, but cannot establish equivalent motion or failure rates across complete videos.

Highlighted landmarks, signs, computer components, and people recur across intervening shots and viewpoint changes. The examples illustrate background geometry, object details, drawing-to-building correspondence, and multiple identities, extending the qualitative evidence beyond a single recurring subject.

Supplementary experiments examine implementation scaling, frozen-weight sparsification, short-video generation, routing choices, and transfer to a second backbone. Figures 5–7 and Tables 2–5 provide evidence under distinct conditions, summarized in the Appendix.

• Figure 5 compares attention computation and latency through 24 shots, with 8 seconds, 12 frames per second, and roughly 23k tokens per shot. It is an implementation benchmark rather than a long-duration generation-quality evaluation.
• Figure 6 shows frozen-weight MoC at >75% sparsity. Figure 7 and Table 2 concern 8-second, 320×192 single-shot videos with roughly 6,300 tokens: six quality metrics improve at 83% sparsity, but end-to-end generation is slower.
• Table 3 shows reduced motion under very small chunks or small k, Table 4 demonstrates the importance of mandatory intra-shot links, and Table 5 reports five metric improvements and one motion-smoothness decrease on Wan-2.1-1.3B at 81% sparsity.

The frozen-weight examples retain recognizable people and settings across café, restaurant, and rainy-action sequences despite >75% sparsity. They illustrate a usable retrieval signal in existing features, while leaving average quality and the benefit of fine-tuning unquantified.

The examples cover rotating televisions, travel through a colorful neighborhood, and approaching mammoths. Both models produce recognizable prompt-related content, providing a qualitative complement to the short-video metrics without measuring the slower runtime reported for MoC in this setting.

MoC improves all six reported metrics at 83% sparsity and reduces attention FLOPs from 1.9 × 10^10 to 4.1 × 10^9. The table reports arithmetic savings rather than latency; the accompanying text states that end-to-end generation is slower for these short sequences.

The chunk-size and k sweeps expose a tradeoff between pruning and motion. For chunk size 256, k = 1 produces Subject Consistency 0.9994 but Dynamic Degree 0.1313, whereas k = 3 produces 0.9916 and 0.4612; high consistency alone can therefore favor outputs with little movement.

Rows without mandatory intra-shot access have near-zero motion extent and very low image quality, even when text access is forced and four additional retrieval slots compensate for the missing local connection. Adding intra-shot attention substantially improves these outcomes; cross-modal connections and context perturbations change the remaining metrics without improving every one.

On Wan-2.1-1.3B, MoC reaches 81% sparsity and improves five metrics, with Dynamic Degree rising from 0.4219 to 0.6250. Motion Smoothness decreases from 0.9869 to 0.9833, retaining a measurable tradeoff on this second backbone.

5 Conclusion

The Conclusion interprets learned sparse routing as an internal memory-retrieval mechanism that makes minute-scale context practical without external geometric retrieval rules. Limitation and Future Work acknowledges that the main model was trained and tested under the LCT setup and that savings on still longer sequences require further study.

• The implementation uses general-purpose variable-length attention and framework-level gathers, leaving a gap between arithmetic savings and realized runtime savings.
• Proposed improvements include chunk-aware sparse kernels, fused routing and attention, more efficient key/value layouts, and quantization, which reduces numerical precision. These remain future directions.

Appendix

• A Memory Complexity Analysis. Mean-pooled descriptors require \(O(Nd)\) storage for N chunks, while routing logits, the scores used to select chunks, require \(O(LN)=O(L^2/C)\) when C denotes chunk size and \(N\approx L/C\); O expresses asymptotic growth. The appendix reports routing overhead below 0.1% of total GPU memory using torch.segment_reduce and a custom torch.autograd.Function, which defines the attention operation’s gradient computation, but the routing-logit expression remains quadratic in sequence length for fixed chunk size.
• B MoC Implementation Benchmark. Figure 5 shows near-linear reported computation and measured latency over the tested shot counts compared with full attention implemented using FlashAttention 2 [8, 9]. The paper attributes this behavior to on-the-fly pooling, head-major token organization for efficient gathers, and packing variable-length attention groups into a single kernel call.
• C Dataset Details. Public narrative videos are segmented into scenes using a scene-boundary detector and then into shots using PySceneDetect, a shot-cut detection tool, with Gemini-1.5, a multimodal annotation model, generating global and shot-level captions. Approximately 500K authentic multi-shot scenes average about 5 shots each, yielding approximately 2.5M shot clips; event-based segmentation of long single-shot videos contributes roughly another 1M scene-level samples. Global captions use “[Character] [Environment] [Story]” and “Character [ID]: [Description]”, shot captions refer to global character IDs, and genuine cuts receive “[SHOT CUT]”.
• D Zero-shot Experiment. The authors insert MoC into a pretrained dense model with all weights frozen, a zero-shot application that requires no additional training, and show selected outputs at >75% sparsity. Figure 6 provides qualitative evidence that pooled keys can retrieve useful context before fine-tuning, but the section supplies no quantitative comparison isolating the effect of learning the routing representations.
• E Single-shot Short Video Generation. On 8-second videos at 320×192 and 12 frames per second, MoC improves all six reported VBench metrics [21] against the native 3B MMDiT [10] baseline at 83% sparsity, reducing FLOPs from 1.9 × 10^10 to 4.1 × 10^9. Dynamic Degree increases from 0.6875 to 0.7500 and Image Quality from 0.6345 to 0.6396, but pooling and gathering overhead produce a slower end-to-end pipeline.
• F Training Details. Single-shot training uses images and videos, chunk size 256, top-k = 3, mandatory intra-chunk and text connections, and no causal routing because closed-loop behavior was not observed. Multi-shot training uses images, single-shot videos, and multi-shot videos, progressively reducing chunk size through 10240, 5120, 2560, and 1280 with top-k = 5 and mandatory intra-shot, local-prompt, and global-prompt connections. Both use a learning rate of 9e−5; single-shot training runs for 10k iterations and multi-shot training for 20k.
• G Ablation Study. Ablations use 16 H100 GPUs, a learning rate of 2e−5, 30k iterations for single-shot experiments, and 10k for multi-shot experiments. Table 3 shows that tiny chunks or small k can suppress motion: at chunk size 256, k = 1 yields Dynamic Degree 0.1313 versus 0.4612 for k = 3, with each query additionally retaining its own chunk. In Table 4, using chunk size 5120 and k = 5, omitting both forced links yields Dynamic Degree 0.0000 and Image Quality 0.1552, whereas enabling both links and Context Drop In & Out yields 0.5469 and 0.5061; configurations without forced intra-shot links receive four additional retrieval slots.
• H Wan-2.1-1.3B Experiment. MoC is applied only to self-attention in Wan-2.1-1.3B, comparing sparse and dense fine-tuning on the Vchitect [11, 33] dataset at 480p with chunk size 1560. Each setting runs on 32 GPUs for 1 day, or 2000 iterations; at 81% sparsity, MoC improves five of six metrics, including Dynamic Degree from 0.4219 to 0.6250, while Motion Smoothness falls from 0.9869 to 0.9833. This supplies evidence on another backbone for single-shot generation, with no corresponding long-video evaluation reported here.
• I Outer Loop Context Routing. A coarse router first selects top-M large context segments using similarity between mean-pooled features of the current query block and candidate segments, then passes those segments and mandatory context to inner MoC routing. The authors report generating 2–3 times more shots under autoregressive sampling, which generates successive segments conditioned on earlier ones, but provide no associated quality table or detailed scaling protocol. Claims of stable behavior for arbitrarily long inputs therefore extend beyond the quantitative evidence presented.
• J Social Impact. The authors describe potential uses in animation, documentary production, education, and simulation, alongside risks of misinformation and non-consensual media synthesis. They advocate gated release, watermarking, and prompt filtering; these are recommendations rather than evaluated safeguards.
• K The Use of Large Language Models (LLMs). The authors state that large language models, models used here for text revision, were used only to rephrase sentences and correct grammar, and declare compliance with the ICLR Author Guide. This statement does not establish a publication venue. The supplied paper is arXiv:2508.21058v3 and lists https://primecai.github.io/moc/ as its project page.

Brief Thoughts

The combination of adaptive context selection with mandatory local and textual access is supported by the design experiments. Table 4 shows that local connections stabilize the tested training setup, and Table 3 shows why consistency scores must be read alongside motion extent: very sparse settings can preserve appearance while generating little movement.

The evidence supports an efficient attention replacement for the tested minute-scale setup, with a runtime advantage that depends on sequence length. Broader memory claims need controlled recall measurements and quantitative evaluation beyond the training horizon; the qualitative examples and outer-loop extension do not provide those measurements, and the illustrative FLOPs calculation needs reconciliation with the stated formulas.