StyleCrafter: Enhancing Stylized Text-to-Video Generation with Style Adapter | Summary

This article explains the key points of StyleCrafter: Enhancing Stylized Text-to-Video Generation with Style Adapter.

• 2023-12-01 (arXiv), ACM Transactions on Graphics, Vol. 43, No. 6, December 2024
• Liu, Gongye, Xia, Menghan, Zhang, Yong, Chen, Haoxin, Xing, Jinbo, Wang, Yibo, Wang, Xintao, Yang, Yujiu, Shan, Ying.
• Tsinghua University, China, Tencent AI Lab, China, The Chinese University of Hong Kong, China
• Paper
• Github

Read this article in Korean

Summary

• StyleCrafter addresses the difficulty of generating videos that follow a text description while matching the artistic style of a reference image. In a 15-user study of single-reference generation, its videos received 68.9% of style preferences versus 23.1% for Gen-2* and 8.0% for VideoCrafter*, both supplied with text descriptions of the reference style.
• The method trains a style adapter, a module that injects reference-image features into a pretrained generator, on artistic images; it then adapts the video model’s temporal blocks using images and photorealistic videos. Separate attention paths and context-dependent scaling balance the reference style against the prompt’s content.
• On 240 single-reference video pairs, StyleCrafter achieves 0.2726 CLIP-Text, a text-image similarity score, and 0.4531 CLIP-Style, an image-reference similarity score; VideoComposer scores 0.0468 and 0.7306, respectively, while often copying reference content. Results remain limited by the base video model and by references that inadequately express a target style.

1 Introduction

The paper identifies two obstacles: extracting style without copying reference-image content and adapting a video generator when stylized video training data are scarce. It proposes image-based adapter training followed by temporal adaptation, with augmentation and adaptive fusion to encourage separation of content and style.

2 Related Works

The related work spans text-to-video synthesis, stylized image generation, and stylized video generation; the paper positions its adapter at the intersection of these areas.

2.1 Text to Video Synthesis

Prior text-to-video systems use autoregressive transformers or diffusion models, while another family extends pretrained text-to-image models with temporal blocks. VideoCrafter follows the latter pattern and provides the base model used here.

2.2 Stylized Image Generation

Style transfer combines one image’s appearance with another’s content, while example-guided image generators learn or extract a style from reference images. The authors note that IP-Adapter can produce image variants but may retain reference content when asked to generate different content.

2.3 Stylized Video Generation

Video style transfer often relies on a source video and frame-consistency constraints. VideoComposer accepts a style reference but can copy its content, whereas AnimateDiff combines a personalized image model with motion modules and ordinarily requires style-specific preparation.

3 Method

StyleCrafter conditions a pretrained video diffusion model on a text prompt for content and an image for style. Its two stages learn reference-based style modulation from images and then adapt temporal processing on a mixture of artistic images and realistic videos.

The diagram shows how reference-image features enter a separate style attention path and how text and style jointly determine the layer-wise fusion scales.

3.1 Reference-Based Style Modulation

Training uses WikiArt and a subset of Laion-Aesthetics with scores above 6.5; BLIP-2, an image-language model, regenerates captions and regular expressions remove some style phrases. The target is centrally cropped after resizing the shorter side to 512, while the style reference is a random 512 × 512 crop after resizing the shorter side to 800, reducing content overlap.

A pretrained CLIP image encoder supplies a global token and 256 local tokens to a trainable Query Transformer (Q-Former), which extracts style features. Dual cross-attention, separate attention operations for text and style, combines those features with 16 layer-wise scale factors predicted from both conditions.

3.2 Temporal Adaptation to Stylized Features

The image-trained adapter alone can produce jitter and artifacts in videos, so the second stage freezes other modules and fine-tunes temporal blocks on stylized images and photorealistic videos. Classifier-free guidance, a way to adjust conditioning strength during generation, uses separate text and style weights: 15.0 and 7.5 for video, and 7.5 and 5.0 for images.

4 Experimental Results

The experiments assess image and video generation, compare single- and multi-reference methods, and ablate the adapter and training choices. They combine automatic similarity and temporal-consistency measures with a user study.

4.1 Experimental settings

The first stage runs for 40k steps with batch size 32 per GPU; the second runs for 20k steps with video batch size 1, image batch size 16, and 20% image batches. Training uses 8 A100 GPUs and takes within 3 days.

The two backbone groups show consistent gains in style metrics, while DreamBooth retains a slightly higher text score in the Stable Diffusion 2.1 group.

The test set contains 400 image pairs and 300 video pairs, including 240 single-reference and 60 multi-reference video pairs. It uses 20 image prompts, 12 video prompts, 20 single references, and eight multi-reference sets of 5 to 7 images; the user study covers 15 single-reference and 10 multi-reference pairs and collects 1,125 votes from 15 users.

VideoComposer’s high style score accompanies very low text alignment, illustrating why reference similarity alone cannot establish successful style transfer.

4.2 Style-Guided Text-to-Image Generation

On 400 image pairs, the Stable Diffusion 2.1-based StyleCrafter obtains CLIP-Style 0.4836 and DINO-Style, a similarity score based on DINO visual features, 0.3652, compared with 0.4183 and 0.2890 for SD*. Its CLIP-Text score is 0.3028, slightly below DreamBooth’s 0.3047; the SDXL-based version leads its comparison group on all three reported scores.

StyleCrafter leads style preference in both settings, while Gen-2* receives the highest single-reference temporal preference.

4.3 Style-Guided Text-to-Video Generation

For single-reference video, StyleCrafter scores 0.2726 on CLIP-Text and 0.9892 on CLIP-Temp, a similarity measure between adjacent frames, versus VideoCrafter* at 0.2209 and 0.9757. VideoComposer’s higher CLIP-Style and lower warping error, a measure of frame differences after motion-based alignment, are confounded by reference copying and often static output.

In the user study, StyleCrafter receives 68.9% of single-reference style preferences, but Gen-2* leads temporal preference at 51.1% versus 44.4%. For multi-reference comparisons, StyleCrafter receives 90.0% of style and 80.7% of temporal preferences versus AnimateDiff’s 10.0% and 19.3%.

Temporal adaptation improves both reported temporal measures over direct application of the image-trained adapter; joint training has higher text similarity but weaker style similarity.

4.4 Ablation Study

Removing augmentation lowers image CLIP-Style from 0.4836 to 0.4005; removing dual cross-attention raises style similarity to 0.7332 while reducing CLIP-Text to 0.0983, consistent with reference-content copying. Without adaptive fusion, CLIP-Text falls from 0.3028 to 0.2807.

Two-stage training reaches CLIP-Temp 0.9892 and warping error 18.73 × 10⁻³, compared with 0.9612 and 47.88 × 10⁻³ without temporal adaptation. Joint training obtains higher CLIP-Text, 0.3138 versus 0.2726, but lower CLIP-Style, 0.2226 versus 0.4531.

5 Conclusion and Limitations

The authors report failures when a reference poorly represents the intended style or the style is highly unfamiliar. The supplementary material also identifies inherited constraints on resolution, video length, temporal consistency and human faces, as well as imperfect reproduction of complex aesthetics such as the blank space in Chinese ink painting.

Appendix

• A Implementation Details and A.1 Comparison methods specify baseline training: DreamBooth uses 500 steps for single-reference and 1,500 for multi-reference styles, while InST uses 1,000 steps; IP-Adapter’s scale is set to 0.5. A.2 Testing Datasets describes GPT-4 prompt generation and manual filtering across human, animal, object and landscape categories.
• B Evaluation Details and B.1 Evaluation Metrics define CLIP cosine similarities and optical-flow-based warping error. B.2 User Study describes randomized option order, masked watermarks and separate questions for text alignment, style conformity and temporal quality; VideoComposer was excluded because it often copied the reference.
• C Extended Comparison includes C.1 Multi-reference Stylized Image Generation, where StyleCrafter scores 0.5333 on style versus CustomDiffusion’s 0.4441; C.2 Comparison with StyleDrop uses an unofficial implementation and reports 0.4836 versus 0.3962. C.3 Comparison with Style Transfer Methods reports a video style score of 0.4531 versus 0.2858 for the MCCNet-based comparison.
• D Extended Ablation Study reports style scores of 0.2843, 0.4149 and 0.4836 for a multilayer perceptron (MLP), Transformer and Q-Former extractor, respectively, with the opposite ordering for text scores. E Application Extension combines independently trained style and depth adapters at inference; F More Results supplies additional examples, and G Limitations documents facial artifacts and incomplete complex-style reproduction.

Brief Thoughts

The ablations support separating style and content, especially because a high style-similarity score can reward copying the reference. The curated prompt and reference set and 15-person study constrain how broadly the comparative results can be generalized; Gen-2* also leads the single-reference temporal preference measure.