Paper2Video: Automatic Video Generation from Scientific Papers | Summary

This article explains the key points of Paper2Video: Automatic Video Generation from Scientific Papers.

• 2025-10-06 (arXiv)
• Zhu, Zeyu, Lin, Kevin Qinghong, Shou, Mike Zheng.
• Show Lab, National University of Singapore
• Paper
• Github
• Project Page

Read this article in Korean

Summary

• Creating a research presentation video requires converting a paper into coordinated slides, narration, subtitles, presenter animation, and cursor cues. On Paper2Video, a benchmark of 101 paper–presentation pairs, the PaperTalker generation system achieves 0.842 accuracy on paper-derived detail questions versus 0.738 for author-recorded videos when a video-interpreting model answers the questions, although human preference ratings favor the author-recorded videos, 4.6 versus 3.8.
• PaperTalker coordinates four specialized builders and uses Tree Search Visual Choice, a procedure that renders alternative slide layouts and selects a candidate through visual comparison. It combines speech synthesized from an author’s voice sample, an animated presenter, and a cursor aligned with narration; parallel generation across slides reduces runtime from 287.2 to 48.1 minutes relative to the serial PaperTalker variant.
• The benchmark evaluates similarity to author-created assets, video preference, paper-derived quiz accuracy, and speaker–work association. Most evaluations use models as audience proxies, so their scores do not directly establish human comprehension or lasting research impact; the human study covers ten papers and ten participants.

1 Introduction

Producing a 2 to 10 minute academic presentation can require several hours of slide design, subtitle writing, recording, and editing. The paper addresses two coupled problems: generating aligned presentation components from a long document containing text, figures, and tables, and evaluating whether the resulting video communicates the research effectively. Paper2Video supplies paired reference material and evaluation measures, while PaperTalker supplies the generation pipeline; the project URL is https://github.com/showlab/Paper2Video.

The diagram separates generation from evaluation. PaperTalker consumes a paper, portrait, and voice sample, while Paper2Video assesses the resulting presentation using author-created reference material.

2 Related Works

The related work connects academic presentation generation with natural video synthesis and AI-assisted research workflows. The comparison distinguishes visual-media generation from systems that coordinate slides, subtitles, speech, a presenter, and a cursor.

2.1 Video Generation

The paper argues that existing video diffusion models, which generate video through iterative denoising, struggle with several-minute outputs, multiple shots, multiple conditioning images, and aligned audio. MovieAgent coordinates planning roles for long-form movie generation, whereas PresentAgent combines PPTAgent, a slide-generation system, with text-to-speech, the conversion of written narration into audio. The authors identify missing presenter personalization and academic slide structure as limitations of PresentAgent.

The comparison identifies coordinated presentation components as the task’s distinguishing requirement. Paper2Video is the only listed entry marked as including subtitles, slides, a cursor, a speaker face, and a speaker voice; the marks describe component coverage rather than measured quality.

2.2 AI for Research

AI for Research (AI4Research), the use of AI to support scholarly workflows, includes scientific writing, literature surveying, paper replication, idea proposal, and visual design. The paper positions presentation video generation as an extension of slide and poster automation that requires coordinated outputs across multiple media. Paper2Agent is discussed as a complementary approach that turns papers into interactive research assistants.

3 Paper2Video Benchmark

Paper2Video combines a task definition, curated paper–presentation pairs, and evaluation measures for research communication. Its references include author-recorded videos and speaker identity information, supporting assessment of content delivery and presentation personalization.

3.1 Task Definition

The task takes a research paper and author identity information as inputs and produces a presentation that faithfully communicates the paper’s contributions. Desired components are organized slides, synchronized subtitles and speech, a natural presenter, and a cursor that directs attention. The main challenges are understanding long documents across text and images, coordinating generation stages, preserving presenter identity and lip synchronization, and grounding narration, meaning aligning its sentences with visual locations and time intervals.

3.2 Data Curation

The dataset contains 101 peer-reviewed conference papers from the preceding three years: 41 in machine learning, 40 in computer vision, and 20 in natural language processing. Each example includes the full LaTeX project, the typesetting source for the paper, an author-recorded presentation, and speaker information such as a portrait and voice sample; original slide PDFs are available for 40% of examples. Presentations contain 4–28 slides and last 2–14 minutes, with averages of 16 slides and 6 minutes 15 seconds.

• The authors select papers with relatively complete metadata from sources including YouTube and SlidesLive, then supplement missing presenter images from author websites. This curation does not establish representativeness beyond the selected AI conference papers.
• The source reports averages of 13.3K words (3.3K tokens), 44.7 figures, and 28.7 pages per paper. Tokens are the text units counted for model input; the reported relationship between word and token counts is unexplained.

The distributions show varied presentation lengths and slide counts. Average and median slide counts are 16.0 and 14.0, while average and median durations are 6:15 and 5:11; both means exceed their corresponding medians.

3.3 Evaluation Metrics

The four evaluation measures target reference similarity, overall preference, information coverage, and speaker–work association. A vision-language model (VLM), which processes images and text, evaluates slide and subtitle similarity, while a video language model (VideoLLM), which interprets video, acts as the audience proxy for preference and quiz evaluations. These measures assess specified presentation properties without directly measuring human learning or scholarly impact.

• Meta Similarity compares generated slides and subtitles with author-created references and separately computes cosine similarity, a measure of vector alignment, between speech embeddings, vector representations of voice characteristics, from 10-second audio segments. The main text describes a five-point content scale, while the appendix prompt specifies integer scores from 0 to 5.
• PresentArena measures preference between generated and human-made videos. It queries each pair twice with reversed input order and averages the judgments to reduce position bias.
• PresentQuiz measures accuracy on paper-derived questions with four answer options, covering fine-grained details and higher-level understanding.
• IP Memory measures speaker–work association by asking the model to identify a relevant question from presentation clips and a speaker image. The paper acknowledges that simulating visibility and impact remains an open problem.

The visual distinguishes comparison with the source paper from comparison with human-made presentations. PresentQuiz tests paper-derived information, Meta Similarity compares intermediate assets, and PresentArena compares video preference; IP Memory is described separately in the text.

4 PaperTalker Agent

PaperTalker separates generation into Slide, Subtitle, Cursor, and Talker builders. The Slide builder produces and repairs LaTeX slides; the Subtitle builder generates narration and visual-focus descriptions; the Cursor builder converts those descriptions into screen locations and timing; and the Talker builder synthesizes personalized speech and presenter video from a portrait and voice sample. The intermediate outputs support compilation checks, visual refinement, and alignment.

The pipeline shows how compilation and layout checks precede script generation, cursor alignment, and personalized presenter synthesis. Independent presenter branches explain the parallelization strategy; the O(1) annotation does not establish constant runtime under a fixed hardware budget.

4.1 Slide Builder

The Slide builder generates Beamer code, a LaTeX format for presentation slides, directly from the paper’s LaTeX project. Compilation errors drive focused code repair, while warnings identify slides that may need layout refinement. Tree Search Visual Choice constructs rule-based variants of figure scale or font size, renders them, and asks a VLM to select a layout that avoids overflow, where content extends outside the slide or becomes clipped, while retaining visible content.

• Beamer provides automatic typesetting, a compact representation, and academic presentation styles. These are the authors’ reasons for using it instead of template editing in an XML-based format, a structured markup representation.
• For text-only slides, the procedure sweeps font size. For slides with figures, the main text gives example scaling factors of 1.25, 0.75, 0.5, and 0.25 before reducing font size; Figure 5 instead displays 1.00, 0.75, 0.50, and 0.25.
• The method separates discrete layout variation from semantic generation, avoiding reliance on a language model to make small numerical edits directly.

The example turns numerical layout editing into a visual choice among rendered alternatives. Its figure-scale candidates are 1.00, 0.75, 0.50, and 0.25; the judge selects a candidate that keeps content visible without clipping.

4.2 Subtitle Builder

The Subtitle builder rasterizes each completed slide, converting it into an image, and uses a VLM to generate sentence-level subtitles with corresponding visual-focus prompts. Each prompt describes the slide region relevant to its sentence and connects narration to cursor placement.

4.3 Talker Builder

The Talker builder uses F5-TTS, a text-to-speech model, to produce audio for each slide from its subtitles and the author’s voice sample. Hallo2 generates head-only presenter animation, while FantasyTalking supports upper-body articulation. Presenter clips are generated independently for each slide and processed in parallel, based on the assumption that slide transitions do not require continuous presenter motion.

4.4 Cursor Builder

The Cursor builder assumes that the cursor remains stationary within each sentence and moves only between sentences. UI-TARS, a model for locating interface targets from screenshots, maps visual-focus prompts to slide coordinates; WhisperX, a speech transcription and alignment system, supplies word-level timestamps that determine each cursor segment’s start and end. This simplifies cursor generation but does not model continuous movement within a sentence.

5 Experiments

The experiments compare end-to-end video generators, which produce video directly from a paper-derived prompt, presentation-generation frameworks, and PaperTalker variants. They assess reference similarity, preference, information coverage, speaker–work association, human ratings, generation efficiency, and component ablations, comparisons that remove a component to test its contribution.

5.1 Baseline and Settings

The baselines include Wan2.2 and Veo3 for direct video generation, and PresentAgent and PPTAgent for composed presentation generation or slide evaluation. GPT-4.1 serves as the VLM and Gemini-2.5-Flash as the VideoLLM; inference uses eight NVIDIA RTX A6000 GPUs. Output durations differ substantially: Wan2.2 and Veo3 average 4.00 and 8.00 seconds, respectively, compared with 234.36 seconds for PaperTalker, so the comparisons also reflect different duration capabilities.

5.2 Main Results

PaperTalker achieves the strongest reported automated scores among the evaluated generated-video systems, but author-recorded presentations receive higher preference scores. Its detail and understanding quiz accuracies are 0.842 and 0.951, compared with 0.738 and 0.908 for HumanMade, the author-recorded reference, despite shorter average duration, 234.36 versus 375.15 seconds. Its PresentArena win rate against human-made videos is 17.0%, and the human study gives it 3.8 versus 4.6 for HumanMade.

• PaperTalker’s speech and content similarities are 0.646 and 1.97, compared with 0.045 and 1.47 for PresentAgent using GPT-4.1. These lead the generated systems in the table, although content similarity remains below the human reference value of 5.00.
• Removing both presenter and cursor lowers PresentArena from 17.0% to 15.2%, detail quiz accuracy from 0.842 to 0.835, understanding accuracy from 0.951 to 0.949, and IP Memory from 50.0% to 37.5%. This combined ablation does not isolate either component’s contribution.
• Ten participants evaluate videos from ten randomly sampled papers on a 1–5 preference scale. Reported scores are 1 for Wan2.2, 2.2 for Veo3, 4.6 for HumanMade, 3.8 for PaperTalker, and 2.8 for PresentAgent; uncertainty estimates are not reported.
• PaperTalker uses 62K tokens versus 241K for PresentAgent. Parallelization reduces PaperTalker runtime from 287.2 to 48.1 minutes, approximately 6×, but full PaperTalker still takes longer than PresentAgent’s 39.5 minutes.
• Table 3 reports costs of $0.001 for PaperTalker and $0.003 for PresentAgent. It does not specify an accounting basis sufficient to interpret these figures as complete deployment costs, including GPU computation.

PaperTalker leads the generated systems in the reported automated measures and exceeds HumanMade on both quiz categories. Its 17.0% PresentArena win rate and 1.97 content similarity show that higher quiz accuracy does not establish overall equivalence to author-recorded presentations; unequal durations also qualify the comparisons.

HumanMade receives 4.6 and PaperTalker 3.8, with PaperTalker ahead of PresentAgent at 2.8, Veo3 at 2.2, and Wan2.2 at 1. The chart supports PaperTalker’s preference advantage over the tested generated baselines while retaining a gap relative to human-made videos.

Parallelization reduces PaperTalker runtime from 287.2 to 48.1 minutes while token usage remains 62K. Removing the presenter lowers runtime to 15.6 minutes, indicating its computational burden; the reported dollar costs have no detailed accounting basis.

5.3 Qualitative Analysis

The qualitative examples show PaperTalker combining slide content, a presenter, speech, and cursor cues. Veo3 produces a visually convincing presenter but blurred slide text, while PresentAgent examples show an incorrect title, incomplete author information, dense text, and content extending beyond the slide. These selected examples illustrate failure modes without establishing their frequency across the benchmark.

The selected examples show blurred text in Veo3 and incorrect, incomplete, or overflowing content in PresentAgent. PaperTalker displays the coordinated components, including cursor cues, but the examples do not quantify benchmark-wide failure rates.

5.4 Key Ablations

The cursor ablation uses four-option localization questions about the slide region associated with a subtitle sentence; model accuracy increases from 0.084 without the cursor to 0.633 with it. In the slide-layout ablation, Tree Search Visual Choice raises design score from 2.73 to 2.85 on a 1–5 scale, while content changes from 4.33 to 4.34 and coherence remains 3.84. These results support improved model localization and a modest design-score gain, with before-and-after examples showing repaired overflow.

Adding the cursor increases localization accuracy from 0.084 to 0.633 in the model-based question-answering task. This supports the usefulness of an explicit visual cue for the evaluator’s localization task without directly measuring human comprehension.

Tree Search Visual Choice raises GPT-4.1 PaperTalker’s design score from 2.73 to 2.85, matching the HumanMade design score. Content changes from 4.33 to 4.34 and coherence remains 3.84, indicating that the measured benefit is concentrated in layout design.

The paired examples show figures extending beyond slide boundaries before refinement and fitting within the layout afterward. They provide qualitative evidence of overflow repair, complementing the design-score improvement in Table 5.

6 Conclusions

The paper concludes that coordinating specialized generation tools can automate academic presentation production and that Paper2Video provides a benchmark for this task. The reported results support higher model-based quiz accuracy than the tested alternatives and approximately 6× faster generation than serial PaperTalker. PresentArena and the human study still favor author-recorded videos.

Appendix

• A Evaluation Metrics and A.1 IP Memory specify the speaker–work association task. The evaluator receives four randomly sampled video–question pairs, each using a five-second presentation clip and an understanding-level PresentQuiz question, with videos and questions shuffled alongside one speaker image. Accuracy in selecting the relevant question defines IP Memory; this tests association under the specified prompt rather than delayed human recall.
• B Experiment directs readers to supplementary materials in B.1 Video Results. B.2 Results of Tree Search Visual Choice presents Figure 8 as qualitative evidence of overflow repair and layout improvement.
• C Prompts documents slide generation, compilation repair, visual layout judging, script generation with cursor descriptions, and automated evaluation. The slide prompt requests around 10 slides; the script prompt limits each slide to 50 words or fewer and pairs each sentence with one cursor description using script | cursor description, separating slides with ###. The layout prompt is labeled MSTS Judge, with no expansion provided, and chooses among four candidates in a 2×2 grid; the Meta Similarity prompt uses a lenient 0–5 rubric, PresentArena permits ties, and the final quiz prompt is labeled PresentationQuiz.

Brief Thoughts

The engineering contribution lies in the intermediate assets: compilation diagnostics identify repair targets, rendered layout candidates support visual selection, and sentence-level prompts connect narration to cursor coordinates. The ablations provide evidence for these choices, with a small design-score improvement from layout refinement and a larger gain in model-based localization from cursor cues.

Higher quiz accuracy measures information accessible to the chosen VideoLLM, while IP Memory measures short-clip speaker–work association; neither directly establishes improved human understanding or lasting impact. The 101-paper AI conference dataset, original-slide availability for only 40% of examples, small human study, unequal video durations, and unspecified cost accounting limit broader conclusions about presentation quality and production economics.