Video models are zero-shot learners and reasoners | Summary

This article explains the key points of Video models are zero-shot learners and reasoners.

• 2025-09-24 (arXiv)
• Wiedemer, Thaddäus, Li, Yuxuan, Vicol, Paul, Gu, Shixiang Shane, Matarese, Nick, Swersky, Kevin, Kim, Been, Jaini, Priyank, Geirhos, Robert.
• Google DeepMind
• Paper
• Project Page

Read this article in Korean

Summary

• The paper asks whether a model trained to generate videos can solve visual tasks from instructions without task-specific training or adaptation, a setting it calls zero-shot learning. With the same image-and-text task setup and up to 10 attempts, Veo 3 solves 78% of 5×5 mazes compared with Veo 2’s 14%, although performance falls on larger mazes.
• The study reports 18,384 generated videos across 62 qualitative and 7 quantitative tasks. It organizes capabilities into perception, understanding visual information; modeling, representing the visual world and its properties; manipulation, changing that world; and reasoning, coordinating changes across space and time.
• The method supplies an initial image and a text instruction to publicly available Veo models, which generate 8s videos at 720p and 24 FPS, or frames per second. The authors call sequential visual problem solving chain-of-frames (CoF), meaning task-relevant changes expressed across generated video frames.
• Selecting the best frame and maximum score across 10 attempts gives Veo 3 an edge-detection score of 0.77, measuring agreement with annotated edges, and a segmentation score of 0.74, measuring overlap with annotated object masks. The task-specific edge-detection reference reaches 0.90; visual reasoning is also uneven, with strong symmetry completion under best-frame selection but reflection and rotation analogies below the 0.33 chance baseline.
• The results demonstrate capabilities under carefully chosen prompts, but usable outputs depend on repeated sampling, output selection, and control of unwanted animation. A language-model-based prompt rewriter, which modifies instructions before video generation, complicates attribution to the generator, and correct intermediate frames are not always preserved until the final frame.

1. Introduction

The paper draws an analogy with large language models (LLMs), generative models that handle multiple language tasks through prompting, and asks whether video generation provides a similar route to general-purpose vision. Here, zero-shot learning replaces task-specific fine-tuning, updating a model for a particular task, or additional inference heads, task-specific output modules, with task instructions.

• The empirical contribution is an investigation of an existing generation system, rather than a newly trained architecture.
• The authors distinguish broad task coverage from specialist-level performance: their introductory claims acknowledge that task-specific models still outperform the zero-shot video model.
• The project page is https://video-zero-shot.github.io/; the supplied paper is arXiv:2509.20328v2.

2. Methods

The authors query Veo 2 and Veo 3 through Google Cloud’s Vertex AI API, the application programming interface used for generation, with an input image serving as the first frame and a textual task description. Outputs have a 16:9 aspect ratio, 720p resolution, 24 FPS, and an 8s duration; the Methods section lists veo-3.0-generate-preview and veo-2.0-generate-001.

• The API includes an LLM-based prompt rewriter [30]. The authors evaluate the rewriter and generator as one black-box system and acknowledge that some solutions, such as Sudoku, may originate in the rewriter.
• They report that standalone Gemini 2.5 Pro could not reliably solve robot navigation, maze solving, and visual symmetry from the input images alone. This comparison probes the attribution problem but does not isolate the generator’s contribution within the combined system.
• Veo was chosen for strong video-generation leaderboard performance [31]. Comparing Veo 2 with Veo 3 measures a difference between publicly available model generations, without identifying which training or architectural changes caused it.

3. Qualitative results: Sparks of visual intelligence?

Each of the 62 qualitative tasks receives 12 Veo 3 generations, and the authors judge the fraction that solves the task. Figures 1 and 2 summarize coverage and examples; these small-sample success rates indicate possible capabilities and approximate reliability.

• Perception includes segmentation, separating distinct objects; keypoint localization, identifying specified image locations; super-resolution, increasing image detail; restoration; and conjunctive search, selecting an object by combined attributes. The paper explicitly excepts denoising, removing image noise, from its claim that these tasks were not trained for, because denoising is a standard diffusion training objective.
• Modeling examples cover flammability, rigid and deformable objects, gravity, buoyancy, whether objects float or sink, optical effects, category distinctions, character patterns, and preservation of scene state through camera movement.
• Manipulation includes inpainting, filling missing image regions; outpainting, generating content beyond the original image; new viewpoints; pose changes; and simulated hand interactions. Object affordances, the actions an object permits, are tested through natural hammer grasping; no actual robot experiments are conducted.
• Reasoning examples include graph traversal, following connections between nodes; sequence completion; tool use; mazes; navigation; and rule extrapolation, applying a rule inferred from examples. CoF interprets frame-by-frame task execution; the paper does not establish it as a causal reasoning mechanism through a controlled comparison.

The distribution across 62 tasks shows broad but uneven capability: many restoration and manipulation tasks have high author-rated success, while several reasoning tasks succeed rarely. Each bar is based on only 12 generations, so the plot provides an exploratory estimate of reliability.

The sequences illustrate the proposed capability hierarchy, from improving image detail to navigating a scene or completing a visual rule. Selected successful sequences demonstrate possible outcomes without establishing their frequency.

4. Quantitative results

The seven quantitative tasks evaluate perception, manipulation, and reasoning; physical modeling receives no quantitative evaluation in this study. Where applicable, the authors compare Veo 2, Veo 3, and Nano Banana, an image-editing model used as a reference [54].

• Best-frame evaluation selects the highest-scoring frame anywhere in a generated video and measures a performance ceiling. Last-frame evaluation uses a predetermined output position and exposes degradation when generation continues after a correct solution.
• For binary success tasks, pass@k measures whether at least one of k attempts succeeds; edge detection and segmentation instead plot the maximum metric over k attempts. Selecting the best attempt or frame uses evaluation information that is not automatically available in deployment.
• Most experiments generate 10 outputs per input with a fixed prompt. Visual analogy aggregation is an exception: its multiple-attempt results use majority voting, selecting the most frequent judged answer, rather than selecting a successful attempt using known correctness.

4.1. Perception: Edge detection

On all 50 test images from BIPEDv2, an edge-detection dataset [59, 60], Veo 3 reaches 0.77 best-frame OIS across 10 attempts, compared with 0.57 for Veo 2 and 0.74 for Nano Banana. OIS, called optimal image scale in the paper, is the maximum F1-score over thresholds; F1 combines precision, the fraction of predicted edges that match annotations, and recall, the fraction of annotated edges recovered.

• Last-frame scores across 10 attempts are 0.74 for Veo 3, 0.51 for Veo 2, and 0.74 for Nano Banana. The task-specific reference score is 0.90 [60].
• Evaluation suppresses non-maximal edge responses, thresholds and thins edge maps, tests 16 evenly spaced thresholds, and permits local edge displacement within 0.75% of the image diagonal, approximately 11 pixels.
• The authors show that some measured false positives correspond to visible foliage, reflections, and tire details missing from annotations. Their graded example also shows genuine missed edges, so annotation coverage does not explain every error.

At 10 attempts, Veo 3 reaches 0.77 best-frame OIS compared with 0.57 for Veo 2 and 0.74 for Nano Banana. Its last-frame score is 0.74, showing a smaller temporal reduction than in segmentation and symmetry.

4.2. Perception: Segmentation

The segmentation experiment uses 50 randomly selected LVIS images, from a dataset of annotated object instances [61], containing one to three objects with at least 5000 pixels each. Models must delineate all objects without receiving classes, locations, or instance counts; performance is measured by mean Intersection over Union (mIoU), the average ratio of mask intersection to mask union.

• With a green-background prompt, Veo 3 achieves 0.74 best-frame mIoU across 10 attempts, versus 0.50 for Veo 2 and 0.73 for Nano Banana. A white-background prompt reduces Veo 3’s score to 0.66; the proposed explanation involving green-screen prevalence is a hypothesis.
• Veo 3’s last-frame score is 0.56 with green and 0.42 with white, compared with Nano Banana’s 0.73. Similar best-frame scores therefore do not imply similar final-output performance.
• Masks are extracted from color differences, and each ground-truth mask is paired with the predicted mask having the highest overlap. The score applies to a deliberately easy image subset and this matching procedure.

The green-background prompt gives Veo 3 a 0.74 best-frame mIoU ceiling, but its last-frame score falls to 0.56 while Nano Banana reaches 0.73. The white-background condition further lowers Veo 3’s performance, exposing prompt dependence and failure to preserve intermediate masks.

4.3. Manipulation: Object extraction

Veo must extract every animal from a scene and arrange them separately on a white background. The custom dataset contains 54 manually checked images, exactly six for each count from one to nine, generated using Nano Banana.

• The evaluator thresholds the last frame and counts connected components, contiguous foreground regions, to estimate the number of extracted animals.
• Veo 3 reaches 93% pass@10, compared with Veo 2’s 63%, which is around the baseline for repeated random guesses of counts from one to nine.
• This metric checks the animal count, without verifying every animal’s identity, appearance, or extraction boundary. The authors acknowledge that a perfect model should achieve 100% on this simple task.

Veo 3’s last-frame count accuracy improves to 0.93 across 10 attempts, while Veo 2 reaches 0.63 and remains around the repeated-guessing baseline. The evaluator checks the animal count after rearrangement rather than complete visual fidelity of extraction.

4.4. Manipulation: Image editing

Three human raters assess edits on 30 randomly sampled Emu-edit test images, from an instruction-based image-editing dataset [62]. They distinguish fidelity, whether the requested edit is correct, from precision, whether that correct edit avoids unintended changes.

• Figure 6 reports 100% fidelity and 60% precision for Veo 3, compared with 20% and 10% for Veo 2.
• The gap between fidelity and precision captures a practical failure mode: a correct edit may accompany unwanted camera motion, movement of people, or other scene changes.
• Appendix B.4 says two samples were generated per image and the first was used for human rating. The reported human evaluation is therefore not a best-of-10 result.

Veo 3’s 100% fidelity versus 60% precision indicates that requested edits can succeed while unrelated scene changes remain. Veo 2’s corresponding ratings are 20% and 10%; this human evaluation uses the first generated sample rather than best-of-10 selection.

4.5. Reasoning: Maze solving

A red circle must reach a green goal through white maze paths without crossing black walls or changing the goal’s position. Frame-by-frame verification checks the complete trajectory, so placing the circle at the goal in the final frame is insufficient.

• At pass@10, Veo 3 scores 78%, 38%, and 12% on 5×5, 7×7, and 9×9 grids, compared with Veo 2’s 14%, 2%, and 2%. Performance declines sharply as grid size increases.
• On irregular mazes, Veo 3 reaches 75%, while Veo 2 and Nano Banana score 0%. Nano Banana scores 74%, 40%, and 22% across the three rectangular sizes.
• Gemini 2.5 Pro receives either images, labeled I2T for image-to-text, or ASCII character-grid representations, labeled T2T for text-to-text. Its text-input scores are 92%, 72%, and 8%, showing that the relative ranking depends on maze size and input representation.
• Appendix B.5 describes 50 random rectangular mazes per size and 40 irregular samples derived by flipping and rotating 10 hand-drawn mazes. Table 1 lists 2 × 4 maze splits with 50 images per split, leaving the aggregate maze accounting unresolved.

Veo 3 improves over Veo 2 at every maze size, but its pass@10 declines from 78% on 5×5 grids to 12% on 9×9 grids. Its 75% on irregular mazes contrasts with Nano Banana’s 0%, while Gemini’s stronger results on small text-represented grids show that input representation changes the comparison.

4.6. Reasoning: Visual symmetry solving

The visual symmetry task requires completing the right half of a 10×16 grid by reflecting its left half across the central vertical axis while preserving existing cells. The two synthetic splits contain 25 recognizable shapes and 25 random patterns; success requires zero incorrectly colored cells.

• At best-frame pass@10, Veo 3 achieves 88% on shapes and 100% on random patterns, versus 20% and 0% for Veo 2 and 37% and 28% for Nano Banana.
• Veo 3’s corresponding last-frame scores fall to 44% and 72%. Correct intermediate solutions are therefore often not preserved until generation ends.
• Table 2 tests 10 prompt variants and reports best-frame pass@1 ranging from 8% to 48% on shapes and 4% to 68% on random patterns: differences of 40 and 64 percentage points.
• Cell correctness uses average-color differences in CIELAB, a perceptual color space, with a threshold of 15.0.

Veo 3’s best-frame pass@10 reaches 88% for shapes and 100% for random patterns, but last-frame results fall to 44% and 72%. The plot separates producing an exact solution from retaining it until generation ends.

4.7. Reasoning: Visual analogy completion

Visual analogy completion asks the model to fill the missing quadrant of a 2×2 arrangement by applying the transformation illustrated by a reference object pair. On KiVA, the Kid-inspired Visual Analogies benchmark [66], the study evaluates 50 samples each for color, resize, reflect, and rotate transformations.

• Last-frame pass@1 scores for Veo 3 are 95%, 67%, 29%, and 19%, respectively, compared with 68%, 40%, 23%, and 22% for Veo 2. Both models fall below the 0.33 chance baseline on reflection and rotation.
• A Gemini 2.5 Pro autorater, an automated image-comparison evaluator, maps each generated object to one of three candidate answers. It receives condition-specific information and exceeds 88% agreement with author judgments in a pilot of 25 samples per condition.
• When no object appears or the object type is wrong, the evaluation randomly assigns one of the three choices. Invalid generations consequently contribute at chance level in expectation.
• Figure 61 shows that majority voting over 10 attempts yields 98%, 68%, 22%, and 14% for Veo 3. Voting reinforces errors on reflection and rotation, consistent with the authors’ observation that transformations often use the wrong axis.

Veo 3 reaches 95% and 67% pass@1 on color and resize analogies. Reflection and rotation remain at 29% and 19%, below the 0.33 chance baseline, limiting claims of general transformation reasoning.

5. Discussion

The authors interpret broad task coverage and the improvement from Veo 2 to Veo 3 as evidence that video generation could support general-purpose vision models. They describe measured performance under particular image-and-text prompts as a lower bound on capability, because alternative task representations and prompts can change success.

• The distinction between observed performance and underlying capability [69, 70] does not establish success on untested prompts or tasks. Documented failures and prompt sensitivity remain part of the empirical result.
• The paper acknowledges that video generation is more expensive than specialist inference and cites declining LLM inference prices [71] as an analogy. It provides no direct video-cost trajectory or measured cost-performance comparison.
• Inference-time scaling, spending more computation on repeated or refined outputs [72–75], and post-training with automatic verification are proposed improvements. The study tests repeated sampling but does not test these post-training interventions.
• Table 1 reports 17,640 quantitative-study videos plus 744 qualitative videos, giving the stated 18,384 total. Table 2 and Figure 61 qualify the benefit of repeated sampling: prompts strongly affect performance, and majority voting can worsen systematic errors.

The reported 17,640 quantitative-study videos plus 744 qualitative videos yield the headline total of 18,384. The breakdown documents the sampling investment, although its maze and editing counts do not fully align with the associated appendix descriptions.

Majority voting across 10 attempts raises Veo 3’s color accuracy to 98% but lowers reflection and rotation to 22% and 14%. Voting can amplify recurring wrong transformations, so repeated sampling does not uniformly improve the selected answer.

Across 10 prompts, best-frame pass@1 ranges from 8% to 48% on shapes and 4% to 68% on random patterns. Wording materially affects measured performance, and adding camera-control instructions does not uniformly improve either success rates or cell errors.

Author contributions

The paper lists Thaddäus Wiedemer, Priyank Jaini, and Robert Geirhos as leads, with the project idea attributed to Robert Geirhos. Yuxuan Li and Paul Vicol are core contributors, Shixiang Shane Gu is a partial contributor, Nick Matarese contributed Figure 38, and Kevin Swersky and Been Kim served as advisors.

Appendix

• A. Qualitative results: Perception, Modeling, Manipulation, Reasoning supplies image sequences, task prompts, and author-rated success rates. A.1. Perception documents edge detection, segmentation, keypoint localization, restoration, attribute-based search, and ambiguous-image interpretation; the Rorschach example, interpreting inkblot patterns as objects, has undefined task correctness and reports 1.0 only for prompt following.
• A.2. Modeling covers material properties, rigid and soft bodies, falling objects, buoyancy, object removal and packing, optics, color mixing, categories, character structure, and scene memory. Reported successes vary substantially: material properties score 0.25, bottle-cap buoyancy 0.58, rock buoyancy 0.83, and returning to a scene after camera zooming 1.0.
• A.3. Manipulation includes background removal, style changes, colorization, adding color to a grayscale image; missing-region completion; scene composition; novel viewpoints; pose changes; and simulated hand actions. Inpainting and doodle-based editing score 1.0, while colorization and rotating Baoding balls, two balls manipulated in one hand, each score 0.08; these rates come from 12 generations per qualitative task.
• A.4. Reasoning covers graph traversal, Tree BFS, breadth-first search represented through progressive tree filling; sequences; matching colors; fitting shapes; sorting numbers; tool use; Sudoku; water puzzles; mazes; robot navigation; and rule extrapolation. Tool-based walnut retrieval scores 0.92, but requiring retrieval without intersecting the glass reduces success to 0.08, showing that apparent task completion can conceal a physical violation.
• B. Quantitative results: experimental details specifies datasets, prompts, sampling, and evaluators in B.1–B.7, corresponding to the seven main quantitative subsections. B.1 refines thresholded edges; B.2 extracts masks from hue differences using scipy.ndimage.gaussian_filter1d with standard deviation 2; B.3 counts thresholded components using scipy.ndimage.label; B.4 uses human ratings; B.5 checks complete maze paths; B.6 checks every symmetry cell; and B.7 compares analogy outputs with three choices.
• The appendix lists Veo 2 as veo-2.0-generate-preview-001, whereas Methods lists veo-2.0-generate-001; Veo 3 is consistently veo-3.0-generate-preview. The reference models are gemini-2.5-flash-image-preview for Nano Banana and gemini-2.5-pro for Gemini 2.5 Pro. Table 1 records 60 editing videos, while B.4 describes two generated samples per image with only the first rated, leaving the generation-count convention unclear.
• C. Prompting best practices recommends removing ambiguity, specifying what must remain unchanged, controlling the camera and change speed, and providing a motion outlet, an unrelated moving element intended to preserve the solved scene. It also suggests a visible completion indicator for extracting the solution and realistic task representations; Table 2 directly measures prompt sensitivity, while the broader recommendations summarize the authors’ experimental observations.
• D. Failure cases documents inaccurate depth and surface-normal maps, which encode distance and surface orientation; failure to follow motion annotations; impossible rope movement; incorrect path and word searches; invented graph edges and text; false image differences; incorrect visual puzzles; and implausible physical interactions. Tiling puzzles lose piece integrity, a golf ball passes through an undersized vase opening, laundry folding violates physics, and sofa motion planning changes the object instead of preserving rigid rotation and translation.
• E. LLM use discloses Gemini 2.5 Flash and Gemini 2.5 Pro assistance with task brainstorming, related-work suggestions, coding, and polishing human writing. F. Image sources distinguishes generated inputs, author drawings and modifications, benchmark images, and external images; many qualitative scenes were generated with Gemini, so the qualitative evidence is not exclusively based on natural photographs.

Brief Thoughts

The paper demonstrates that an existing video-generation system can produce solutions to a broad range of visual tasks specified through prompts. Complete-path maze verification and exact-cell symmetry checks provide more specific evidence of task competence than selected plausible-looking demonstrations. The best-frame versus last-frame gap identifies an engineering obstacle: output extraction and stopping must preserve a correct intermediate solution. The prompt rewriter, easy segmentation subset, generated qualitative inputs, and imperfect analogy autorater constrain conclusions about generality and attribution. Replacement of specialist vision models remains a hypothesis: specialist accuracy, operational reliability, and cost are not jointly established.