VLM2Vec-V2: Advancing Multimodal Embedding for Videos, Images, and Visual Documents | Summary

Contents

Summary
1 Introduction
2 MMEB-V2: Expanding Embedding Benchmarks Beyond Image-Text
3 Unified Embedding Model for Video, Image, and Visual Document
4 Experiments
5 Related Works
6 Conclusion
Appendix
Brief Thoughts

This article explains the key points of VLM2Vec-V2: Advancing Multimodal Embedding for Videos, Images, and Visual Documents.

This review uses arXiv v1 (7 July 2025).

2025-07-07 (arXiv)
Meng, Rui, Jiang, Ziyan, Liu, Ye, Su, Mingyi, Yang, Xinyi, Fu, Yuepeng, Qin, Can, Chen, Zeyuan, Xu, Ran, Xiong, Caiming, et al.
Salesforce Research, UC Santa Barbara, University of Waterloo, Tsinghua University
Paper
Github
Project Page

Read this article in Korean

Summary

Multimodal embedding models map different input types into a shared vector space for matching and retrieval, but existing models offer limited support for videos and visually structured documents. Across the 78 tasks of MMEB-V2, the expanded Massive Multimodal Embedding Benchmark, Table 2 reports an overall score of 58.0 for VLM2Vec-V2 (2 billion parameters), compared with 57.8 for GME, a multimodal embedding baseline with 7 billion parameters, and 47.0 for the earlier VLM2Vec-Qwen2VL (2 billion parameters).
VLM2Vec-V2 adapts Qwen2-VL, a vision-language model that processes text and visual inputs, through instruction-conditioned contrastive learning: task instructions specify what to match, and training increases similarity for matching query–target pairs relative to nonmatching pairs. Training combines image, video, and document data with interleaved sub-batching, which groups examples from individual sources inside a mixed global batch.
The model has the highest reported aggregate score but does not lead every modality. Table 2 reports image, video, and document scores of 64.9, 34.9, and 65.4; GME (7B) scores higher on video and documents, and several video aggregates differ between the main and appendix tables.

1 Introduction

The introduction identifies a coverage gap in embedding training and evaluation: MMEB and M-BEIR, an earlier multimodal retrieval benchmark, largely emphasize natural images, while practical search also involves videos, PDFs, websites, and slides. The paper contributes an expanded benchmark and a unified model, then investigates transfer between visual modalities, training-data composition, and the representation of temporal and document structure.

2 MMEB-V2: Expanding Embedding Benchmarks Beyond Image-Text

MMEB-V2 expands the original 36 image tasks with 42 tasks: five video retrieval tasks, three moment retrieval tasks, five video classification tasks, five video question-answering tasks, and 24 visual document retrieval tasks. Figure 1 groups the resulting 78 tasks into nine meta-tasks, or task categories; although parts of the prose describe five original categories, the figure and counts show four original image categories.

Video retrieval ranks candidate videos for a query; moment retrieval selects a relevant temporal segment from approximately 10 candidate clips.
Video classification ranks textual class labels, while video question answering ranks candidate answers to a question about a video.
Visual document retrieval ranks pages or documents for natural-language queries, including both page-level and document-level variants of ViDoSeek and MMLongBench-Doc, two document question-answering datasets adapted for retrieval.

MMEB-V2 task organization, with blue borders for original image tasks and red borders for added video and visual document tasks.

Tasks use instructions to specify the desired query–target relationship, and some datasets are downsampled to keep evaluation practical. Image and video tasks use Hit@1, the fraction of queries with a correct target ranked first; document tasks use NDCG@5, normalized discounted cumulative gain measuring ranking quality over the top five results, averaged across 24 tasks.

Input modalities, domains, query counts, and candidate counts for the 42 added benchmark tasks.

3 Unified Embedding Model for Video, Image, and Visual Document

The method aligns different input types in a shared embedding space while conditioning query representations on natural-language task instructions. Its components are a shared vision-language backbone, contrastive optimization, standardized input formatting, and sampling across training sources.

3.1 Unified Representation of Multimodal Data

The backbone is Qwen2-VL, selected for variable-resolution visual processing, Multimodal Rotary Position Embedding (M-RoPE), which represents spatial and temporal positions, and an architecture combining two-dimensional and three-dimensional convolutions for images and video. These inherited capabilities support a common encoder for interleaved text and visual inputs.

3.2 Contrastive Learning

Each query and target is encoded using the final-layer representation of its last token. The model minimizes InfoNCE, a contrastive loss that distinguishes a positive target from in-batch negatives, the nonmatching targets in the batch, and hard negatives, more difficult nonmatching targets: L = −log[φ(h_qinst, h_t+) / (φ(h_qinst, h_t+) + Σ_{t−∈N} φ(h_qinst, h_t−))], with φ(h_q, h_t) = exp(cos(h_q, h_t)/τ), where h denotes an embedding, N is the negative set, and τ is the temperature controlling score scaling.

Training examples pair a query with a positive target, with either side containing text, one or more images, or interleaved text and images. The query template is [VISUAL_TOKEN] Instruct: {task_instruction}\n Query: {q}; [VISUAL_TOKEN] denotes modality-specific tokens such as <|image_pad|> and <|video_pad|>.

The task instruction describes the required relationship, such as finding a video that contains specified visual content.
An optional target instruction, such as “Understand the content of the provided video:”, guides target encoding.

3.4 Data Sampling Strategies

A predefined sampling-weight table controls the probability of drawing from each dataset during training. Interleaved sub-batching groups similar examples from a source to make contrastive discrimination harder, while combining multiple source groups in the global batch to retain task diversity.

Section 3.4 illustrates a batch of 1,024 as eight sub-batches of 128; the training setting in Section 4.1.2 uses sub-batches of 64.
The sub-batch-size ablation tests how this grouping affects performance across modalities.

4 Experiments

The experiments compare embedding models across image, video, and visual document tasks, then vary modality combinations, sampling strategies, adaptation capacity, and training duration. Evaluation covers the original image benchmark and the added video and document tasks.

4.1 Experiment Setting

4.1.1 Training Data combines LLaVA-Hound, a source of synthetic video-language examples, document retrieval examples, and MMEB-train image-text tasks. The video data contain 300k video-caption pairs and 240k video question-answering pairs; document training uses colpali train set (118k), VisRAG synthetic (239k), and VisRAG in-domain (123k).

Caption pairs are used in both text-to-video retrieval and video-to-caption retrieval formats.
MMEB-train supplies image classification, question answering, retrieval, and visual grounding, where grounding identifies relevant visual regions.

4.1.2 Training Setting uses Qwen2-VL 2B, a global batch size of 1,024, 2K/5K training steps, sub-batches of 64, and temperature 0.02. LoRA, low-rank adaptation for parameter-efficient tuning, uses rank 16 and scaling factor α = 32; GradCache, a method for enabling large contrastive batches under memory constraints, supports training on 8 H100 GPUs.

Each video is represented by 8 uniformly sampled frames during both training and evaluation.
The reported training setting does not specify a learning rate, optimizer, or training wall-clock measurement.

4.1.3 Baselines includes GME, VLM2Vec, LamRA, a multimodal retrieval model, and the document retrieval model ColPali v1.3. GME and LamRA use a single middle frame for video evaluation, while the remaining models use 8 uniformly sampled frames, so the comparison includes differences in both embedding training and visual input coverage.

ColPali uses multiple vectors and late interaction, a matching mechanism that compares query and document representations after encoding, whereas VLM2Vec-V2 uses a single last-token representation.

4.2 Main Results

Table 2 reports the highest overall score for VLM2Vec-V2 (2B): 58.0, compared with 57.8 for GME (7B), 54.1 for GME (2B), and 47.0 for VLM2Vec-Qwen2VL (2B). Against the earlier 2B VLM2Vec, image performance rises from 59.7 to 64.9, video from 29.0 to 34.9, and document retrieval from 41.6 to 65.4.

The 64.9 image score is below VLM2Vec-Qwen2VL (7B) at 65.5.
GME (7B) leads video at 38.6 and document retrieval at 75.2 in Table 2; ColPali also exceeds VLM2Vec-V2 on documents, scoring 71.0.
The 0.2-point overall advantage over GME (7B) is reported without uncertainty estimates or repeated-run statistics.

Main performance comparison across 36 image, 18 video, and 24 visual document tasks.

4.3 Ablation Analysis

4.3.1 Generalization Across Modalities excludes ViDoRe-V2, the second version of the visual document retrieval benchmark, and compares training on individual modalities and selected combinations. Image-only training gives the strongest single-modality average, 45.2; Image+Video reaches 48.3, Image+VisDoc reaches 47.7, and all three reach 49.1, where VisDoc denotes visual documents.

All three modalities produce the highest document score in this ablation, 52.2, but the highest image score is 63.3 with Image+Video.
The highest video score is 33.3 with Image+VisDoc, compared with 32.4 when all three are used; adding video training data does not uniformly improve video evaluation.

Evaluation scores for models trained on individual modalities and selected modality combinations, excluding ViDoRe-V2.

4.3.2 Ablation Study on Data Sampling Strategies tests interleaved sub-batch sizes 0, 32, 64, 128, and 1024. Image performance peaks at size 64 with 63.2, while document performance rises from 48.6 at size 0 to 54.3 at size 1024; video performance is nonmonotonic at 34.6, 33.2, 33.5, 34.5, and 35.4.

Size 0 independently mixes examples without source grouping; size 1024 draws the entire batch from one source.
The prose claims consistently improving video performance as sub-batch size increases, but Table 4 shows a decrease from size 0 to 32 before subsequent increases.

Image, document, and video performance across interleaved sub-batch sizes 0, 32, 64, 128, and 1024.

4.3.3 Ablation Study on Model Settings compares LoRA ranks 8, 16, and 32. Figure 2 reports the highest displayed scores at rank 16 for all three modalities: 63.2 for images, 52.6 for documents, and 33.5 for video.

The training-step plots show higher final scores at 5K steps than at the earliest plotted checkpoints, with intermediate declines or plateaus.
The authors suggest that longer training may help, especially for documents and video, but do not test training beyond 5K steps.

Modality performance under different LoRA ranks and across training checkpoints up to 5K steps.

The related-work discussion covers multimodal embedding benchmarks, video representations, visual document representations, and unified retrieval. It positions MMEB-V2 and VLM2Vec-V2 as extending benchmark and model coverage across these settings.

5.1 Multimodal Embedding Benchmarks

Earlier image-text datasets support captioning and retrieval, while M-BEIR and MMEB broaden evaluation to multiple embedding tasks. The paper argues that these benchmarks still emphasize static images and short contexts, motivating shared instruction-based evaluation across videos and structured documents.

Linear probing, which trains a linear classifier on representations, is discussed as an existing way to assess image representation quality.
Video benchmarks such as MSR-VTT and QVHighlights provide task-specific evaluation but do not themselves supply a unified embedding evaluation framework.

5.2 Video Representation Learning

The paper reviews video representation methods that combine visual processing with language supervision, discussing VideoCLIP and VideoCoCa in relation to contrastive and captioning objectives. InternVideo2 combines masked video modeling, which learns from partially hidden video inputs, cross-modal contrastive learning, which aligns different input types, and next-token prediction, which predicts the next sequence element.

LLaVE and LamRA are cited as evidence that image-text training can transfer to text-video retrieval without video-specific training.
The reported baseline comparison does not establish superiority over dedicated video specialists.

5.3 Visual Document Representation Learning

Visual document methods preserve information carried by page images, including text, charts, tables, and layout. ColPali provides document embeddings, VisRAG embeds documents as images for retrieval-augmented generation (RAG), which supplies retrieved evidence to a generator, and ViDoRAG uses iterative reasoning by multiple agents over visual documents.

MMLongBench-Doc evaluates long-context document understanding and supplies evidence-page annotations that this paper repurposes for retrieval.

5.4 Unified Modality Retrieval

GME and Uni-Retrieval pursue retrieval across different input types within one framework. UniversalRAG and UniRAG route queries to suitable modalities and granularities, such as pages or documents; the paper distinguishes its contribution by jointly targeting image, video, and visual document embeddings.

6 Conclusion

The conclusion presents MMEB-V2 as a broader evaluation suite and VLM2Vec-V2 as a contrastively trained unified baseline. The reported results demonstrate improved aggregate performance over the compared models and provide diagnostics for training across visual modalities.

Appendix

A Appendix contains author contributions, baseline descriptions, benchmark construction details, and full scores. A.1 Author Contributions assigns technical leadership and benchmark development to Rui Meng and Ziyan Jiang, most training experiments to Ye Liu, overall evaluation to Can Qin, and document curation and evaluation contributions to Xinyi Yang. A.2 Details of Baseline Models distinguishes VLM2Vec’s instruction-following ranking, ColPali’s multi-vector late interaction, LamRA’s language-only pretraining followed by multimodal instruction tuning, and GME’s contrastive multimodal retrieval.
A.3 Details of Benchmark Construction documents selected splits and adaptations. A.3.1 Video Retrieval uses 1,000 MSR-VTT pairs, 1,004 DiDeMo paragraph-video pairs, 670 MSVD pairs, 3,179 YouCook2 pairs after removing overlap with HowTo100M, a large narrated-video dataset, and 4,468 available VATEX videos.
A.3.2 Moment Retrieval evaluates QVHighlights by ranking candidate clips using Recall@1, whether a relevant clip is retrieved first; it filters Charades-STA to examples whose relevant segment occupies less than one-third of the video and samples nine negative clips per MomentSeeker query alongside all positive clips. Candidate ranking differs from temporal localization evaluated with Intersection over Union (IoU), the overlap ratio between predicted and reference temporal intervals.
A.3.3 Video Classification samples 1,000 examples each for Kinetics-700, Something Something v2, HMDB51, and UCF101, while Breakfast uses 433 clips from camera 01. A.3.4 Video QA describes multiple-choice evaluation using MVBench, Video-MME, NExT-QA, and a public 500-question EgoSchema subset; it does not provide a corresponding construction paragraph for ActivityNetQA.
A.3.5 Visual Document Retrieval explains how questions become retrieval queries and annotated pages become relevant targets. ViDoRe includes queries from academic question-answering datasets and synthetic queries for collected documents, ViDoRe-V2 adds four challenging multilingual datasets, VisRAG filters context-dependent questions, and ViDoSeek and MMLongBench-Doc supply page evidence for adapted retrieval tasks.
A.4 Detailed Scores provides Table 5 with all 78 task results. For VLM2Vec-V2, it reports 21.9 on ViDoSeek-page versus 80.2 on ViDoSeek-doc, and 11.9 on MMLongBench-page versus 43.7 on MMLongBench-doc, showing substantially different outcomes at page and document granularity.

Brief Thoughts

The benchmark’s common ranking formulation makes it useful for comparing one embedding model across several visual forms. VLM2Vec-V2 substantially improves the earlier 2B VLM2Vec, but its 58.0 aggregate should be read alongside the modality breakdown: its image score is close to the strongest compared model, while GME (7B) remains stronger on video and documents.

The numerical evidence needs reconciliation before exact video comparisons can be reproduced: Table 2 gives VLM2Vec-V2 video overall and moment retrieval scores of 34.9 and 38.5, whereas Table 5 gives 34.6 and 36.8. Unequal video frame counts across baselines, absent uncertainty estimates, and candidate-ranking adaptations limit what the results establish about temporal understanding and the 0.2-point overall advantage over GME (7B).

The ablations support tuning source composition and batch structure, but no tested setting maximizes every modality. The project URL is https://tiger-ai-lab.github.io/VLM2Vec/; the reviewed source is arXiv:2507.04590v1, dated 7 July 2025.

Detailed scores for all compared models across the full 78-task MMEB-V2 benchmark.