Kimi-VL Technical Report | Summary

This article explains the key points of Kimi-VL Technical Report.

• 2025-04-10 (arXiv)
• Kimi Team, Du, Angang, Yin, Bohong, Xing, Bowei, Qu, Bowen, Wang, Bowen, Chen, Cheng, Zhang, Chenlin, Du, Chenzhuang, Wei, Chu, et al.
• The University of Hong Kong, Moonshot.ai
• Paper
• Github
• Project Page

Read this article in Korean

Summary

• The report addresses how to build an efficient vision-language model, which processes images alongside text, that can reason over visual problems and handle long documents, videos, and computer interfaces. On the MathVision full set, a benchmark of mathematics competition problems with visual context, Kimi-VL-A3B-Thinking achieves 36.8% Pass@1, the success rate of one generated answer, compared with 21.4% for Kimi-VL, with both using 2.8B activated language-decoder parameters.
• The architecture combines MoonViT, an encoder that processes images at varying resolutions, with Moonlight, a Mixture-of-Experts (MoE) language model that routes tokens through selected expert modules. Training mixes text and multimodal data, extends the context window, the maximum input sequence length, to 131072 tokens, and adds long chain-of-thought (CoT), explicit intermediate reasoning, through supervised fine-tuning (SFT), learning from example responses, and reinforcement learning (RL), optimization using answer rewards.
• The original Kimi-VL exceeds Qwen2.5-VL-7B on 19 of 24 reported benchmarks. Its thinking extension improves results on MathVista, MMMU, and MathVision over the original model, though it does not lead every comparison. The report acknowledges limits in specialized problems and demanding long-context applications; activated parameter counts alone do not establish deployment latency or memory requirements.

Kimi-VL-Thinking scores 36.8% Pass@1 on MathVision with 2.8B activated language-model parameters. The horizontal axis supports a parameter-count comparison, but does not directly measure latency or memory.

1 Introduction

The introduction identifies gaps between open-source language models and vision-language models in computational efficiency, context length, and extended reasoning. Kimi-VL pairs a language decoder with 2.8B activated parameters and 16B total parameters with a 400M vision encoder, supporting single-image, multi-image, video, and long-document inputs. Figures 1 and 2 summarize reasoning performance relative to activated parameter counts and results across several task families.

The selected results cover infographic reading, multiple images, long inputs, and computer interaction. The 57.0 MMMU result also makes clear that these strengths do not imply a lead in every academic reasoning task.

2 Approach

The approach combines a native-resolution visual encoder, which accepts images at varying resolutions, a projection network, an MoE decoder, joint text and multimodal pre-training, and instruction and reasoning post-training. Text data remain present whenever the language model is updated, with the stated aim of preserving language capabilities while adding visual understanding.

2.1 Model Architecture

MoonViT divides variable-resolution images into patches and packs them into sequences, following NaViT, a method for processing images with different sizes and aspect ratios. It starts from SigLIP-SO-400M, a pretrained image-text encoder, and combines interpolated absolute positional embeddings with two-dimensional rotary positional embeddings (RoPE), which encode positions across image height and width. After a pixel-shuffle operation reduces each spatial dimension by a factor of two while expanding channels, a two-layer multilayer perceptron (MLP), a feed-forward projection network, maps visual features to language-model embeddings.

• The Moonlight decoder starts from an intermediate checkpoint trained on 5.2T text tokens with an 8192-token context window, then receives another 2.3T tokens of joint text and multimodal pre-training.

Inputs with different sizes and aspect ratios pass through the same MoonViT encoder and projection path before reaching the expert-routed decoder. The examples illustrate support for small images, video frames, screenshots, and wide text images.

2.2 Muon Optimizer

An enhanced Muon optimizer, the algorithm used to update model parameters, trains the vision encoder, projector, and language model. The modifications add weight decay, adjust parameter update scales, and distribute optimizer states using ZeRO-1, a strategy that reduces replicated optimizer memory. The report provides no controlled optimizer comparison for Kimi-VL.

2.3 Pre-Training Stages

The four pre-training stages consume 4.4T tokens: visual encoder training and alignment use 2T + 0.1T, joint pre-training uses 1.4T, joint cooldown uses 0.6T, and joint long-context training uses 0.3T. The visual objective combines a SigLIP contrastive loss, which aligns matching images and text, with image-conditioned caption generation: L = L_siglip + λL_caption, where \(\lambda=2\).

• During alignment, only MoonViT and the projector are updated. Joint pre-training begins with text data and progressively increases the multimodal proportion; cooldown continues training on high-quality text and multimodal sources, including a limited proportion of synthesized question-answer pairs.
• Long-context activation extends the window from 8192 to 131072 tokens in two fourfold steps and changes the RoPE base from 50,000 to 800,000. Each step uses 25% long data and 75% replay of shorter data.
• In the Needle-in-a-Haystack (NIAH) test, which measures retrieval of inserted information within long text or video sequences, recall is 100.0% through 65536 tokens. In the (65536, 131072] interval, it falls to 87.0% for text and 91.7% for video.

The diagram separates the decoder's preceding 5.2T text tokens from the subsequent 4.4T-token visual and joint training pipeline. Text remains part of each stage that updates the decoder, and the final stage introduces long text, videos, and documents.

Joint stages use 8192-token sequences until long-context training moves through 32768 to 131072. Although the visual-training column lists the vision encoder as trainable, the accompanying alignment description specifies updates to the projector as well.

Recall is perfect through 65536 tokens and declines to 87.0% for text and 91.7% for video in the longest interval. The test demonstrates retrieval across the advertised window with measurable degradation near its upper end.

2.4 Post-Training Stages

Joint SFT updates all three model components using text and vision-language conversations in ChatML, a dialogue format with explicit role markers. Only answers and special tokens receive supervision; system and user prompts are masked. Training runs for one epoch at 32K tokens and another at 128K tokens.

• The learning rate decays from 2×10⁻⁵ to 2×10⁻⁶ in the first stage, rises to 1×10⁻⁵ for the second stage, and then decays to 1×10⁻⁶.
• Long-CoT SFT uses verified reasoning examples covering planning, evaluation, reflection, and exploration. Subsequent RL uses a variant of online policy mirror descent, an iterative method that improves answer rewards while penalizing divergence from the current reference model.
• The answer reward is binary, based on correctness against ground truth. A response-length penalty discourages redundant reasoning, while curriculum and prioritized sampling select training examples using difficulty labels and observed success rates. At inference, the model generates tokens sequentially without a separate planning algorithm.

The branching point distinguishes instruction-following Kimi-VL from its thinking extension. Additional example-based training introduces extended reasoning before answer-reward optimization, with response length and example difficulty controlled during RL.

2.5 Infrastructure

The data system stores visual inputs in their original formats in S3-compatible object storage, a cloud interface for storing and retrieving data objects, and performs shuffling, mixing, tokenization, masking, and packing during training. It controls worker random states to preserve the data sequence after interrupted runs. Training combines data parallelism, replicating models across batches; expert parallelism, distributing expert modules; pipeline parallelism, distributing model layers; and context parallelism, splitting long sequences across devices.

• ZeRO-1 and selective activation checkpointing, recomputation of selected intermediate activations, reduce memory usage. Longer sequences trigger recomputation across more layers.
• The authors report approximately 60% higher training throughput than a 7B dense vision-language model, which uses its full parameter set rather than selected experts, such as one based on Qwen2.5-7B. The section does not specify the hardware configuration or detailed measurement conditions for this comparison.

3 Data Construction

Data construction combines filtering, synthesis, and deduplication across text, image, and video sources. The report describes pre-training data for broad capabilities, instruction data for dialogue and instruction following, and reasoning data for long-CoT training and RL.

3.1 Pre-Training Data

Multimodal pre-training covers captions, interleaved image-text documents, optical character recognition (OCR), which converts text in images into editable text, knowledge, agent interactions, and video. Chinese and English captions support image-text alignment, while interleaved documents preserve the order of images and surrounding text. Synthetic captions and dense video descriptions are limited to reduce the risk of unsupported generated content.

• OCR data include multilingual text, handwriting, dense layouts, tables, formulas, diagrams, and multi-page documents, with rotation, distortion, color changes, and noise augmentation. Knowledge data draw on academic and internet sources processed with layout parsing and OCR.
• Agent data combine screenshots and actions collected in virtual machines, icon data, and human computer-use demonstrations with synthesized reasoning. Grounding means locating the interface element associated with an instruction.
• Video data support long sequences and detailed spatial and temporal correspondence. Text data reuse Moonlight’s English, Chinese, Code, Mathematics & Reasoning, and Knowledge domains, with source evaluation and selective upsampling, increasing the sampling frequency of selected subsets.

3.2 Instruction Data

For non-reasoning instruction tasks, human annotations train a seed model, which generates multiple candidate responses that annotators rank and refine. For visual coding, visual reasoning, and mathematics or science problems, rejection sampling retains examples accepted by rule-based or model-based verification. The resulting standard SFT dataset has approximately a 1:1 ratio of text tokens to image tokens.

3.3 Reasoning Data

Reasoning data begin with ground-truth question-answer examples requiring multiple steps, including mathematics and domain-specific visual question answering, answering questions about visual inputs. Kimi k1.5 generates several detailed reasoning trajectories per question. A reward model evaluates predictions against true labels, and model judgments plus rule-based rewards filter incorrect responses.

4.1 Comparison to the State-of-the-Art Models

The evaluation first compares overall performance, then examines individual capabilities. Table 3 compares Kimi-VL with GPT-4o, GPT-4o-mini, Qwen2.5-VL-7B, Llama-3.2-11B-Inst., Gemma-3-12B-IT, and DeepSeek-VL2 across 24 benchmark rows. Kimi-VL uses 2.8B activated language-model parameters plus 0.4B vision parameters and exceeds Qwen2.5-VL-7B on 19 rows; missing entries mean that evaluation coverage differs across models.

• The table distinguishes accuracy from Pass@1 and reports OCRBench, a text-recognition and extraction benchmark, on its 1000-point scale.
• For WindowsAgentArena, a Windows computer-interaction benchmark, the footnote specifies that GPT-4o and GPT-4o-mini use Omniparser, an interface-parsing component, without UIA. The PDF does not explain UIA, so this setup difference cannot be fully assessed from the report.

The comparison supports the reported 19-of-24 advantage over Qwen2.5-VL-7B while showing lower results on MathVision, MMMU, and several general visual tasks. It separates language and vision activation counts and exposes gaps in competing models' reported agent results.

4.1.1 College-level Academic Problems

On MMMU validation, a college-level multimodal question benchmark, Kimi-VL scores 57.0%, above DeepSeek-VL2 at 51.1% but below Qwen2.5-VL-7B at 58.6% and Gemma-3-12B-IT at 59.6%. On VideoMMMU, a college-level video knowledge benchmark, Table 3 reports 52.6%, compared with 47.4%, 44.4%, and 57.2% for Qwen2.5-VL-7B, DeepSeek-VL2, and Gemma-3-12B-IT, respectively. Its 52.2% on MMVU validation, an expert-level video understanding benchmark, remains below GPT-4o-mini at 61.6% and GPT-4o at 67.4%.

4.1.2 General Visual Ability

Kimi-VL scores 83.1% on MMBench-EN-v1.1, a broad perception and reasoning benchmark, matching GPT-4o, and 84.9% on AI2D, a science-diagram benchmark, compared with GPT-4o’s 84.6%. It scores 66.7% on MMVet, which tests combinations of vision-language abilities, and 68.1% on RealWorldQA, which assesses real-world spatial understanding. On BLINK, a multi-image perception benchmark, its 57.3% exceeds Qwen2.5-VL-7B’s 56.4% but remains below GPT-4o’s 68.0%.

4.1.3 Mathematical Reasoning

On MathVista, a benchmark combining mathematical and visual reasoning, Kimi-VL achieves 68.7%, compared with Qwen2.5-VL-7B at 68.2% and GPT-4o at 63.8%. Its 21.4% on MathVision is lower than Qwen2.5-VL-7B at 25.1% and Gemma-3-12B-IT at 32.1%. The thinking extension’s gains on these tasks are evaluated separately in Table 4.

4.1.4 Document Understanding and OCR

Kimi-VL achieves 83.2 on InfoVQA, which tests interpretation of complex infographics, compared with Qwen2.5-VL-7B at 82.6 and GPT-4o at 80.7. It scores 867 out of 1000 on OCRBench, above Qwen2.5-VL-7B at 864 and DeepSeek-VL2 at 811. Table 3 labels InfoVQA as accuracy, whereas Appendix B.1 names its test-set scoring metric as ANLS without defining it; the value is therefore reported here without treating it as ordinary percentage accuracy.

4.1.5 Agent Grounding and Multi-turn Agent Interaction

Table 3 reports 92.8% on ScreenSpot-V2, a cross-platform interface-grounding benchmark, and 34.5% on ScreenSpot-Pro, which targets high-resolution professional interfaces, compared with Qwen2.5-VL-7B at 86.8% and 29.0%. On OSWorld, an execution-based computer-task benchmark, Kimi-VL achieves 8.22% Pass@1 versus GPT-4o’s 5.03%; on WindowsAgentArena, it reaches 10.4% versus 9.4%. These relative gains coexist with low absolute completion rates on multi-step tasks.

• The subsection’s prose gives 92.0% for ScreenSpot-V2, whereas Table 3 gives 92.8%; the value above follows the table.

4.1.6 Long Document and Long Video Understanding

On MMLongBench-Doc, which asks questions about documents reaching 100+ pages, Kimi-VL scores 35.1%, above Qwen2.5-VL-7B at 29.6% but below GPT-4o at 42.8%. On Video-MME, a video question-answering benchmark, it scores 67.8% without subtitles and 72.6% with subtitles, compared with Qwen2.5-VL-7B at 65.1% and 71.6%. It also reaches 74.2% on the multiple-choice subset of MLVU, a multi-task long-video benchmark, and 64.5% on LongVideoBench validation, a long-form video perception and relationship benchmark.

• GPT-4o scores 64.6% on MLVU and 66.7% on LongVideoBench. Kimi-VL’s advantage therefore varies across long-video benchmarks.

4.1.7 Egocentric and Fine-grained Video Perception

On the hidden full test set of EgoSchema, a first-person video understanding benchmark, Kimi-VL scores 78.5%, compared with GPT-4o at 72.2% and Qwen2.5-VL-7B at 65.0%. It reaches 37.4% on VSI-Bench, which evaluates spatial understanding in videos, versus 34.0% for GPT-4o and 34.2% for Qwen2.5-VL-7B. On TOMATO, which tests detailed temporal reasoning, its 31.7% exceeds Qwen2.5-VL-7B’s 27.6% but remains below GPT-4o’s 37.7%.

4.2 Kimi-VL-A3B-Thinking: A Reasoning Extension of Kimi-VL

The original Kimi-VL-A3B-Thinking reaches 71.3% on MathVista mini, 61.7% on MMMU validation, and 36.8% on MathVision full, improving over the non-thinking results by 2.6, 4.7, and 15.4 percentage points. Figure 13 tests inference budgets from 1k to 16k thinking tokens: MathVision rises from 18.7% to 36.8%, and MMMU from 49.2% to 61.7%. MathVista largely saturates by 4k tokens, scoring 70.9% there, 70.6% at 8k, and 71.3% at 16k.

• This test-time scaling experiment, which varies computation allowed during inference, shows task-dependent returns from additional thinking tokens. It measures aggregate benchmark accuracy rather than whether longer reasoning improves each individual answer.

Table 4 compares Kimi-VL-Thinking with non-thinking and other models on MathVision, MathVista, and MMMU. Its 36.8% MathVision, 71.3% MathVista, and 61.7% MMMU results improve on the base Kimi-VL results, though other models lead some table entries.

MathVision and MMMU improve at every plotted increase in the maximum thinking budget. MathVista reaches 70.9% at 4k and changes little afterward, including a small decline at 8k, showing task-dependent returns from additional inference computation.

5 Conclusion, Limitation, and Future Work

The authors attribute the model’s capabilities to joint text and multimodal training, sparse expert activation, native-resolution perception, and reasoning post-training. They identify three limitations: insufficient capacity for specialized or language-dependent problems, remaining difficulty with intricate multi-step reasoning, and limited attention-layer capacity for especially demanding long contexts despite the 128K window. Larger models, more pre-training data, and improved post-training and inference scaling are proposed future directions.

Appendix

• A Contributions separates Core Contributors from Contributors, marks Haoning Wu, Haoyu Lu, and Y. Charles as project leads, and states that names are alphabetized by first name. The affiliation note for authors marked # lists The University of Hong Kong and Moonshot.ai.
• B Evaluation Details describes B.1 Image Benchmark, B.2 Video and Long Document Benchmark, and B.3 Agent Benchmark. Examples include MMMU’s 11.5K questions, MathVision’s 3,040 competition problems, VideoMMMU’s 300 videos and 900 questions, and Video-MME’s 900 videos and 2,700 question-answer pairs. OCRBench is explicitly scored out of 1000, and Video-MME distinguishes evaluations with and without subtitles.
• The agent appendix distinguishes interface grounding from execution-based computer tasks: ScreenSpot Pro contains 1,581 high-resolution images with expert-annotated tasks, while OSWorld supports interactive tasks across Ubuntu, Windows, and macOS. WindowsAgentArena builds on OSWorld for realistic Windows environments. These descriptions explain the task families but do not provide a complete reproduction protocol for every reported model result.

Brief Thoughts

The clearest result is the thinking extension’s improvement over base Kimi-VL on MathVista, MMMU, and MathVision. Controlled ablations, experiments isolating individual design choices, are absent, so the report does not establish how much the encoder, optimizer, data mixture, or RL contributes to these gains. Deployment assessment also requires considering the 16B total parameter count, visual-input size, output-token budget, and low absolute computer-task success rates alongside the 2.8B activated-decoder figure. This article follows the 10 Apr 2025 version, arXiv:2504.07491v1. Its code and model release URL is https://github.com/MoonshotAI/Kimi-VL.