MiniMax-01: Scaling Foundation Models with Lightning Attention | Summary

Contents

Summary
1. Introduction
2. Model Architecture
3. Computation Optimization
4. Pre-Training
- 4.1. Data
- 4.2. Training Strategy
5. Post-training
6. Vision-language Model
7. Conclusion and Future work
Appendix
Brief Thoughts

This article explains the key points of MiniMax-01: Scaling Foundation Models with Lightning Attention.

2025-01-14 (arXiv)
MiniMax, Li, Aonian, Gong, Bangwei, Yang, Bo, Shan, Boji, Liu, Chang, Zhu, Cheng, Zhang, Chunhao, Guo, Congchao, Chen, Da, et al.
MiniMax
Paper
Github

Summary

Processing millions of input tokens is computationally expensive, and MiniMax-01 addresses this cost while maintaining general language-model capability. MiniMax-Text-01 achieves 0.910 average accuracy on RULER, a suite of 13 long-context tasks, compared with 0.850 for Gemini-1.5-Pro (002) when both are evaluated at 1M input tokens.
The model combines lightning attention, a memory-transfer-aware implementation of linear attention whose computation grows linearly with sequence length at fixed feature and block sizes, with one softmax attention layer after every seven lightning layers; softmax attention computes normalized weights over preceding tokens. A Mixture of Experts (MoE), which routes each token to selected feed-forward networks, provides 456 billion total parameters with 45.9 billion activated per token.
Training extends the context window, the input length available to the model, to 1M tokens, and a synthetic retrieval test demonstrates extrapolation to 4M tokens. Communication scheduling and specialized inference kernels, routines that execute model operations, support this scale, while MiniMax-VL-01 adds image processing for joint image and text understanding.
MiniMax-Text-01 scores 56.5 on LongBench v2, a long-context question-answering benchmark, with chain-of-thought, an explicitly generated reasoning process, versus 51.4 for GPT-4o (11-20) under the same evaluation mode. Results vary across tasks: coding remains weaker than several competitors, the 4M-token result concerns simplified retrieval, and the retained softmax layers remain costly at extreme lengths.

1. Introduction

The report targets applications that exceed the 32K–256K context windows described as common at the time, including entire books, programming projects, and learning from examples supplied in the input. Its deployment constraint is one machine with up to 8 graphics processing units (GPUs), 640GB memory, and 8-bit quantization, which reduces the numerical precision used to store model weights.

The contribution combines architecture, training data, distributed execution, and inference implementation; the report does not isolate attention as the sole cause of the final model’s performance.
Prefilling processes the input before subsequent token generation. The latency comparison runs MiniMax-Text-01 and Llama3-70B on H800 GPUs with tensor parallelism of 8, which partitions model computation across eight devices, and W8A16, meaning 8-bit weights and 16-bit activations; other models use their official application programming interfaces (APIs).

Prefilling latency across context lengths for locally hosted and API-accessed models.

2. Model Architecture

MiniMax-Text-01 has 80 layers, with seven lightning attention blocks followed by one softmax attention block in each repeating group. Each layer contains 32 experts with top-2 routing, which selects two experts per token, and uses DeepNorm, a normalization and residual-scaling scheme for stabilizing deep networks.

The hidden size is 6144, each expert’s feed-forward hidden dimension is 9216, and attention uses 64 heads, parallel attention components, with head dimension 128.
Softmax layers use Group Query Attention (GQA), which shares keys and values among groups of query heads, with group size 8. Rotary Position Embedding (RoPE), which encodes position through rotations within attention, covers half the head dimension and initially uses base frequency 10,000.

MiniMax-Text-01 blocks with lightning attention, softmax attention, normalization, and top-2 expert routing.

2.1. Mixture of Experts

MoE combines expert outputs using weights assigned by a learned router. Training assigns each expert a capacity limit and drops excess routed tokens; a global router synchronizes expert token counts across expert parallel (EP) groups, which distribute experts across devices, to reduce uneven loading and token drops.

A comparison trains a 7-billion-parameter dense model, which uses its full feed-forward network for each token, and a 24-billion-parameter MoE with 2 billion activated parameters on 1 trillion tokens. Across the five plotted benchmarks, the MoE reaches comparable performance with less computation.
The global router supplements an auxiliary load-balancing loss with an allgather operation, which collects token-count information from participating groups before dispatch.

2.2. Linear Attention

Linear attention changes the multiplication order from Norm((QKᵀ)V) to Norm(Q(KᵀV)), where Q, K, and V are query, key, and value matrices, and Norm denotes normalization. This avoids constructing a sequence-length-by-sequence-length matrix, but causal prediction, which permits access only to current and preceding tokens, introduces cumulative dependencies.

Lightning Attention divides the sequence into tiles: masked attention handles interactions within a tile, while an accumulated key-value state summarizes preceding tiles. Algorithm 1 computes both contributions, updates the state, and writes each output tile.
The forward-pass derivation omits normalization, sigmoid linear unit activation, an elementwise nonlinear transformation, and gating, which modulates feature contributions, for analytical simplicity. Its stated time complexity is \(O(nd^2+nBd)\), where n is sequence length, d is feature dimension, and B is tile size.

The effectiveness experiments compare softmax attention using FlashAttention-2, an optimized exact-attention implementation, pure lightning attention, and Hybrid-lightning, which inserts softmax every eighth layer. Model sizes are 70 million, 160 million, 410 million, 1 billion, 3 billion, and 7 billion parameters, with up to 300 billion training tokens, context length 8192, a global batch of 4 million tokens, and Adam, an adaptive optimizer, using learning rate 3e-4 and weight decay 0.1.

A fixed learning-rate scheduler is used because of resource constraints. Training loss serves as a proxy for test loss when fitting scaling laws, empirical relationships between compute, model size, data size, and loss.
Within the studied regime, the fitted compute-to-loss curves favor lightning and hybrid attention over pure softmax. Hybrid-lightning has the fitted relationship \(L(C)=3.4797C^{-0.0763}\), compared with \(3.7087C^{-0.0798}\) for softmax, where L is loss and C is compute budget; these fits do not establish performance at arbitrary scale.

The downstream experiments identify retrieval as the main weakness of pure lightning attention: general task scores are often comparable to softmax, but Needle in a Haystack (NIAH), which tests retrieval of inserted information, is substantially worse. Hybrid-lightning reaches NIAH scores of 95.7, 98.0, and 97.7 at 1B, 3B, and 7B parameters.

The speed experiment evaluates 3B models on a single H800 node while increasing sequence length toward the memory limit. Pure lightning attention maintains approximately stable tokens processed per GPU per second, whereas softmax throughput decreases with sequence length.
Hybrid-lightning throughput also declines with length because it retains softmax layers. The approximately constant-throughput observation applies to pure lightning attention within the tested range.

Downstream task performance of softmax, lightning, and Hybrid-lightning models at four parameter scales.

The hybrid comparisons include hybrid-cosformer2 and hybrid-hgrn2, alternative sequence models with periodic softmax layers, and hybrid-window models, which combine local sliding-window attention with full softmax every eighth layer. At 1B parameters, Hybrid-lightning provides 33.4K tokens per GPU per second and 95.7 NIAH, versus 29.5K and 91.8 for hybrid-hgrn2; hybrid-hgrn2 has the higher average commonsense-reasoning score, 49.87 versus 49.55.

At 3B parameters, Hybrid-lightning reaches 98.0 NIAH, compared with 40.9–57.9 for hybrid-window configurations with window sizes 256, 512, and 1024. It leads the aggregate commonsense and long-sequence scores, but the table does not support the surrounding claim that it wins every metric.
The discussion interprets the results through recurrent-state capacity, the size of the summary carried between tokens, and softmax attention’s repeated access to previous inputs. This is a proposed explanation of the experiments, rather than proof that all pure linear attention models inherently lack retrieval.

2.3. Module Ablations in MoE

The attention ablation trains MoE models with 28B total and 5B activated parameters on 1 trillion tokens. Hybrid-lightning improves six of eight reported metrics relative to softmax, including BIG-Bench Hard (BBH), a difficult reasoning benchmark, from 28.2 to 32.2 and MATH, a mathematical problem-solving benchmark, from 4.6 to 6.8; Chinese knowledge and grade-school mathematics scores decrease.

The normalization ablation uses 48-layer models with 60B total and 9.3B activated parameters, trained on 500B tokens. Post Layer Normalization, which normalizes after the residual operation, uses DeepNorm and improves seven of eight metrics over Pre Layer Normalization, which normalizes before the sublayer.
The table qualifies the surrounding claim of improvement across all metrics: WinoGrande, a commonsense benchmark, is 65.4 with post-normalization versus 65.5 with pre-normalization.

Attention and normalization ablations within MoE models across eight benchmarks.

2.4. Model Spec

Model selection minimizes predicted loss subject to a training-compute budget and fewer than 500B total parameters. The parameter limit is intended to support 1M-token inference on an 8×80G node under 8-bit quantization, while small-model experiments examine attention mixing, depth relative to width, linear-attention memory, activated feed-forward capacity relative to attention, and the fraction of dimensions using RoPE.

The authors report benefits from greater depth, larger linear-attention memory, and applying RoPE to half the softmax dimensions.
Scaling experiments cover 44 million to 1.2 billion activated parameters, 16, 32, and 64 experts, and 500 billion training tokens. Because earlier scaling formulas extrapolate poorly to 9.3B activated parameters, the authors add an interaction term between activated parameters and training tokens before selecting the 456B-total, 45.9B-active configuration.

3. Computation Optimization

Training uses a changing cluster of 1500–2500 H800 GPUs. Computation optimization addresses expert communication and weight storage, distribution of million-token sequences, and inference batches containing different input lengths and cached prefixes, which reuse computation from earlier input segments.

3.1. MoE Optimization

The implementation separates Expert Tensor Parallel (ETP), which partitions expert weights, from Expert Data Parallel (EDP), which replicates identical experts across data partitions. This lets expert components use parallel configurations independent of the rest of the network and allows expert communication to overlap with token-group computation.

The EP-ETP schedule balances expert weight memory, computation intensity, and communication. Longer computation intervals hide more communication, while too few token groups leave idle intervals and too many increase scheduling overhead.
The authors report a 50% reduction in pure MoE communication overhead relative to their pre-optimization implementation. This percentage does not describe the reduction in total training time.

3.2. Long Context Optimization

Data-packing concatenates variable-length samples to reduce padding, extra positions added to meet computational shape requirements. Varlen Ring Attention distributes packed variable-length softmax attention across devices and tracks each sample’s mask offsets so computation respects sample boundaries without extensive per-sample padding.

Linear Attention Sequence Parallelism (LASP) originally passes intermediate key-value states sequentially between context-parallel ranks, processes assigned different sequence partitions. LASP+ computes local prefix summaries, synchronizes them through AllGather, and constructs the required global prefixes.
LASP+ removes the serial inter-rank dependency at the cost of greater communication volume and temporary memory. Its variable-length implementation pads each sample to a multiple of 256 before concatenation; the statement that computation speed reaches 1/Npcn of LASP is ambiguously worded and does not establish a clear measured speedup factor.

Computation and communication schedules for LASP and LASP+.

3.3. Lightning Attention Inference Optimization

Batched Kernel Fusion combines operations limited by memory access and reports a 10% end-to-end latency reduction for decoding, the generation of subsequent tokens, and short-text inputs. Separated Prefill and Decoding Execution assigns single-token generation and longer input processing to distinct kernels on separate CUDA streams, queues that can schedule GPU work concurrently.

In the stated example, a batch of 20 cached inputs contains one or two 50-token inputs and otherwise single-token inputs. Separating execution reduces latency from 100 milliseconds to approximately 50 milliseconds.
Multi-level Padding adds block sizes 32, 64, and 128 alongside 256 to reduce redundant work on short inputs. StridedBatchedMatmul Extension uses cublasGemmStridedBatchedEx for batched matrix multiplication; further Hopper-specific fusion is described partly as ongoing work.
The authors report more than 75% Model Flops Utilization (MFU), model floating-point computation relative to hardware peak capacity, for end-to-end H20 inference. At 1,024,000 tokens, their attention-versus-feed-forward latency accounting assigns 95% to softmax attention, versus less than 12% for lightning attention in the corresponding comparison.

4. Pre-Training

Pre-training combines corpus construction, controlled data experiments, and a staged training schedule for MiniMax-Text-01. Context extension trains on lengths up to 1M tokens; the reported 4M-token capability is an inference extrapolation result.

4.1. Data

The corpus combines academic literature, books, web content, and programming code, with rule-based cleaning, deduplication, and document scoring by a previous model with 60B total and 5B activated parameters. Sampling emphasizes knowledge depth and practical helpfulness while retaining category diversity, because removing all lower-scoring data harms downstream performance in the authors’ experiments.

Dialogue and question-answering data use a nested document format with varied templates to limit rigid formatting patterns. Tokenization uses byte-level Byte Pair Encoding (BPE), which builds tokens by merging byte sequences, with a 200K-token vocabulary and multilingual upsampling.
Controlled data experiments use MoE models with 8B total and 1B activated parameters, trained on 40B tokens split between 20B web tokens and 20B tokens from the candidate configuration. Byte-normalized answer likelihoods support hypothesis tests designed for 95% confidence and 80% power, the probability of detecting the specified effect when it is present.
Repetition-aware experiments align repetition rates with the eventual training schedule after global deduplication. The authors report deterioration for low-quality data beyond two epochs, passes through the data, while high-quality data remains useful for up to four epochs.

4.2. Training Strategy

Initial pre-training uses sequence length 8192, Xavier initialization, which sets initial weight scales, and AdamW, an adaptive optimizer with decoupled weight decay, with β₁ = 0.9, β₂ = 0.95, and weight decay 0.1. Batch size increases from 16M tokens to 32M at 69B training tokens, 64M at 790B, and 128M at 4.7T, following a fitted relationship between loss and critical batch size, the batch scale used to balance training time and data efficiency.

The learning rate warms up for 500 iterations to 2×10⁻⁴, remains constant for 7.2T tokens, then decreases to 1.3×10⁻⁴ for 3.2T tokens after abnormal gradient norms. A final 1T-token phase exponentially decays it to 3×10⁻⁵; the MoE auxiliary-loss coefficient is 0.01.
Long-context extension proceeds through 128K, 512K, and 1M training lengths using 300B, 32B, and 26B tokens, respectively. RoPE frequencies are 5M, 10M, and 10M; short/medium/long data proportions are 30/70/0, 35/35/30, and 30/30/40, with short below 32K tokens, medium from 32K to 128K, and long above 128K.
During the last 20% of each extension stage, 10% high-quality long-context question-answering data is mixed in, and source sampling weights transition by linear interpolation. The 4M-token NIAH test uses Paul Graham essays as background and an inserted magic-number sentence; the authors recognize that this test saturates too early to monitor broader long-context learning.

Vanilla NIAH retrieval across insertion depths and context lengths up to 4M tokens.

Long-context training lengths, RoPE frequencies, token budgets, and data-length mixtures.

5. Post-training

Post-training combines Supervised Fine-Tuning (SFT), which learns from selected example responses, offline preference optimization, and online Reinforcement Learning (RL), which updates the model using rewards for sampled responses. A hierarchy of correctness, truthfulness, helpfulness, and harmlessness guides response selection and optimization while short- and long-context stages alternate.

5.1. Prompt Collection

The prompt collection contains millions of queries tagged by task, knowledge domain, and difficulty. Filtering removes redundancy while retaining varied difficulty across long-context tasks, programming, mathematics, logical reasoning, creative writing, function calling, requests to invoke external functions, general knowledge, and safety.

5.2. Reward Model

The reward framework evaluates four response properties. Correctness uses answer consistency for mathematics and reasoning and test-case success for programs; truthfulness uses statement decomposition, clustering, human verification, and model comparisons.

Helpfulness combines automatic constraint checking with human assessments of coherence, depth, relevance, and style.
Harmlessness uses safety rules and model evaluations calibrated against human annotations, drawing on Constitutional AI, an approach that evaluates responses against stated principles.

5.3. Supervised Fine-Tuning

Domain-specific expert models generate candidate SFT responses after iterative supervised and reinforcement training. Rejection sampling retains candidates according to evaluated quality across multiple sampling temperatures, settings that control response variability, while n-gram filters for repeated token sequences and semantic similarity filters reduce duplication.

5.4. Reinforcement Learning

Offline Reinforcement Learning uses Direct Preference Optimization (DPO), which trains on preferred and dispreferred response pairs, with the best and worst sampled responses selected by the reward hierarchy. The authors reuse SFT prompts after finding negligible differences from previously unused prompts drawn from a similar distribution.

Online Reinforcement Learning uses prompts absent from SFT and favors diverse prompts with moderate success rates. Reusing earlier prompts reportedly causes saturation, indicated by reduced response perplexity, a measure of uncertainty in predicted text.
The modified Group Relative Policy Optimization (GRPO), a policy-update method using relative response rewards, adds clipping of importance weights, ratios used to weight policy updates, for tokens with large ratios and negative estimated advantage, meaning worse-than-baseline outcomes. It also reformulates the Kullback–Leibler divergence term, a measure of deviation from a reference policy, and balances positive and negative reward contributions.
These changes target training stability, but the section provides no separate numerical ablations establishing the contribution of each modification.

5.5. Safety Alignment

Safety Alignment constructs category-specific prompts, collects safety-related questions from web documents, and augments adversarial prompts with an earlier MiniMax-Text-01 model. A harmless reward model combines detailed safety rules with helpfulness criteria to reduce inappropriate responses and unnecessary refusals.

The section describes the training procedure and its intended behavior; it does not establish protection against all forms of misuse.

5.6. Training Methodology with Long-Context Adaptation

The five-stage long-context adaptation schedule alternates SFT and DPO between sequence lengths 8,192 and 1,032,192, then finishes with online RL at 8,192. RoPE base frequency remains 10 million throughout post-training.

Stage I uses short-context SFT; Stage II extends SFT with 50% long-context prompts. Stage III applies short-context DPO, Stage IV applies DPO entirely to long-context data, and Stage V performs short-context online RL.
The five stages use 2, 2, 1, 1, and 1 epochs, with batch sizes 128, 80, 64, 64, and 512. Maximum learning rates are 1e-5, 3e-6, 5e-7, 5e-7, and 1e-6.

5.7. Academic Benchmarks

Core benchmark evaluation uses greedy decoding, which chooses the highest-probability next token, and zero-shot chain-of-thought, which requests reasoning without worked examples, on designated tasks. MiniMax-Text-01 reaches 88.5 on Massive Multitask Language Understanding (MMLU), a broad knowledge benchmark, and leads the listed models on C-SimpleQA, a Chinese factuality benchmark, with 67.4.

Performance is uneven: MATH is 77.4 versus 76.6 for GPT-4o (11-20), but MBPP+, a program-generation benchmark, is 71.7 versus 78.8 for DeepSeek-V3. HumanEval, another code-generation benchmark, is 86.9 versus 93.7 for Claude-3.5-Sonnet (10-22).
The detailed table reports 89.1 for IFEval, an instruction-following benchmark, while the introductory figure labels it 89.0. The detailed table is used here.

Core academic benchmark results for MiniMax-Text-01 and comparison models.

Long-context evaluation separates retrieval, understanding, and learning from supplied context. Long-Context Retrieval introduces Multi-Round Needles-In-A-Haystack (MR-NIAH), which asks the model to repeat a particular earlier response within a synthetic dialogue of 2K–1M tokens.

Target interactions occur at 25%, 50%, and 75% of the dialogue. Each target response has three core components, and adjusted recall counts correctly recovered components divided by three.
The English and Chinese curves show less degradation at long lengths than the compared models, but recall still declines toward 1M tokens. This harder dialogue retrieval task exposes errors that the nearly saturated vanilla NIAH test misses.

Long-Context Understanding evaluates RULER and LongBench v2, whose questions cover documents, dialogues, code repositories, and structured data. MiniMax-Text-01 leads the listed RULER models from 128K onward, with 0.947 at 128K, 0.928 at 512K, and 0.910 at 1M.

At 1M, Gemini-1.5-Pro (002) scores 0.850. Missing entries for other models indicate absent results at those lengths.
LongBench v2 overall scores are 56.5 with chain-of-thought and 52.9 without, compared with GPT-4o (11-20) at 51.4 and 50.1. MiniMax does not lead every subset: without chain-of-thought, Qwen2.5-72B-Inst. scores 44.4 on the long subset versus 43.5 for MiniMax; comparison-model results are taken from the benchmark’s reported results.
LongBench v2 length categories use words rather than tokens: short is 0–32K words, medium 32K–128K, and long 128K–2M.

RULER scores across input lengths from 4k to 1M tokens.

LongBench v2 overall and subset results with and without chain-of-thought.

Long In-Context Learning, learning a task from information supplied in the input, uses Machine Translation from One Book (MTOB). Models translate English and Kalamang using a grammar book, a word list, and 375 examples, with approximately 81K tokens for half-book context and 133K for full-book context.

For English-to-Kalamang, the reported ChrF translation-quality score rises from 6.0 without context to 51.74 with half a book and 51.60 with the full book. The reported gains, 45.7 and 45.6, lead the table, but absolute scores do not: Gemini-1.5-Pro (002) reaches 57.90 with the full book.
For Kalamang-to-English, the reported BLEURT translation-evaluation score is 58.00 with the full book, below Gemini-1.5-Pro (002) at 63.09. Figure 16 shows improvement during long-context extension; the paper’s suggestion that competitors had greater Kalamang training exposure is not established by the comparison.

MTOB translation scores without context, with half-book context, and with full-book context.

5.8. User-in-the-loop

In-house evaluation combines automatic assessments with expert human evaluation using queries drawn largely from Hailuo AI, the authors’ chatbot service, with substantial Mandarin and Chinese-cultural content. MiniMax-Text-01 leads the listed models in general assistant performance at 73.9, creative writing at 81.3, knowledge question answering at 78.6, and long context at 93.8, but its instruction-following score of 46.3 trails several competitors.

Search in Hailuo AI trains direct search decisions through special tokens for scenarios estimated to cover approximately 30–40% of user queries. Human evaluation reports an increase from 58% to 71.5% on an out-of-domain end-to-end evaluation.
The search result is a within-system comparison. The authors avoid cross-assistant comparisons because comparable search-tool availability is uncertain; the internal benchmark’s language distribution also limits generalization to other user populations.

6. Vision-language Model

MiniMax-VL-01 is a vision-language model, a model that processes images together with text, extending MiniMax-Text-01 with an image encoder and adapter. The encoder converts images to representations, and the adapter connects those representations to the language model; training first aligns the visual components and subsequently updates the full pipeline.

6.1. Multimodal Data

Caption Data contains 694 million unique image-caption pairs, including refined captions for 180 million images; raw and refined captions are sampled with equal probability, p = 0.5, where refined captions are available. Description Data contains 100 million images with model-generated, human-refined descriptions averaging approximately 300 text tokens per image.

Instruction Data synthesizes visual question-answer pairs for tasks including text extraction, object localization, identifying object positions, and geometry, with balanced sampling across prompt types.
Data Distribution analyzes 1 million sampled image-instruction pairs using another vision-language model. Approximately 50,000 unique tags are produced, with 2,817 appearing more than 10 times and grouped into 14 major categories; these are model-assigned descriptions of the sampled tasks.

6.2. Architecture

The architecture combines a 303-million-parameter Vision Transformer (ViT), which processes images as patch representations, a two-layer multilayer perceptron (MLP) projector, which maps visual features to the language model, and MiniMax-Text-01. Dynamic image resolution ranges from 336×336 to 2016×2016, with non-overlapping 336×336 crops and a separate 336×336 thumbnail independently encoded and concatenated without feature downsampling.

The vision encoder trains ViT-L/14 from scratch using CoCa, an approach combining matching-versus-mismatching image-text contrastive learning with caption-generation training.
The encoder processes 37 billion image-caption pairs at 224×224 and then 1.2 billion pairs at 336×336, with captions truncated to 76 tokens. It reaches 80.55% zero-shot classification accuracy on ImageNet-1K, a 1,000-class image benchmark, without task-specific supervised adaptation.

6.3. Training Recipes

The four-stage visual training recipe starts with Modality alignment using 80 billion description tokens at 336×336 while updating the encoder and adapter. Enhancement of Vision Understanding updates all parameters using 420 billion multimodal tokens, tokens representing image and text inputs, combined with text post-training data at a 20:1 ratio.

Enhancement of User Experience trains for one epoch on 44.8 billion multimodal tokens from application-oriented, human-labeled conversations.
Enhancement of Preference uses DPO on 40,000 image-text pairs plus text-only pairs, selecting responses with substantial evaluation-score differences. Candidate construction varies sampling temperature, weakens images, and deliberately introduces response errors; training stops before one full epoch to limit overfitting.
The abstract reports 512 billion vision-language tokens, while the stated quantities for the first three stages sum to 544.8 billion. The report does not reconcile these figures.

6.4. Benchmarks

MiniMax-VL-01 leads the listed models on ChartQA, a chart question-answering benchmark, at 91.7 and OCRBench, which evaluates optical character recognition or reading text from images, at 865. It scores 96.4 on DocVQA, which asks questions about document images, but OlympiadBench, a benchmark of advanced mathematical and scientific problems, is 24.2 versus 46.1 for Gemini-2.0-Flash (exp).

On MMLongBench-Doc, a long-document understanding benchmark, MiniMax scores 32.5 versus 41.4 for GPT-4o (11-20). Its single-page and cross-page subset accuracies are 47.3% and 28.4%, respectively.
MEGA-Bench, a broad multimodal evaluation suite, gives MiniMax a macro score, an average across evaluation tasks, of 47.4, above the listed open-source alternatives but below several commercial models. The paper identifies planning and metric assessment as weaknesses.
The internal visual benchmark covers 90 tasks and 524 annotated Chinese and English samples, primarily Chinese, with training-set deduplication and per-sample checklists. MiniMax scores 56.6, compared with 62.3 for GPT-4o (11-20) and 72.1 for Gemini-2.0-Flash (exp).

MiniMax-VL-01 results on academic and internal multimodal evaluations.

7. Conclusion and Future work

The report presents a large hybrid attention model with competitive general evaluations and its clearest advantages in long-context text processing. Models are released at https://github.com/MiniMax-AI, with https://www.hailuo.ai/ identified for the search-enabled chatbot and https://intl.minimaxi.com for the API.

The authors acknowledge that long-context evaluations remain artificial or simplified, one eighth of layers still use softmax attention, and complex programming performance needs improvement.
Removing softmax entirely and improving realistic long-context reasoning are proposed research directions. Unlimited context and elimination of the remaining attention overhead are not demonstrated.

Appendix

A. Contributors lists the report’s contributors alphabetically.
B. MiniMax-Text-01 Case Demonstrations presents benchmark examples and evaluator-written requests intended to resemble user interactions while protecting user privacy. B.1. Learning A ‘New’ Language From Long Context illustrates Kalamang translation from a grammar book, word list, and parallel sentences. B.2. Memorizing Long History Dialogues explains retrieval from up to 1889 English or 2053 Chinese history interactions; B.3. Summarizing Long Papers with Figures and B.4. Translating Long Papers show document-processing examples. B.5. Lyrics Writing, B.6. Story Writing, and B.7. Poetry Composition provide creative outputs and human commentary, B.8. Safety shows a benign response to a restriction-removal prompt, and B.9. Search contrasts accepting a false premise with correcting it using search. These selected cases illustrate behavior without establishing aggregate reliability.
C. Vision In-house User Experience Benchmark Examples provides images, requests, checklists, and responses. C.1. Nature tests interpreting an illustration of selected Solar System moons; C.2. Daily Life covers navigation and clothing counts; C.3. Digital Scenarios covers workflow bottlenecks and email-based scheduling; C.4. Special Imaging compares satellite images across dates. The clothing example’s final checklist bullet repeats “second row” where the displayed response describes the third row.
D. Evaluation Details for Selected Benchmarks documents differing input configurations. D.1. MMLongBench-Doc concatenates document pages into 5 images for evaluated open-source models and 10 for Claude-3.5-Sonnet-1022, while describing a default maximum of 120 pages and resolution 144 for other commercial models and the named MiniMax-Text-01; that text-model identifier is an apparent inconsistency in the visual evaluation description. D.2. MEGA-Bench allows MiniMax-VL-01 128 images and 16 demonstration video frames, compared with 64/8 for GPT-4o-2024-1120 and 10/1 for Qwen2-VL-72B-Instruct. D.3. MMMU & DocVQA uses GPT-4o-2024-05-13 as a judge when rule-based evaluation fails to recognize alternative valid answer forms; MMMU evaluates multidisciplinary multimodal understanding.

Brief Thoughts

The strongest contribution is the combination of a hybrid architecture that preserves retrieval with the systems work needed to train and serve it at 456B parameters. The attention ablations, RULER results, and LongBench v2 comparisons support this assessment more directly than the 4M-token window alone. Evidence for reliable million-token reasoning remains limited: vanilla retrieval saturates early, harder dialogue recall declines, and the authors acknowledge gaps in realistic evaluation. Deployment expectations should account for the retained softmax cost and the differing multimodal input limits used in comparisons. The results establish specific long-context gains, with weaker coding, cross-page understanding, and advanced visual mathematics.