End-to-End Vision Tokenizer Tuning | Summary

Contents

Summary
1 Introduction
2 Related Work
3 Methodology
4 Experimental Results
5 Conclusion
6 Limitations and Future Topics
Appendix
Brief Thoughts

This article explains the key points of End-to-End Vision Tokenizer Tuning.

2025-05-15 (arXiv)
Wang, Wenxuan, Zhang, Fan, Cui, Yufeng, Diao, Haiwen, Luo, Zhuoyan, Lu, Huchuan, Liu, Jing, Wang, Xinlong.
Institute of Automation, Chinese Academy of Sciences, School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing Academy of Artificial Intelligence, Dalian University of Technology, Tsinghua University
Paper

Summary

A vision tokenizer compresses images into discrete representations, but training it only to reconstruct images can discard information needed for image understanding and generation. Against the frozen discrete-index baseline in the controlled ablation, end-to-end vision tokenizer tuning (ETT) raises SEEDBench-Img, an image understanding benchmark, from 54.8 to 60.0, TextVQA, a benchmark requiring reading text in images, from 40.1 to 46.9, and GenEval, a text-to-image alignment benchmark, from 0.40 to 0.43.
ETT feeds codebook embeddings, the learned vectors associated with discrete visual codes, through a projector that maps them into a large language model’s (LLM’s) input space. Caption prediction and image reconstruction jointly update the tokenizer and LLM during semantic learning; subsequent post-training freezes the tokenizer and produces separate ETT-Chat and ETT-Gen specialists for understanding and generation.
The full system uses a 1.5B-parameter language backbone and reaches 0.63 on GenEval, compared with 0.55 for SDXL, a diffusion-based image generator, and 0.66 for Emu3, a model that predicts discrete multimodal tokens. Selected reconstructions look similar before and after tuning, but reconstruction Fréchet Inception Distance (rFID), a reconstruction quality metric for which lower is better, worsens from 1.033 to 1.648 at the default loss weight in the controlled ablation.

1 Introduction

Autoregressive modeling predicts each token from preceding tokens, allowing image and text sequences to share a next-token prediction framework. The paper argues that separately training a tokenizer for pixel reconstruction can discard information needed for downstream semantics, especially text recognition and rendering. Conventional pipelines also learn new visual embeddings from discrete indices while leaving the tokenizer frozen, separating downstream feedback from visual representation learning.

Vector quantization (VQ) maps continuous image features to entries in a finite codebook; this mapping can create a representation bottleneck.
ETT uses the tokenizer’s existing codebook vectors and caption supervision to connect downstream learning to the tokenizer without changing the LLM’s original text vocabulary.

Frozen discrete-index training and ETT with projected codebook embeddings, joint objectives, and downstream benchmark gains.

The related work distinguishes improvements to image reconstruction and quantization from methods that add semantic information to visual tokens. VQ-VAE, an image autoencoder with a quantizer, maps continuous features into discrete representations, while VQGAN adds perceptual and adversarial objectives that respectively compare visual features and train against a discriminator. ETT adopts the IBQ tokenizer framework, which supports large codebooks by updating the entire codebook at each training step.

For visual generation, prior systems use sequential token prediction or masked token modeling, which predicts hidden visual tokens.
Emu3 and other unified systems use frozen discrete tokenizers, while Janus, a model for both understanding and generation, separates visual encoding for the two tasks. ETT studies joint optimization of the existing tokenizer and autoregressive model.

3 Methodology

The method combines an IBQ-based image tokenizer, a lightweight visual projector, and a pretrained LLM. It supplies quantized codebook embeddings to the LLM and jointly optimizes caption prediction and reconstruction before task-specific post-training.

3.1 Vision Tokenizer

The tokenizer encodes an image into a spatial feature map, assigns each feature to its nearest codebook vector, and decodes the resulting quantized embeddings back into an image. ETT uses downsampling factor s = 16, embedding dimension D = 256, and codebook size 131,072, reduced from the cited IBQ configuration of 262,144 entries.

The tokenizer objective is \(\mathcal{L}_{vq}=\mathcal{L}_{rec}+\mathcal{L}_{quant}+\mathcal{L}_{lpips}+\lambda_G\cdot\mathcal{L}_{GAN}+\lambda_E\cdot\mathcal{L}_{entropy}\).
These terms measure pixel reconstruction, agreement between encoded and quantized features, perceptual similarity using LPIPS, a feature-based image comparison metric, adversarial realism using PatchGAN, an image-patch discriminator, and codebook entropy, respectively.

3.2 End-to-End Vision Tokenizer Tuning

ETT replaces the conventional use of discrete visual indices as LLM inputs with the corresponding tokenizer codebook embeddings. A multilayer perceptron, a small neural projection network using the GeLU activation function, maps each 256-dimensional visual embedding into the LLM’s hidden dimension. The pretrained text tokenizer and text embedding layer continue to process text.

This connection allows caption supervision to update the visual tokenizer through the training computation graph described by the paper.
The projector avoids learning the LLM’s visual input representations solely from randomly initialized embeddings associated with discrete indices.

Caption supervision uses cross-entropy, the negative log probability of each target text token conditioned on the image and preceding text. The joint objective is \(\mathcal{L}=\mathcal{L}_{cap}+\alpha\cdot\mathcal{L}_{vq}\), where α controls the balance between semantic supervision and reconstruction. Updating the tokenizer encoder and decoder together with the LLM is intended to improve understanding while limiting reconstruction degradation.

The caption objective is L_cap = −Σ_{t=1}^{N} log P(x_t^T | x^I, x_<t^T), where N is the text sequence length, x^I represents image embeddings, and x_<t^T represents preceding text embeddings.
The default reconstruction weight is α = 0.25, selected from the reconstruction and understanding ablation.

3.3 Training Recipe for Multimodal Generation and Understanding

Stage 1, alignment learning, freezes the pretrained tokenizer and LLM and trains only the visual projector with caption loss. It uses a 12M-image subset of SOL-recap, the authors’ collection of 32M recaptioned image-text pairs drawn from the public image datasets SA-1B, OpenImages, and LAION.

Stage 2, semantic learning, unfreezes the tokenizer, projector, and LLM. Joint caption and reconstruction training on 12M SOL-recap pairs implements end-to-end tokenizer tuning.

Stage 3, post-training, freezes the tuned tokenizer and trains the projector and LLM separately for ETT-Chat and ETT-Gen. Chat training combines 32.0M SOL-recap pairs with instruction datasets containing 3.5M LLaVA-OneVision examples and 31.8M Infinity-MM examples, totaling 67.3M. Generation training uses 14.0M samples generated with the Flux image model and 16.0M high-aesthetics web pairs, totaling 30.0M.

Datasets and example counts for alignment learning, semantic learning, and Chat and generation post-training.

4 Experimental Results

The experiments assess multimodal understanding, text-to-image generation, and the contribution of tokenizer tuning under controlled ablations. The full-system comparisons use different model sizes and training datasets, while the ablations provide a closer test of the embedding interface and joint objective.

4.1 Training Settings

All experiments use Qwen2.5-1.5B as the language backbone. The paper reports training on 8-A100 nodes with the Adam optimizer, batch size 1024 in each of the three stages, maximum learning rates of 4 × 10⁻⁵, 4 × 10⁻⁵, and 2 × 10⁻⁵, a warm-up ratio of 0.03, and cosine learning-rate decay. Unless specified otherwise, images are processed at 512 × 512; understanding ablations use LLaVA-mix-665K, a visual instruction dataset, for Stage 3.

The 32M-image recaptioning collection contains 11M images from SA-1B, 7M from OpenImages, and 14M from LAION.
Generation data are filtered by image resolution and aesthetic scores; numerical filtering thresholds are not reported.

Tokenizer training uses Adam with fixed learning rate 1 × 10⁻⁴, optimizer coefficients β₁ = 0.5 and β₂ = 0.9, and 500,000 steps, with global batch size 256 and 256 × 256 inputs. The adversarial and entropy loss weights are λ_G = 0.1 and λ_E = 0.05. LeCAM, a discriminator regularization method, is used to improve training stability.

4.2 Multimodal Understanding Evaluation

The understanding evaluation covers GQA for compositional visual question answering, POPE for detecting object hallucinations, meaning claims about objects absent from an image, ScienceQA-IMG for image-based scientific reasoning, and general multimodal benchmarks. ETT scores 58.8 on MMBench-EN, the English MMBench evaluation, 66.6 on SEEDBench-Img, 29.3 on MMVet, an integrated multimodal capability benchmark, 1532.6 on MME, a multimodal evaluation benchmark, 82.4 on POPE, 59.4 on GQA, 91.7 on ScienceQA-IMG, and 56.8 on TextVQA.

ETT exceeds Liquid, a discrete-token multimodal model, on all four jointly reported metrics: MME, POPE, GQA, and TextVQA. Against Emu3, it is slightly higher on MMBench-EN but lower on SEEDBench-Img, MMVet, POPE, GQA, and TextVQA.
The 91.7 ScienceQA-IMG result is marked with an asterisk because images from related training datasets were observed during training; Emu3’s 89.2 carries the same qualification.
Table 2 reports ETT data as 32M pre-training and 35.3M fine-tuning examples. Table 1 reports a 67.3M Stage 3 Chat mixture that also includes 32M SOL-recap pairs; these totals account for SOL-recap differently.

Multimodal understanding results with language-model sizes, training data volumes, and dataset-exposure qualifications.

4.3 Visual Generation Evaluation

With top-k = 131,072 and top-p = 1.0, sampling settings that retain the full visual vocabulary, ETT-Gen reaches 0.63 overall on GenEval using a 1.5B language backbone and 30M generation examples. Table 3 reports 0.55 for SDXL, 0.62 for SD3, another diffusion-based generator, 0.66 for Emu3, and 0.67 for DALL-E3, a text-to-image generator; these comparisons use differing model and data conditions.

ETT’s GenEval scores are 0.98 for single objects, 0.81 for two objects, 0.25 for counting, 0.84 for colors, 0.48 for position, and 0.45 for color attributes. Counting is a weakness relative to several compared systems.
On T2I-CompBench, which evaluates compositional text-to-image generation, ETT scores 81.03 for color, 58.19 for shape, and 72.14 for texture. DALL-E3 scores 81.10, 67.50, and 80.70, respectively.
The text discusses prompt rewriting, meaning revising the input description before generation, but presents no separate numerical ablation isolating its effect. The qualitative examples show 512 × 512 outputs with simplified displayed prompts.

Text-to-image generation comparisons on GenEval and T2I-CompBench, including category scores and reported model and data scales.

Selected 512 × 512 ETT generations across subjects and styles, accompanied by simplified prompts.

4.4 Ablation Study

The controlled comparison trains understanding and generation models with SOL-recap and adds LLaVA-mix-665K supervised fine-tuning for understanding. Replacing frozen discrete indices with frozen codebook embeddings changes GQA from 50.9 to 52.6 and TextVQA from 40.1 to 40.9, but reduces GenEval from 0.40 to 0.34. Adding end-to-end tuning raises SEEDBench-Img to 60.0, GQA to 54.8, TextVQA to 46.9, MME-Perception, the perception subset of MME, to 1124.7, and GenEval to 0.43.

Relative to the discrete-index baseline, the gains are 5.2 points on SEEDBench-Img, 3.9 points on GQA, 6.8 points on TextVQA, and 0.03 on GenEval.
The frozen-embedding result shows that the input interface alone does not explain the generation improvement: joint tuning recovers and exceeds the baseline in this experiment.

Controlled comparison of frozen discrete indices, frozen codebook embeddings, and end-to-end tuned embeddings.

The reconstruction ablation uses ImageNet, an image dataset, at 256 × 256 and reports rFID. Understanding-only tuning gives the highest understanding scores but worsens rFID from 1.033 to 45.701; adding reconstruction loss with α = 0.25 improves it to 1.648 while retaining most understanding gains. Increasing α to 1.0 gives rFID 1.500 but lowers SEEDBench-Img to 59.3 and TextVQA to 45.7.

At α = 0.25, SEEDBench-Img is 60.0 and TextVQA is 46.9, compared with 61.2 and 48.0 under understanding-only tuning.
The intermediate α = 0.5 configuration gives rFID 1.655, so reconstruction quality does not improve monotonically across the tested weights.
Figure 3 shows similar reconstructed appearance before and after tuning, with selected text details improved. These examples complement the metric but do not establish unchanged reconstruction quality.

Understanding and ImageNet reconstruction results under different tokenizer tuning objectives and reconstruction weights.

Image reconstructions before and after ETT alongside their inputs, including natural scenes and text-bearing images.

5 Conclusion

The conclusion attributes ETT’s gains to using tokenizer codebook embeddings and token-level caption supervision to optimize visual representations jointly with downstream learning. The experiments support improvements in both understanding and generation over the frozen-tokenizer ablation. The auxiliary reconstruction objective limits degradation, although rFID remains worse than the untuned baseline.

6 Limitations and Future Topics

The authors identify model capacity and end-to-end training data scale as areas for further study. ETT fine-tunes an existing tokenizer rather than learning a tokenizer from scratch specifically for understanding and generation. Larger-scale training, tokenizer training from scratch, and extension to video and audio are proposed future directions, with no results for those settings in this paper.

Appendix

The supplied 13-page paper contains no appendix. Pages 10–13 contain References; no additional experimental details or supplementary results follow them.

Brief Thoughts

The strongest evidence is the controlled ablation: embeddings without tuning hurt generation, whereas joint tuning improves both generation and understanding. This supports the practical benefit of downstream supervision reaching the tokenizer more directly than comparisons against systems with different datasets and parameter counts. The preservation claim needs qualification: α = 0.25 prevents the severe reconstruction degradation of understanding-only tuning, but rFID still increases from 1.033 to 1.648. Generality remains untested beyond the reported Qwen2.5-1.5B backbone and IBQ-based tokenizer. The paper reports no repeated-run uncertainty estimates, and Stage 3 produces separate specialists rather than demonstrating one final checkpoint for both tasks.