Gemini Embedding: Generalizable Embeddings from Gemini | Summary

Contents

Summary
1. Introduction
2. Related Work
3. Gemini Embedding
4. Datasets
- 4.1. Training Data Mixture
- 4.2. Improving Data Quality with Gemini
5. Evaluation
6. Ablation Study
- 6.1. Does Gemini Embedding Generalize to Multilingual Tasks?
- 6.2. How Does Gemini Improve Data Quality?
7. Future Work
8. Conclusion
Appendix
Brief Thoughts

This article explains the key points of Gemini Embedding: Generalizable Embeddings from Gemini.

2025-03-10 (arXiv)
Lee, Jinhyuk, Chen, Feiyang, Dua, Sahil, Cer, Daniel, Shanbhogue, Madhuri, Naim, Iftekhar, Ábrego, Gustavo Hernández, Li, Zhe, Chen, Kaifeng, Vera, Henrique Schechter, et al.
Gemini Embedding Team, Google
Paper
Project Page

Summary

The paper addresses how to build one text embedding model, which maps text into reusable numerical vectors, that works across languages, code, and tasks. On the March 10th, 2025 multilingual evaluation in the Massive Multilingual Text Embedding Benchmark (MMTEB), labeled MTEB(Multilingual), Gemini Embedding achieves a task mean of 68.32 versus 63.23 for multilingual-e5-large-instruct, the highest competing task mean reported, and ranks first on the public leaderboard.
The model combines initialization from Gemini, a pretrained large language model, with contrastive learning, which distinguishes matching query–target pairs from nonmatching pairs, and Gemini-assisted data generation and filtering. Training uses pre-finetuning, an initial adaptation from text generation to vector encoding, followed by task-specific fine-tuning and Model Soup, which averages parameters from selected saved model states called checkpoints. The full vectors have 3,072 dimensions, with training also supporting their first 768 and 1,536 dimensions.
Synthetic classification data raises the four-task average from 57.57 to 75.17, while filtering multilingual retrieval data raises its average from 59.8 to 63.7. Training-mixture comparisons show a tradeoff: multilingual retrieval-only fine-tuning performs better on XTREME-UP, a benchmark for retrieving English passages from queries in underrepresented languages, while the final model performs better across the broader task collections. The report leaves model size, training compute, and parts of the training configuration undisclosed.

1. Introduction

The goal is to produce representations that can be computed in advance and reused for retrieval, finding relevant passages; classification, assigning labels; clustering, grouping related texts; and similarity scoring. Gemini supplies both the initial model parameters and training-data judgments, including generated examples and assessments of relevant or irrelevant passages. Table 1 summarizes multilingual, English, code, and cross-lingual retrieval results, where queries and passages use different languages.

MMTEB provides the principal evaluation framework and covers 250+ languages. Its multilingual task-type mean, which weights task categories equally, is 59.64 for Gemini Embedding versus 56.00 for gte-Qwen2-7B-instruct, a gain of 3.64 points.
The paper links model access at https://ai.google.dev/gemini-api/docs/embeddings and identifies https://huggingface.co/spaces/mteb/leaderboard as the public leaderboard used for its March 10th, 2025 comparison.

Aggregate embedding benchmark comparisons, multilingual task-type scores, cross-lingual retrieval results, and commercial-use indicators.

Related work motivates two uses of large language models (LLMs): generating embedding training data and adapting pretrained model parameters into an embedding model. Earlier sentence encoders established reusable representations for multiple tasks, while subsequent work used larger language models as their starting point. The paper identifies benchmark-specific overfitting, improved performance that does not transfer beyond the training domain, as a concern with extensive in-domain training.

Prior synthetic-query methods show that the passage used to generate a query need not be its best positive target, the passage considered relevant for training.
The cited literature includes multilingual generation of query, positive-target, and hard-negative triplets. A hard negative is a target that resembles a relevant passage semantically but does not answer the query.

3. Gemini Embedding

Gemini Embedding adapts Gemini into a shared encoder, a model that converts inputs into vectors, for multilingual text and code. The method combines a transformer architecture that builds contextual token representations, sequence pooling that combines them, a projection into the output vector space, and contrastive training. Figure 1 illustrates the intended semantic relationships among multilingual inputs.

Multilingual text inputs mapped through Gemini into dense vector representations.

3.1. Model Architecture

The transformer initialized from Gemini uses bidirectional attention, allowing each token representation to incorporate context from both directions. Mean pooling averages these representations across the input sequence, and a randomly initialized linear projection maps the average to the output embedding dimension. The report describes this adaptation but does not identify the exact Gemini checkpoint or parameter count.

3.2. Training Objective

Training uses noise-contrastive estimation (NCE), a loss that distinguishes positive targets from negative targets, with in-batch negatives taken from other examples in the batch. A task string such as “question answering” or “fact checking” is prepended to the query, while targets are encoded without that string. Cosine similarity, the normalized vector dot product, scores the resulting representations.

Each example contains a query, a positive target, and optionally a hard negative. A mask excludes comparisons involving repeated queries or repeated positive targets, which matters when classification examples share labels.
The explicit hard-negative denominator term is omitted when no hard negative is supplied. Same-tower negatives, comparisons between queries or between targets, are omitted because the authors report that including them reduces performance on most tasks, potentially through false negatives: relevant pairs incorrectly treated as nonmatches.
Matryoshka Representation Learning (MRL), training with losses over overlapping vector prefixes, supports 768- and 1,536-dimensional prefixes within the full 3,072-dimensional embedding.

3.3. Recipe

Pre-finetuning adapts Gemini from autoregressive generation, producing text token by token, to encoding using many potentially noisy query–target pairs, large batches, and no explicit hard-negative term. Fine-tuning then uses task-specific query–target–hard-negative triples, smaller batches such as fewer than 1024 examples, and one dataset per batch so that competing targets come from the same task.

Pre-finetuning runs for substantially more steps than fine-tuning because its dataset is larger. Exact batch sizes and step counts are not disclosed.
Grid searches vary training hyperparameters and dataset inclusion to obtain candidate checkpoints. Model Soup averages parameters within or across runs, including weighted combinations, with checkpoints selected through deliberate data variation and manual experimentation.
The paper does not report a separate quantitative ablation, a comparison isolating one component, for Model Soup.

4. Datasets

The training mixture spans multilingual embedding tasks and code retrieval, finding relevant code for a query. Gemini improves the data through synthetic example generation, removal of low-quality examples, and hard-negative mining, selecting difficult irrelevant targets for training.

4.1. Training Data Mixture

Pre-finetuning uses title–passage positive pairs from a billion-scale web corpus. Fine-tuning prepares three mixtures emphasizing task diversity, language diversity, and coding capability, using selected academic datasets and synthetic datasets.

The authors exclude many in-domain datasets from the classic Massive Text Embedding Benchmark (MTEB) because they observed improvements confined to those datasets’ own test splits. They associate this behavior with train–test leakage, information shared between training and evaluation data, or dataset bias.
Mixture rates are selected through a fine-grained grid search initialized from the training steps needed for convergence on individual datasets. The complete dataset inventory, mixture weights, and selected schedules are not supplied.

4.2. Improving Data Quality with Gemini

For retrieval, Gemini generates queries for web passages and acts as an automatic quality rater to remove examples such as unrealistic queries. For classification, it generates English counterfactual, sentiment, and review examples through multi-stage prompts conditioned on generated users, products, or movies. Sampling later entries in longer generated lists is used to increase diversity.

Few-shot prompting, supplying examples in the prompt, supports retrieval generation and data-quality assessment. Gemini also filters human-annotated retrieval examples with incorrect positive or negative targets.
Hard-negative mining starts with an embedding model trained without hard negatives and retrieves the top k neighboring passages for each query. Gemini scores these candidates using graded classification, assigning relevance levels, and query likelihood, assessing how likely the query is given a passage.
Reciprocal Rank Fusion (RRF), a method for combining candidate rankings, combines the two assessments. The authors select the lowest Gemini-scoring neighbors within the retrieved set as hard negatives.

5. Evaluation

Evaluation tests whether one model and shared embedding space work across task types, domains, languages, and query–passage language pairs. It combines MMTEB evaluations with XOR-Retrieve and XTREME-UP, two benchmarks that retrieve English passages using queries in other languages.

5.1. Benchmarks and Tasks

The paper reports 164 individual MMTEB evaluation tasks and separately lists 132 tasks for MTEB(Multilingual), 41 for MTEB(Eng, v2), and 12 for MTEB(Code). These collection counts sum to 185; the report does not explain how they relate to the stated total of 164. The framework covers 250+ languages and 10 task types.

The task types are Bitext Mining, finding corresponding texts across languages; Classification; Clustering; Instruction Retrieval, finding passages under an instruction; Multilabel Classification, assigning multiple labels; Pair Classification, assigning a label to a text pair; Reranking, reordering candidate results; Retrieval; semantic textual similarity (STS), scoring similarity between texts; and Summarization.
XOR-Retrieve pairs English passages with queries in 7 languages, while XTREME-UP uses queries in 20 underrepresented languages.
Task mean weights tasks equally, while task-type mean weights task types equally. The paper uses Borda rank as the official leaderboard ranking metric and reports MRR@10, mean reciprocal rank within the first 10 results, for XTREME-UP. It reports XOR-Retrieve recall as Recall@5kt without defining that cutoff notation in the report.

5.2. Overall Performance

Gemini Embedding ranks first on the reported March 10th, 2025 multilingual, English, and code leaderboards. Table 1 reports multilingual task and task-type means of 68.32 and 59.64, English means of 73.30 and 67.67, and a seven-task code mean of 74.66. Tables 2–4 provide task-type and code-task breakdowns.

Multilingual performance does not lead every category: multilingual-e5-large-instruct scores 80.13 versus Gemini Embedding’s 79.32 on Bitext Mining, and gte-Qwen2-7B-instruct scores 85.13 versus 83.64 on Pair Classification.
The English leaderboard reports 73.3 task mean and 67.7 task-type mean. Gemini Embedding leads the displayed Retrieval, STS, and Summarization columns, while other models score higher in several remaining categories.
The code leaderboard covers eight tasks and excludes CodeFeedbackMT, CodeFeedbackST, StackOverflowQA, and SyntheticText2SQL. Gemini Embedding scores 75.5 across all eight and 74.7 when excluding COIRCodeSearchNetRetrieval, a code retrieval task missing from many competing submissions; voyage-code-3 leads several individual tasks but lacks a complete mean.
XOR-Retrieve reaches 90.42 Recall@5kt versus 68.76 for text-embedding-3-large and 65.67 for Gecko. XTREME-UP reaches 64.33 MRR@10; Table 5 reports the highest displayed non-Gemini average as 39.2 for voyage-3-large.

Multilingual leaderboard rankings with aggregate scores and task-type performance.

English embedding leaderboard rankings and performance by task type.

Code embedding leaderboard results with eight-task means, means excluding COIR, and individual task scores.

XTREME-UP cross-lingual retrieval results across 20 query languages.

5.3. Qualitative Examples

Figure 2 presents Assamese and Hindi queries encoded directly, without translation, to retrieve English passages by cosine similarity. Gemini Embedding retrieves passages about Mohenjo-daro’s river setting and the Indian roller as Andhra Pradesh’s state bird, while text-embedding-3-large retrieves unrelated passages. These selected cases illustrate cross-lingual entity and context matching, including a Hindi query with a typo, but do not measure a general robustness rate.

Assamese and Hindi query examples comparing English passages retrieved by Gemini Embedding and text-embedding-3-large.

6. Ablation Study

The ablation study examines training stages, fine-tuning mixtures, synthetic classification data, retrieval-data filtering, and hard-negative counts. Tables 6 and 7 explicitly exclude Model Soup from their ablation models, so these component comparisons differ from comparisons against the final combined model.

6.1. Does Gemini Embedding Generalize to Multilingual Tasks?

Pre-finetuning raises the multilingual task mean from 30.55 without embedding-specific training to 48.89, the English mean from 28.17 to 50.99, and the code mean from 9.86 to 46.18. Subsequent English-only diverse-task fine-tuning reaches 66.75 on MTEB(Multilingual), 72.77 on MTEB(Eng, v2), and 49.34 on XTREME-UP. This supports transfer beyond the fine-tuning language after Gemini initialization and pre-finetuning; it does not establish that the model’s entire training history used only English.

Multilingual retrieval-only fine-tuning scores 58.24 on MTEB(Multilingual), below English diverse-task fine-tuning, but reaches 65.06 on XTREME-UP, above the final model’s 64.33. The mixtures therefore produce different tradeoffs between broad task performance and cross-lingual retrieval.
Code-only retrieval fine-tuning reaches 72.08 on MTEB(Code), compared with 58.68 for English-only and 58.75 for multilingual-only fine-tuning.
The authors interpret the results as evidence that task diversity matters more than language diversity for broad embedding performance. Because the compared mixtures differ in both task and language composition, the experiment does not isolate those factors independently.

Training-stage and fine-tuning-mixture ablations across five benchmarks, with ablation models excluding Model Soup.

6.2. How Does Gemini Improve Data Quality?

Synthetic classification training raises the average across AmazonCounterfactual, AmazonPolarity, AmazonReviews, and Emotion from 57.57 to 75.17, a gain of 17.6 points. The authors state that these datasets were generated without original dataset examples in the prompts, which they call zero-shot synthetic generation. Filtering MIRACL, a retrieval dataset spanning 18 languages, raises its average from 59.8 to 63.7.

All four classification tasks improve with synthetic data, with different gains: AmazonCounterfactual rises from 65.43 to 91.30, while Emotion rises from 48.70 to 55.90.
Filtering improves the average but reduces Arabic from 74.8 to 74.2, Hindi from 59.4 to 59.3, and Yoruba from 85.0 to 83.3.
Figure 3 varies hard-negative counts over 0, 1, 3, 5, and 7 on four retrieval datasets: FEVER, HotpotQA, NQ, and SciFact. The plotted metric is nDCG@10, normalized discounted cumulative gain at rank 10, which measures ranking quality. Moderate counts generally help, but one hard negative reduces HotpotQA performance, and larger counts often decline from the best setting.
The authors attribute degradation with excessive hard negatives to overfitting and propose regularization and improved sampling as future work. Their suggestion that controllable synthetic generation might reduce bias is not tested through a direct bias evaluation.

Classification results with and without synthetic training data, alongside Gecko and final Gemini Embedding results.

MIRACL retrieval results across 18 languages before and after training-data filtering.

Retrieval nDCG@10, normalized discounted cumulative gain at rank 10, as the number of hard negatives varies on FEVER, HotpotQA, NQ, and SciFact.

7. Future Work

Future work proposes extending the shared embedding space to image, video, and audio inputs. The authors identify multimodal task curation, preparing tasks involving different input types, and training recipes that balance single-modality and multimodal performance as research directions. These extensions are not evaluated in this text embedding report.

8. Conclusion

The paper concludes that Gemini initialization and Gemini-assisted training data can produce one reusable representation model for multilingual text, English, and code tasks. Its benchmark results support broad embedding quality, while precomputing and caching vectors provides the proposed mechanism for deployment in latency-sensitive settings. The report does not quantify latency, throughput, or inference cost.

Appendix

1. Full Results provides per-task multilingual scores in Table 9, English and code scores in Table 10, and per-language cross-lingual retrieval scores in Table 11. The multilingual results include −2.41 for Robust04InstructionRetrieval, 2.96 for TempReasonL1, and 99.31 for HagridRetrieval; XTREME-UP scores include 25.66 for brx and 70.82 for ml. Scores from different task metrics are not directly comparable, but the within-benchmark breakdowns show variation hidden by aggregate means.
1. Contributions and Acknowledgments distinguishes Core Contributors, Contributors, and Leadership. Jinhyuk Lee, Feiyang Chen, Sahil Dua, Daniel Cer, and Madhuri Shanbhogue are marked as equal contributors. Leadership lists Zach Gleicher, Karan Gill, Zhe Dong, Mojtaba Seyedhosseini, Yunhsuan Sung, Raphael Hoffmann, and Tom Duerig; the Contributors list recognizes additional team members, and the final page acknowledges further collaborators.

Brief Thoughts

The most informative evidence combines broad benchmark coverage with ablations measuring gains from pre-finetuning, synthetic classification examples, and retrieval-data filtering. The mixture comparisons also show that the final model’s broader performance comes with a lower XTREME-UP score than multilingual retrieval-only fine-tuning. Reproduction and deployment assessment are limited by undisclosed model size, incomplete training specifications, manual checkpoint selection, and absent latency or compute measurements; the report also leaves its task counts unreconciled and gives an English final score of 73.28 in Table 6 versus 73.30 in Table 1.