Llama-Embed-Nemotron-8B: A Universal Text Embedding Model for Multilingual and Cross-Lingual Tasks | Summary

This article explains the key points of Llama-Embed-Nemotron-8B: A Universal Text Embedding Model for Multilingual and Cross-Lingual Tasks.

• 2025-11-10 (arXiv)
• Babakhin, Yauhen, Osmulski, Radek, Ak, Ronay, Moreira, Gabriel, Xu, Mengyao, Schifferer, Benedikt, Liu, Bo, Oldridge, Even.
• NVIDIA
• Paper

Read this article in Korean

Summary

• The paper addresses how to build one text embedding model, which maps text into numerical vectors, that works across tasks and languages. On MTEB(Multilingual, v2), the multilingual split of the Massive Text Embedding Benchmark, llama-embed-nemotron-8b ranks first by Borda count, which aggregates relative rankings across tasks, with 39,573 votes versus 39,368 for gemini-embedding-001 and 39,364 for Qwen3-Embedding-8B as of October 21, 2025.
• The model converts Llama-3.1-8B into a bidirectional encoder, allowing every input token to attend to the entire sequence, and averages token representations into a 4096-dimensional vector. Training combines 7.7 million public-data pairs and 8.4 million synthetic pairs, examples generated with language models, and uses contrastive learning to bring related texts together and separate unrelated texts. The final model averages the parameters of six checkpoints, saved model states from different training runs.
• The ablations, experiments comparing specific design choices, find a competitive loss using only mined hard negatives, challenging irrelevant examples selected by similarity, and favor mixing synthetic classification data from multiple generators. Most ablations use a 1B model, with 1 billion parameters, and Qwen3-Embedding-8B retains higher mean benchmark scores than the final model. In the classification ablation, about 75k in-domain examples from the training splits of five evaluation datasets outperform about 1M synthetic examples on all five datasets.

1 Introduction

“Llama-Embed-Nemotron-8B: A Universal Text Embedding Model for Multilingual and Cross-Lingual Tasks” targets text representations across tasks, domains, and languages, including comparisons between texts in different languages. The introduction motivates text embeddings through retrieval, finding relevant documents for a query, question answering, semantic textual similarity (STS), measuring similarity in meaning between texts, and Retrieval-Augmented Generation (RAG), supplying language models with external context. It also identifies a continuing role for text-only models in text-native documents and pipelines that convert scanned documents into text.

The table establishes first place under Borda ranking while showing that the leading rank and the highest mean scores belong to different models. llama-embed-nemotron-8b leads gemini-embedding-001 by 205 votes and Qwen3-Embedding-8B by 209 votes, but Qwen3-Embedding-8B has the higher task and task-type means.

2 Model

The authors initialize from Llama-3.1-8B, remove causal attention masking, which restricts each token to itself and preceding tokens, in every transformer layer, and fine-tune all model weights. Global average pooling, which averages final-layer token representations over the sequence, produces a 4096-dimensional embedding. Task instructions use the exact template Input = f”Instruct: {task_instruction}\nQuery: {T}”.

• Retrieval uses a bi-encoder, which encodes queries and documents independently with the shared model. Queries receive instructions, documents require no specific formatting, and cosine similarity, which compares vector directions, supplies the relevance score.
• For STS and classification, assigning texts to labels, each text receives the appropriate instruction and is encoded individually; embeddings support similarity comparisons or serve as classifier features.

3 Training

Training combines a task-adapted contrastive objective, retrieval pretraining, multitask fine-tuning, and checkpoint averaging. The shared encoder learns from retrieval, classification, STS, and bitext mining, which identifies corresponding texts across languages.

3.1 Training Objective

The model uses InfoNCE, a contrastive loss that assigns the positive item a higher similarity than competing negative items: \(\mathcal{L}(q,d^+,D_N)=-\log\frac{\exp(\operatorname{sim}(q,d^+)/\tau)}{\sum_{d_i\in\{d^+\}\cup D_N}\exp(\operatorname{sim}(q,d_i)/\tau)}\). Here q is the anchor embedding, d+ is the positive embedding, DN is the negative set, sim is cosine similarity, and τ is the temperature controlling the scale of similarity scores.

• Retrieval uses an instructed query, an unmodified positive document, and mined negative documents. The loss excludes in-batch negatives, examples borrowed from other training instances in the batch, and same-tower negatives, comparisons among queries or among documents.
• Classification treats the input text as the anchor, the correct label name as the positive, and randomly selected incorrect label names as negatives. The anchor receives a task instruction, while label names remain unmodified.
• STS treats a positive text pair symmetrically and mines negatives from the dataset corpus. Every text receives the same instruction: “Retrieve semantically similar text.”

3.2 Training Stages

Stage 1: Retrieval Pretraining adapts the foundation model to bidirectional attention and embedding generation using web-derived query–document pairs with one hard negative per pair; the paper describes this as about 70% of the data mix. Stage 2: Fine-Tuning uses retrieval, classification, STS, and bitext mining data and is described as the remaining 30%. The starting Llama-3.1-8B model was pretrained on about 15T multilingual tokens.

3.3 Model Merging

Model merging combines trained models by averaging their parameters. The final model averages six checkpoints from runs with different data mixes and hyperparameters, retaining the architecture without increasing inference time, according to the authors. Section 6.4 evaluates the merged model against its constituent checkpoints.

4 Datasets

The training mix totals 16.1 million query–document pairs in the abstract and Table 2; the main text also refers to this approximately as 16 million. It contains 7.7 million non-synthetic pairs and 8.4 million synthetic pairs. The authors plan to release the curated mix through https://huggingface.co/collections/nvidia/nemotron-rag; the paper describes this release as future work.

The reported counts give 11.8M pretraining pairs and 4.3M fine-tuning pairs, totaling 16.1M. Synthetic data supplies 6.8M pretraining pairs and 1.6M fine-tuning pairs, so its largest contribution occurs during retrieval pretraining.

4.1 Pretraining Data Mix

The pretraining mix contains approximately 11.8M pairs based on the Diverse-QA split of Nemotron-CC-v2, the source corpus used for this stage. Existing questions and their positive documents contribute 5.0M pairs, with negatives mined from a pool of 1M document chunks sampled from the same corpus. Newly generated questions for existing documents contribute another 6.8M pairs, using the same negative-mining procedure.

4.2 Fine-tuning Data Mix

The fine-tuning mix starts from the NV-Embed data mix and contains approximately 2.7M public-data samples and 1.6M synthetic samples. Public sources include MIRACL, a multilingual retrieval dataset, HotpotQA, a question-answering dataset requiring multiple reasoning steps, MS MARCO, a machine-reading-comprehension dataset, Natural Questions, a question-answering benchmark, and SQuAD, a text-comprehension dataset. To improve zero-shot evaluation coverage, meaning evaluation on datasets whose splits were not used for training, the authors remove the majority of data originating from MTEB(Multilingual, v2) and MTEB(eng, v2); this section reports neither complete removal nor an exact zero-shot percentage.

4.3 Synthetic Data Generation

Synthetic Data Generation (SDG) covers retrieval, classification, STS, and bitext mining with an emphasis on multilingual and cross-lingual examples. One strategy generates complete query–positive–negative triplets from scratch; another generates queries for positive documents sampled from a seed corpus, an existing collection of texts, and mines negatives from that corpus. Existing datasets are also translated into target languages.

• The generators are gpt-oss-20b, gpt-oss-120b, Mixtral-8x22B-Instruct-v0.1, Llama-3.3-70B-Instruct, Llama-4-Scout-17B-16E-Instruct, and Llama-4-Maverick-17B-128E-Instruct.
• These large language models (LLMs), used here to generate tasks, examples, and translations, have publicly available weights. Section 6.2 tests generator choice specifically for classification.

4.4 Hard Negative Mining

Hard negative mining uses the NV-Retriever top-k with percentage to positive threshold strategy with a threshold of 0.95. For each query, the procedure selects the top K most relevant negative candidates whose query similarity is less than 95% of the query–positive similarity, aiming to exclude false negatives, relevant examples incorrectly treated as irrelevant. Mining combines e5-mistral-7b-instruct and Qwen3-Embedding-8B.

5 Results

Evaluation uses MTEB(Multilingual, v2), introduced in the Massive Multilingual Text Embedding Benchmark (MMTEB), comprising 131 tasks, 9 task types, and 250+ languages. Instructions come from the evaluation datasets when available and otherwise are adapted from https://github.com/QwenLM/Qwen3-Embedding. The model leads the October 21, 2025 leaderboard by Borda votes, but its Mean (Task), the average across tasks, of 69.46 and Mean (Type), the average across task types, of 61.09 are below Qwen3-Embedding-8B’s 70.58 and 61.69.

• Table 3 shows the highest task-type averages among the listed models for bitext mining (81.72), multilabel classification, assigning multiple labels to a text (29.86), and reranking, reordering candidate results by relevance (67.78). Qwen3-Embedding-8B leads classification, clustering, grouping related texts, pair classification, assigning labels to text pairs, retrieval, and STS; Qwen3-Embedding-4B leads instruction reranking, reranking guided by an instruction.
• Borda count aggregates relative task rankings, whereas the mean columns aggregate numerical scores. The first-place claim applies to the leaderboard’s ranking rule; the model does not lead every task type or aggregate metric.

The task-type breakdown limits the interpretation of the overall first-place rank. llama-embed-nemotron-8b leads bitext mining, multilabel classification, and reranking among the listed models, while Qwen3-Embedding-8B leads five task types and both mean columns.

6 Ablation Study

Unless otherwise stated, the ablations fine-tune Llama-3.2-1B on the fine-tuning data mix because experiments at 8B scale are costly. These experiments inform the final training design, but do not independently establish that every design choice has the same effect at 8B scale.

6.1 Contrastive Loss Formulations

The loss comparison changes the negative sets used by Gecko, Qwen3-Embedding, Gemini Embedding, and the authors’ hard-negative-only formulation. Other hyperparameters remain fixed, while the learning rate is tuned separately for each loss. The authors’ formulation obtains 38,225 Borda votes and a Mean (Task) of 64.03, versus 38,135 and 63.83 for the Gemini formulation.

• Gecko includes a hard negative, other positive passages in the batch, and other queries. Qwen3-Embedding also uses same-tower comparisons among positive and negative documents; Gemini omits same-tower negatives but retains other positive passages in the batch.
• The proposed negative set contains only hard negatives: one during pretraining and four during fine-tuning. The positive item remains in the InfoNCE denominator.
• The simpler formulation leads the aggregate metrics in Table 4, while other formulations lead some task types. No uncertainty estimates or significance tests accompany the comparison.

Under the 1B ablation protocol, the hard-negative-only formulation achieves the highest Borda votes and both mean scores. Its 38,225 votes exceed the Gemini formulation’s 38,135, but the task-type winners vary and no uncertainty estimates are reported.

6.2 Choice of LLM for Synthetic Data Generation

The 1B generator ablation adds 100k English synthetic classification examples to each training variant. Generation first proposes classification tasks and then produces texts, correct labels, and plausible incorrect labels; the mixed variant draws equally from six generators, approximately 16.7k samples each. This mix reaches 37,812 Borda votes and a Mean (Task) of 62.89, improving over the no-synthetic-classification baseline by +464 votes and +0.94 points.

• The mixed data also leads the classification, clustering, and multilabel classification averages in Table 5. Mixtral-8x22B-Instruct-v0.1 is the strongest individual generator by Borda votes, at 37,797.
• Generator size does not consistently predict downstream performance in this experiment. The authors suggest that a more diverse generated task list may explain the mixed variant’s advantage, but do not isolate that mechanism.
• The full training pipeline also uses cross-model generation, with one model proposing tasks and another generating examples. This extension is described without a separate quantitative comparison.

6.3 Impact of Synthetic vs. In-Domain Data

The 1B study compares a baseline without synthetic classification data against approximately 1M synthetic classification examples and approximately 75k in-domain examples from the training splits of five evaluation datasets. Synthetic data improves all five scores, but in-domain training produces the highest score on every dataset. These in-domain examples are used only for this ablation and are excluded from the final submitted llama-embed-nemotron-8b model.

• The datasets are AmazonCounterfactualClassification (17.7k training observations), CzechProductReviewSentimentClassification (24.0k), GreekLegalCodeClassification (28.5k), EstonianValenceClassification (3.3k), and TweetTopicSingleClassification (1.5k).
• GreekLegalCodeClassification scores are 38.21 for the baseline, 42.60 with synthetic data, and 60.51 with in-domain data. TweetTopicSingleClassification scores are 72.85, 78.00, and 80.12, respectively.
• The in-domain model is trained on all five datasets together. Its TweetTopic result therefore does not isolate the effect of training on only 1.5k examples, although 1.5k is the number contributed by that dataset.

Approximately 1M synthetic samples improve every tested classification dataset, yet approximately 75k in-domain samples from the five datasets together improve them further. The Greek score increases from 38.21 to 42.60 with synthetic data and to 60.51 with in-domain data, illustrating the remaining performance gap.

6.4 Model Merging

At the final model scale, equal-weight averaging of six checkpoints improves Borda votes from the best individual checkpoint’s 39,454 to 39,573, a gain of +119. Mean (Task) rises from 68.62 to 69.46, a gain of +0.84. The merged model exceeds all constituent checkpoints on eight of nine task-type averages; clustering remains higher for Individual model 4, at 54.91 versus 54.35.

The final merged model exceeds the best constituent checkpoint by 119 Borda votes and 0.84 mean task points. It also leads eight task-type columns among its constituents, while Individual model 4 retains the highest clustering score.

7 Conclusion

The paper concludes that bidirectional adaptation of Llama-3.1-8B, the public and synthetic data mix, and checkpoint averaging yield the leading Borda-ranked model on the reported multilingual leaderboard snapshot. Model weights are released at https://huggingface.co/nvidia/llama-embed-nemotron-8b. Release of the curated training data remains planned in this paper.

Appendix

• A Implementation Details reports 25.0 hours of pretraining and 21.5 hours of fine-tuning, both using 64 NVIDIA A100 80GB GPUs. Pretraining/fine-tuning settings are peak learning rates 1e-5/2e-6, batch sizes 2,048/128, 5,773/33,668 steps, and 1/4 hard negatives. Both stages use the AdamW optimizer, linear learning-rate decay, 100 warm-up steps, weight decay 0.01, temperature 0.02, and query and document maximum lengths of 512.
• B Fine-Tuning Data Mix for llama-embed-nemotron-8b lists 4,309,502 samples in total. Synthetic components comprise 169,534 bitext-mining samples, 1,044,212 classification samples, 182,814 retrieval samples, and 239,997 STS samples. The largest listed public components are MS MARCO and PAQ, a collection of generated questions, each with 500,000 samples, followed by MAmmoTH2 stackexchange with 317,180.

Brief Thoughts

The strongest evidence for the training design comes from the fixed-budget 100k-example generator comparison and the final-scale merging experiment: mixing generators improves the reported aggregate scores, and checkpoint averaging improves both Borda votes and mean task score. The hard-negative-only loss is competitive in the reported 1B setting, although its modest advantage over the Gemini formulation lacks uncertainty estimates. Application-specific evaluation remains necessary because the model does not lead every task family, synthetic classification data leaves an in-domain performance gap, and the paper reports no inference-throughput or latency comparison.