NeoBERT: A Next-Generation BERT | Summary

This article explains the key points of NeoBERT: A Next-Generation BERT.

• 2025-02-26 (arXiv)
• Breton, Lola Le, Fournier, Quentin, Mezouar, Mariam El, Morris, John X., Chandar, Sarath.
• Chandar Research Lab, Mila – Quebec AI Institute, Polytechnique Montréal, Cornell University, Royal Military College of Canada, Canada CIFAR AI Chair
• Paper
• Github
• Project Page

Read this article in Korean

Summary

• NeoBERT addresses the limited modernization of bidirectional text encoders, models that use context on both sides of a token to produce representations for classification, clustering, and retrieval. After the same 2,000-step fine-tuning procedure, its 250M-parameter model scores 51.3 on the English Massive Text Embedding Benchmark (MTEB), versus 49.1 for BERT large, the strongest baseline in that comparison.
• NeoBERT combines a 28-layer, 768-dimensional architecture with RefinedWeb, a filtered and deduplicated web corpus, and masked language modeling, a training objective that predicts hidden tokens. Two-stage pre-training extends its training context from 1,024 to 4,096 tokens, with a theoretical total of 2.1T tokens that includes padding.
• NeoBERT scores 89.0 on GLUE, a benchmark of language understanding tasks, and 66.60 on MTEB after Contextual Document Embeddings (CDE) training, which groups related examples and conditions representations on corpus context. Controlled ablations, experiments that change selected design choices, show gains from larger data and model size; two retained training changes initially reduce performance, and their proposed benefits at scale remain unverified.

1 Introduction

The paper distinguishes improvements to encoder pre-training from improvements to downstream fine-tuning. Existing embedding systems often refine older backbones, the underlying pre-trained encoders, through contrastive learning, which brings related text representations closer and separates unrelated ones; NeoBERT updates the backbone and compares models through a shared fine-tuning procedure.

• The intended contribution is a reusable encoder with the 768-dimensional hidden size of BERT base, allowing compatibility with components that expect that representation size.
• The authors validate successive design changes through ten pre-trained models and separately test context extension and inference throughput, measured in tokens processed per second.

2 Related work

Related work progresses from BERT and RoBERTa pre-training to increasingly specialized embedding fine-tuning. SimCSE constructs positive pairs by processing the same text twice with dropout, which randomly suppresses activations during training; GTE uses multiple contrastive stages, and jina-embeddings uses Low-Rank Adaptation (LoRA), task-specific low-rank parameter updates.

• CDE provides another route to stronger embeddings through related-example batching and corpus-conditioned representations.
• NomicBERT and the concurrently developed ModernBERT modernize encoder architectures and training; NeoBERT evaluates a distinct combination of architecture, data, and training choices.

3 NeoBERT

NeoBERT is a 250M-parameter encoder with 28 layers, hidden size 768, and 12 attention heads, parallel attention computations within each layer. Table 1 positions it between conventional base and large encoders and compares their architecture, data, objectives, context lengths, and training configurations.

NeoBERT preserves the 768-dimensional interface of base encoders while adding depth and parameters. ModernBERT has a longer trained context window, 8,192 tokens versus NeoBERT’s 4,096, so the table does not support a claim that NeoBERT has the largest native context.

3.1 Architecture

The architecture retains BERT base’s width of 768 and increases depth to 28 layers. Rotary Position Embeddings (RoPE), which inject relative positional information into attention, replace absolute position embeddings; the model is also compatible with Yet Another RoPE Extension (YaRN), a method for adapting models to longer contexts.

• Pre-Layer Normalization places normalization inside residual branches. NeoBERT uses Root Mean Square Layer Normalization (RMSNorm), which requires one fewer statistic than conventional LayerNorm.
• SwiGLU, a gated activation function, replaces the Gaussian Error Linear Unit (GELU) activation. The feed-forward hidden-unit count is scaled by 2/3 to compensate for SwiGLU’s additional weight matrix.

3.2 Data

RefinedWeb contains 600B tokens. The authors emphasize its scale and diversity while acknowledging that it lacks strict high-quality constraints.

• Stage one runs for 1M steps with a maximum length of 1,024 tokens, producing NeoBERT1024. Stage two adds 50k steps at a maximum length of 4,096 tokens, producing NeoBERT4096.
• During stage two, batches are drawn from the original corpus and subsets containing sequences longer than 1,024 and 2,048 tokens with probabilities 20%, 40%, and 40%, respectively.

3.3 Pre-Training

Pre-training uses only masked language modeling, with 20% of tokens selected and all selected tokens replaced by masks. A theoretical batch size of 2M tokens across 1,050,000 steps yields 2.1T tokens; this count includes padding and therefore does not measure the number of actual text tokens processed.

• AdamW, the optimizer with decoupled weight decay, uses β1 = 0.9, β2 = 0.95, ε = 10−8, weight decay 0.1, and gradient clipping at norm 1.0. A 2,000-step warmup reaches 6 × 10−4, followed by cosine-shaped learning-rate decay to 10% of the peak and a constant-rate tail covering the last 100k short-context steps and 50k long-context steps.
• DeepSpeed with ZeRO, a distributed-training memory optimization, reduces duplicated state across accelerators. xFormers supplies fused operators, and FlashAttention computes exact attention without storing full attention matrices; dimensions are multiples of 64, and biases are removed.

4 Effect of Design Choices

The ablation sequence starts from M0, a BERT-base-like model with pre-layer normalization and no next-sentence prediction objective, the task of predicting whether two sentences are consecutive. Models are trained for 1M steps with controlled random seeds and dataloader states; the initial GLUE search uses batch sizes {16, 32} and learning rates {1e−5, 2e−5, 3e−5}.

• The architectural bundle raises the score from 79.2 to 81.1, and RefinedWeb raises it to 84.0. Increasing model size from 120M to 250M raises 82.9 to 85.3; reallocating those parameters from 16 layers of width 1056 to 28 layers of width 768 raises it to 86.0.
• M3, the LLaMA tokenizer experiment that changes how text is split into tokens, scores 82.2. M6, sequence packing that concatenates examples without preventing attention across their boundaries, scores 80.5; both changes are discarded from subsequent models.
• AdamW with cosine decay and the 20% masking change produce reported relative decreases of −0.5% and −0.7%, respectively, but remain in the final recipe. Their proposed benefits during larger-scale training are hypotheses that the authors could not verify because of computational cost.

The sequence separates model-size growth from the later depth-to-width reallocation, allowing comparison at approximately fixed parameter count. Several rows change multiple factors together, including architecture, optimization, and training scale, which limits attribution to individual components.

Training-scale changes produce 87.3, and a larger downstream hyperparameter search produces 89.0. The final improvement therefore includes fine-tuning configuration selection as well as backbone changes; the cumulative experiments do not isolate every component or interaction.

RefinedWeb and model-size growth produce the largest positive transitions, while the gray tokenizer and packing experiments are discarded. The final rise from 87.3 to 89.0 comes from a larger fine-tuning search, rather than another pre-training change.

5 Experiments

Experiments cover GLUE for established language understanding comparisons and MTEB for broader embedding evaluation. Separate tests measure long-sequence modeling and synthetic inference throughput.

5.1 GLUE

NeoBERT1024 and NeoBERT4096 achieve GLUE development-set averages of 88.8 and 89.0. NeoBERT4096 is comparable to RoBERTa large at 88.9, but remains below ModernBERT large at 90.5 and DeBERTaV3 large at 91.4.

• Baseline scores in Table 3 come from prior publications, so this comparison does not use the uniform fine-tuning protocol applied in the MTEB experiment.
• Appendix C specifies training-split fine-tuning, validation-based selection, and transfer from the best Multi-Genre Natural Language Inference (MNLI) checkpoint, trained to assess relationships between sentences, to related tasks. The reported results are development-set scores rather than held-out test-set submissions.

NeoBERT4096’s 89.0 average is close to RoBERTa large’s 88.9 despite using fewer parameters. ModernBERT large and DeBERTaV3 large remain ahead, and the baseline results come from separate publications.

5.2 MTEB

MTEB’s English subset contains 56 datasets across seven task types. Since masked-token pre-training does not directly optimize sentence-level representations, the authors fine-tune all backbones through a common embedding procedure before comparing them.

5.2.1 Unified Contrastive Learning uses positive query-document pairs drawn from open datasets totaling nine million documents, optional hard negatives, and task-homogeneous in-batch negatives, nonmatching examples from the same task and training batch. Hard negatives are nonmatching documents that resemble the contrasted example; task-specific instructions and dataset sampling adjusted by a temperature parameter are shared across models.

• After 2,000 steps, NeoBERT4096 averages 51.3, compared with BERT large at 49.1, RoBERTa large at 48.9, NomicBERT2048 at 47.1, and ModernBERT large at 46.9.
• The paper reports a +4.5% relative improvement over the second-best model. NeoBERT leads the full comparison in clustering, pair classification, retrieval, and semantic textual similarity (STS), which measures similarity in sentence meaning.
• Training beyond 2,000 steps produces minimal gains in this experiment. The comparison establishes performance under the shared adaptation procedure; it does not establish each backbone’s maximum achievable performance.

Under the shared 2,000-step procedure, NeoBERT has the highest overall average at 51.3, with BERT large second at 49.1. NeoBERT leads four of seven categories; ModernBERT large leads classification, RoBERTa large leads reranking, and BERT large leads summarization.

5.2.2 Contextual Document Embeddings applies the more intensive CDE procedure to NeoBERT. Its MTEB average reaches 66.60, versus 65.68 for ModernBERT and 65.00 for NomicBERT; Table 5 reports first place among models under 400M parameters as of April 2025.

• NeoBERT leads these three models in classification, clustering, pair classification, reranking, which reorders candidate documents by relevance, and retrieval.
• ModernBERT remains stronger in STS, 83.30 versus 82.30, and summarization, 31.60 versus 30.98. The 66.60 result uses a substantially different training regime from the 51.3 unified-protocol result.

Under CDE training, NeoBERT exceeds ModernBERT by 0.92 average points and NomicBERT by 1.60. It leads five categories, while ModernBERT retains the strongest STS and summarization scores; the under-400M ranking is dated April 2025.

5.3 Sequence Length

The context experiment samples 2,467 long English Wikipedia sequences and independently masks each token to compute pseudo-perplexity, the exponential of the mean masked-token cross-entropy. NeoBERT1024 maintains low pseudo-perplexity near 3,000 tokens before it rises sharply, whereas NeoBERT4096 maintains low pseudo-perplexity across its extended training range.

• Figure 2 supports the effectiveness of the additional 50k long-context training steps.
• This evaluates masked-token modeling on Wikipedia; it does not directly measure retrieval accuracy or reasoning over long documents.

NeoBERT1024’s pseudo-perplexity rises sharply beyond roughly 3,000 tokens, whereas NeoBERT4096 remains comparatively stable through 4,096. This supports the long-context training stage for masked-token modeling on Wikipedia, without directly establishing downstream long-document performance.

5.4 Efficiency

Efficiency is measured on synthetic sequences of lengths {512, 1024, 2048, 4096, 8192}, with batch sizes increased from 1 to 512 or until memory exhaustion. Each setting runs for 100 inference steps on one A100 accelerator, and the authors report the best throughput for each model and sequence length.

• At 4,096 tokens, NeoBERT reaches 17.2 ± 1.2 thousand tokens/second, versus 11.7 ± 0.8 for ModernBERT base; the main text reports a 46.7% speedup.
• BERT base is faster at 512 tokens, at 27.6 ± 3.6 thousand tokens/second versus NeoBERT’s 24.5 ± 1.4. The throughput advantage depends on sequence length and the optimized batch size.
• The 8,192-token throughput measurement demonstrates that inference can run at that length; it does not establish reliable modeling quality there.

NeoBERT’s throughput declines more slowly with sequence length than ModernBERT’s in the single-A100 experiment. Its advantage over ModernBERT base appears beyond 1,024 tokens, while BERT base is faster at 512; each configuration uses its best tested batch size.

6 Discussion

The discussion argues that encoder rankings depend strongly on evaluation and adaptation. GLUE covers a narrow collection of tasks and is susceptible to overfitting, while MTEB leaderboard performance can depend on expensive fine-tuning procedures that obscure the contribution of the pre-trained backbone.

• DeBERTaV3’s strong GLUE score and weak performance under this paper’s shared MTEB protocol illustrate the mismatch between evaluations.
• The authors advocate standardized, affordable fine-tuning protocols and improved zero-shot evaluation, meaning evaluation without task-specific model adaptation.

7 Conclusion

The conclusion presents NeoBERT as an openly released encoder backbone with competitive language understanding results, stronger embeddings under shared fine-tuning, and improved long-sequence throughput in the reported setup. Code, model checkpoints, data, and training scripts are linked at https://huggingface.co/chandar-lab/NeoBERT and https://github.com/chandar-lab/NeoBERT.

Broader Impact Statement

NeoBERT inherits biases and limitations from its pre-training corpus. Because the largest ablation gain comes from changing the dataset, the authors expect further improvements to require retraining on newer, larger, or more diverse data; the experiments do not establish retraining as a requirement.

Appendix

• A Training details reports 8 H100 accelerators, 1,050,000 steps, and 6,000 GPU hours. Stage one uses local batch size 32, eight gradient accumulation steps, which combine gradients before an optimizer update, and length 1,024; stage two retains the theoretical 2M-token batch while increasing length to 4,096.
• B Ablations and B.1 Baseline clarify M0’s differences from BERT base: pre-layer normalization and removal of next-sentence prediction. B.2 Tokenizers discusses algorithm, training-corpus, casing, and character-handling differences, but does not isolate their contributions; it also reports a 2.9% tokenizer drop, inconsistent with the −2.1% in the main text and Figure 1.
• C GLUE describes training for 10 fine-tuning epochs, validation every min(500, len(dataloader) // 10) steps, and early stopping after 15 evaluations without improvement. The Winograd Natural Language Inference (WNLI) task, a sentence-inference evaluation, is excluded, and paired metrics are averaged; Table 6 gives a broader search space than the appendix prose: batch sizes {2, 4, 8, 16, 32}, learning rates {5e−6, 6e−6, 8e−6, 1e−5, 2e−5, 3e−5}, and weight decay {1e−2, 1e−5}.
• D MTEB includes D.1 Evaluation of pre-trained models, where zero-shot BERT base scores 37.4 and exceeds BERT large at 36.3 and RoBERTa large at 36.0. D.2 Contrastive learning specifies cosine similarity, a measure of alignment between representation vectors, temperature τ = 0.07 controlling similarity scaling in the loss, dataset sampling proportional to the square root of dataset size, 2,000 fine-tuning steps, and float16 evaluation using 16-bit floating-point values. D.3 Task instructions provides the training and evaluation instructions in Tables 7–9.
• E Context Length finds that NeoBERT4096 maintains relatively low Wikipedia pseudo-perplexity toward 6,000 tokens, followed by substantial degradation at greater lengths. F Efficiency provides the complete throughput table, including 13.0 ± 0.2 thousand tokens/second at 8,192 tokens for NeoBERT and 6.8 ± 0.2 for ModernBERT base.
• G Contextual Document Embeddings and G.1 Method describe clustering examples into related batches and using two backbones: one encodes contextual documents, and another conditions query or document representations on those encodings. Both are trained end to end through a two-stage gradient-caching procedure, which caches intermediate gradients to support training the contextual architecture.
• G.2 Data and hyperparameters uses 235M noisy query-document pairs per epoch followed by 1.5M high-quality examples, with three epochs per stage. Adam uses 1,000 warmup steps to 5e−5 and linear decay to zero; batch and cluster sizes are 512, negatives are not shared between accelerators, and 512 contextual documents with per-document length 512 produce a second-stage input of 1024 tokens. G.3 Results reports 66.60 and suggests that the smaller STS score relative to ModernBERT may reflect the pre-training data mix, a hypothesis the paper does not test.

Brief Thoughts

NeoBERT improves the MTEB average under both the shared adaptation protocol and the more intensive CDE procedure. The ablations support gains from dataset replacement, model size, and depth-to-width reallocation, while bundled changes leave individual component effects unresolved.

The evidence supports evaluating NeoBERT for English representation workloads, including those with longer inputs. Model selection still depends on the task: larger models lead GLUE, other encoders lead some embedding categories, and the throughput comparison uses synthetic inputs with separately optimized batch sizes.