Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference | Summary

This article explains the key points of Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference.

• 2024-12-18 (arXiv)
• Warner, Benjamin, Chaffin, Antoine, Clavié, Benjamin, Weller, Orion, Hallström, Oskar, Taghadouini, Said, Gallagher, Alexis, Biswas, Raja, Ladhak, Faisal, Aarsen, Tom, et al.
• Answer.AI, LightOn, Johns Hopkins University, NVIDIA, HuggingFace
• Paper

Read this article in Korean

Summary

• ModernBERT addresses the limited context length, outdated training data, and inference costs of bidirectional encoders, models that represent each token using its surrounding text for classification and retrieval. Against the evaluated encoders in comparable size groups, it achieves the highest average short-context retrieval scores and processes fixed 8192-token inputs at 123.7 thousand tokens/s for base and 46.8 thousand tokens/s for large on one NVIDIA RTX 4090, compared with 47.5 and 16.5 thousand tokens/s for GTE-en-MLM using xformers, a library providing efficient transformer implementations.
• The 149-million-parameter base and 395-million-parameter large models combine hardware-aware dimensions with local attention over nearby tokens, global attention across the sequence, and unpadding, which removes computation on padding tokens throughout the model. Both train on approximately 2 trillion tokens, including code, and support a native 8192-token context.
• On General Language Understanding Evaluation (GLUE), a suite of sentence and sentence-pair understanding tasks, ModernBERT-base scores 88.4 versus 88.1 for DeBERTaV3-base, while ModernBERT-large scores 90.4 versus 91.4 for DeBERTaV3-large. On the English subset of MLDR, a long-document retrieval benchmark, multi-vector retrieval, which retains separate token representations, reaches 80.2 and 80.4 without additional MLDR fine-tuning, versus 69.3 and 71.3 for GTE-en-MLM.
• Single-vector retrieval, which compresses each document into one representation, remains weaker than GTE-en-MLM on MLDR without long-context fine-tuning. Efficiency measurements use synthetic inputs on one RTX 4090; the original BERT remains faster on fixed-length 512-token inputs.

1 Introduction

The introduction identifies a gap between the continued production use of encoder models and their limited modernization. Older backbones commonly restrict inputs to 512 tokens, lack code in pretraining, and use architectures or vocabularies that underuse contemporary hardware.

• Retrieval-Augmented Generation, which supplies retrieved context to a language model, is one motivating application; classification and query routing provide others.
• The authors release ModernBERT-base, ModernBERT-large, the modular FlexBERT framework for architecture experiments, and intermediate training checkpoints at https://github.com/AnswerDotAI/ModernBERT.

2.1 Architectural Improvements

The method combines established transformer updates, more efficient attention and input processing, and model dimensions selected for common graphics processing units (GPUs). Architectural choices were informed by ablations, experiments that vary individual design choices, mostly conducted at 8–20 billion training tokens rather than the final training scale.

2.1.1 Modern Transformer

ModernBERT removes bias terms from linear layers except the final token-prediction decoder and removes biases from LayerNorm, which normalizes hidden activations. It replaces absolute positional embeddings with rotary positional embeddings (RoPE), a method of encoding token positions chosen for its performance, efficient implementations, and ease of context extension.

• Pre-normalization applies LayerNorm before transformer sublayers. An additional normalization follows the embeddings, with the first attention-layer normalization removed to avoid duplication.
• GeGLU, a gated activation based on the Gaussian error linear unit (GeLU), replaces the original BERT activation arrangement.

2.1.2 Efficiency Improvements

Every third attention layer is global, while the remaining layers use a 128-token local sliding window. Global layers use RoPE theta, the positional frequency parameter, of 160,000 after context extension; local layers retain theta 10,000.

• Inputs are unpadded before embedding and remain unpadded across layers, with optional repadding at the output. The authors report a 10–20 percent performance improvement over approaches that repeatedly unpad and repad.
• Flash Attention, a family of memory- and compute-efficient attention kernels, supplies variable-length attention. The implementation combines Flash Attention 3 for global layers with Flash Attention 2 for local layers because Flash Attention 3 lacked sliding-window support when the work began.
• Compiling compatible modules with PyTorch’s torch.compile improves training throughput by a reported 10 percent with negligible compilation overhead.

2.1.3 Model Design

The design favors Deep & Narrow configurations, using more layers with smaller hidden dimensions while limiting inference slowdown. Base has 22 layers, hidden size 768, and gated expansion 2304; large has 28 layers, hidden size 1024, and gated expansion 5248. Their parameter counts are 149 million and 395 million.

• Dimension selection considers matrix tiling, which divides matrix operations into hardware-sized blocks, and utilization across NVIDIA T4, A10, L4, RTX 3090, RTX 4090, A100, and H100 GPUs, with greater weight on inference GPUs.

2.2 Training

Training combines a broader data mixture with a staged increase in context length and schedules for learning rate and batch size. The large model is initialized from the base model’s weights.

2.2.1 Data

Both models train on approximately 2 trillion tokens of primarily English web documents, code, and scientific literature. A modified tokenizer from the OLMo language model uses byte-pair encoding, a subword tokenization method, with a vocabulary of 50,368 entries including 83 unused tokens.

• The tokenizer preserves BERT’s [CLS] and [SEP] special tokens and input templating.
• Greedy sequence packing, which fills training sequences with examples to reduce unused space, achieves over 99 percent packing efficiency and reduces minibatch-size variation caused by unpadding.

2.2.2 Training Settings

Masked Language Modeling (MLM), which predicts masked input tokens, uses a 30 percent masking rate and omits Next-Sentence Prediction, BERT’s sentence-pair pretraining objective. StableAdamW adds update clipping as a per-parameter learning-rate adjustment to AdamW, an optimizer with decoupled weight decay; the authors report more stable training and better downstream performance than with standard gradient clipping.

• The learning-rate schedule uses warmup, a long constant phase, and a 1 − sqrt decay. Base uses 8e-4 after a 3-billion-token warmup; large initially uses 5e-4 after a 2-billion-token warmup, then restarts at 5e-5 for the remaining 800 billion short-context tokens after a plateau.
• Batch sizes increase from 768 to 4608 over 50 billion tokens for base and from 448 to 4928 over 10 billion tokens for large. Large initialization tiles base weights into larger matrices to accelerate initial training.
• After approximately 1.7 trillion tokens at sequence length 1024, context extension increases global RoPE theta from 10,000 to 160,000 and adds 250 billion tokens at length 8192, followed by 50 billion tokens with higher-quality sources upsampled and learning-rate decay. The extension learning rates are 3e-4 for base and 5e-5 for large.

3.1 Evaluation Setting

Downstream evaluation compares ModernBERT with BERT, RoBERTa, DeBERTaV3, GTE-en-MLM, and, in the base group, NomicBERT. It covers language understanding, short- and long-context text retrieval, and code retrieval using task-specific fine-tuning.

• These are size-group comparisons rather than exactly parameter-matched experiments: ModernBERT-large has 395 million parameters versus 435 million for GTE-en-MLM-large in the efficiency table.

3.1.1 Natural Language Understanding

Language understanding is measured on GLUE with a separate hyperparameter search for each subset. Tasks include sentiment classification and entailment, where the model determines whether one sentence follows from another.

• Prior-model GLUE scores are drawn from the literature. For GTE-en-MLM, the authors retain published scores after their own sweep produced inconsistent results, so the comparison does not use a fully uniform evaluation pipeline.

3.1.2 Text Retrieval

Text retrieval is evaluated on BEIR, a collection of retrieval datasets from different domains, using nDCG@10, normalized discounted cumulative gain over the first ten retrieved results. Dense Passage Retrieval (DPR) represents each document with one vector; ColBERT retains token vectors and uses MaxSim, the sum over query tokens of their highest similarity to any document token.

• Single-vector training uses 1.25 million samples from MS-MARCO, a passage-retrieval dataset, with mined hard negatives, irrelevant examples selected for their difficulty, a batch size of 16, and 5 percent learning-rate warmup. Contrastive learning brings relevant query-document representations closer and separates irrelevant ones.
• Multi-vector training follows the JaColBERTv2.5 training procedure with 810k MS-MARCO samples, batch size 16, and 5 percent warmup, using the PyLate retrieval library. Knowledge distillation transfers scores from the BGE-M3 teacher model through KL-Divergence, a measure of discrepancy between normalized teacher and student score distributions.
• Learning rates are selected using a subset of BEIR datasets; the appendix identifies NFCorpus, SciFact, TREC-Covid, and FiQA.

3.1.3 Long-Context Text Retrieval

Long-context retrieval uses the English subset of MLDR, which contains over 200,000 long documents. Three settings distinguish transfer from adaptation: single-vector evaluation after short-context training only, single-vector evaluation after additional MLDR fine-tuning, and multi-vector evaluation without additional MLDR fine-tuning.

• The paper acknowledges that standardized long-context encoder benchmarks are scarce. These experiments measure long-document retrieval and do not establish general long-context reasoning performance.

3.1.4 Code Retrieval

Code retrieval uses CodeSearchNet, where code is matched to relevant docstrings or comments, and StackOverflow-QA, where relevant answers may mix natural language and code. The latter has average query and document lengths of 1400 and 1200 words, respectively, producing average token counts above 2000.

• Both are evaluated as single-vector retrieval tasks through CoIR, the code information retrieval evaluation framework, reusing the selected text-retrieval hyperparameters. The results table abbreviates these benchmarks as CSN and SQA and calls the latter StackQA.

3.2 Downstream Results and Discussion

Table 1 shows the highest BEIR averages for ModernBERT in both retrieval settings: base scores 41.6 for DPR and 51.3 for ColBERT, while large scores 44.0 and 52.4. The corresponding GTE-en-MLM scores are 41.4 and 48.2 for base, and 42.5 and 50.7 for large.

• MLDR multi-vector scores reach 80.2 and 80.4, exceeding GTE-en-MLM’s 69.3 and 71.3. The proposed explanation involving longer pretraining and local attention remains a hypothesis.
• Single-vector MLDR transfer is weaker: base scores 27.4 versus 34.3 for GTE-en-MLM, and large scores 34.3 versus 36.4. Additional MLDR fine-tuning narrows the gaps to 44.0 versus 44.4 and 48.6 versus 48.9.
• ModernBERT leads the evaluated models on CodeSearchNet and StackQA at both sizes: 56.4 and 73.6 for base, and 59.5 and 83.9 for large. It leads GLUE in the base group; DeBERTaV3-large retains the higher large-model score.

ModernBERT leads both size groups on average BEIR retrieval, multi-vector MLDR, and both code tasks. GTE-en-MLM leads single-vector MLDR, while DeBERTaV3-large leads large-model GLUE. The base multi-vector MLDR gap, 80.2 versus 69.3, is much larger than the base DPR BEIR gap, 41.6 versus 41.4; the table reports no uncertainty estimates for these task scores.

4 Efficiency

Efficiency evaluation measures inference throughput and maximum batch size across short and long inputs. Varying sequence lengths tests the benefit of avoiding computation on padding.

4.1 Evaluation Setting

The experiment uses four synthetic datasets of 8192 documents each: fixed lengths of 512 or 8192 tokens, and variable lengths drawn from normal distributions centered on 256 or 4096 tokens. Results are averaged over ten runs on one NVIDIA RTX 4090.

• GTE-en-MLM is tested both out of the box and with xformers, which enables optimizations such as unpadding.
• The sampled long variable-length dataset has an actual average length of 4102 tokens; its distribution is centered on 4096.

4.2 Results

Table 2 reports base throughput of 148.1 and 147.3 thousand tokens/s for fixed and variable short inputs, and 123.7 and 133.8 thousand tokens/s for fixed and variable long inputs. Large reaches 52.3, 52.9, 46.8, and 49.8 thousand tokens/s in the same order.

• For variable inputs, ModernBERT processes 14.5–30.9 percent more tokens/s at short lengths and 98.8–118.8 percent more at long lengths than GTE-en-MLM with xformers.
• Base maximum batch sizes are 1604 for short inputs and 98 for long inputs, versus 640 and 38 for GTE-en-MLM. Large supports 770 and 48, versus 472 and 28.
• Fixed-short throughput remains below BERT’s 180.4 thousand tokens/s for base and 54.4 thousand tokens/s for large. BERT-large also accommodates a slightly larger short-input batch of 792.

The measurements show how input length distribution changes relative performance. ModernBERT-base reaches 133.8 thousand tokens/s on variable long inputs versus 67.3 for GTE-en-MLM with xformers, but its fixed-short throughput of 148.1 remains below BERT's 180.4. Maximum batch size measures memory capacity under this benchmark and does not directly report allocated memory in bytes.

5 Conclusion

The conclusion presents ModernBERT as an open encoder family combining stronger downstream results, native 8192-token support, and efficient inference. The reported evaluations show improvements over BERT and RoBERTa across the downstream categories, with particularly strong results for code retrieval and multi-vector long-context retrieval, alongside the task-specific exceptions above.

6 Limitations

The study focuses on English, leaving applicability to other languages, especially low-resource languages, unestablished. Web-heavy pretraining can transmit source-data biases, and masked-token prediction can produce harmful token suggestions; the authors distinguish this capability from extended text generation, for which ModernBERT was not trained.

• The authors identify the MLM-only objective as a limitation and propose investigating Replaced-Token Detection, an objective that detects substituted tokens, in combination with MLM.
• Parameter scaling beyond the released model sizes remains unexplored. The final comparisons also combine changes to architecture and training data, limiting attribution of the gains to individual changes.

8 Contribution Statement

Benjamin Warner, Antoine Chaffin, and Benjamin Clavié jointly led the project. Warner led model design and training, Chaffin coordinated the project and co-led ablations and final evaluation, and Clavié co-led final evaluation and led paper writing.

• Orion Weller led data selection and processing. Oskar Hallström and Said Taghadouini contributed architecture, training, ablations, and evaluation; Hallström investigated training issues, while Taghadouini adapted RoPE to unpadded sequences and ran final GLUE benchmarks.
• Other authors contributed sequence packing, early and long-context evaluation, code-data quality, software integration, and project guidance.

Appendix

• A Training Settings describes the staged training recipe, batch-size scheduling, initialization of large from base, weight-decay exclusions, and final checkpoint selection. Base uses checkpoint averaging, while large uses its best annealing checkpoint; exponential moving averages did not improve the tested configurations.
• B Model Design explains how model dimensions were selected for a basket of GPUs. Approximate hardware-utilization calculations serve as a design heuristic because a configuration cannot align perfectly with every target GPU.
• C Training Log records a sampling problem that caused an early base run to diverge and a separate plateau that led the authors to restart large with lower learning rate and weight decay. The sampling threshold was not rigorously established; base continued improving with diminishing gains during its completed pretraining run.
• D Architecture ablations compares activations, normalization, positional encoding, attention patterns, and tokenizers, mostly at smaller training scales. Several alternatives performed similarly, parallel attention degraded results at the tested sizes, and alternating local/global attention matched all-global attention on the tested downstream tasks. The final choices also reflect implementation efficiency and caution about extrapolating small-scale results.
• E Extended results gives task-level GLUE and BEIR results and their tuning procedures, including early stopping, MNLI-based initialization for selected tasks, and retrieval-model selection on a subset of BEIR. Strong average results do not imply leadership on every dataset; the authors identify TREC-COVID as an important contributor and discuss data recency as a possible, incomplete explanation.
• F Efficiency provides the synthetic-data statistics and detailed inference runtimes, variability, and maximum batch sizes that supplement the main efficiency evaluation.
• G Licensing releases the architecture, model weights, and training codebase under Apache 2.0.

Brief Thoughts

The clearest results are the combined retrieval and throughput improvements under the tested conditions. The multi-vector MLDR and code-retrieval gains provide stronger evidence of a practical quality improvement than the 0.2-point base DPR advantage on BEIR. Workload conditions matter: fixed 512-token inputs can favor older BERT models, and single-vector long-document transfer favors GTE-en-MLM in these experiments. The ablations support several design choices, but their mostly smaller training budgets and the simultaneous changes to data and architecture limit causal attribution. Throughput on real corpora and other GPUs remains untested by the reported efficiency experiment.