BitNet b1.58 2B4T Technical Report | Summary

This article explains the key points of BitNet b1.58 2B4T Technical Report.

• 2025-04-16 (arXiv)
• Ma, Shuming, Wang, Hongyu, Huang, Shaohan, Zhang, Xingxing, Hu, Ying, Song, Ting, Xia, Yan, Wei, Furu.
• Microsoft Research, University of Chinese Academy of Sciences, Tsinghua University
• Paper

Read this article in Korean

Summary

• Deploying large language models (LLMs), models that process and generate language, requires substantial memory, energy, and time to generate output tokens. BitNet b1.58 2B4T, a 2-billion-parameter model trained on 4 trillion tokens, achieves a reported Table 1 average of 54.19 versus 55.23 for instruction-tuned Qwen2.5 1.5B, with 0.4GB versus 2.6GB of weight memory excluding embeddings and 29ms versus 65ms per output token on the reported central processing unit (CPU) laptop benchmark.
• The model is trained from scratch with ternary weights, restricted to {−1, 0, +1}, and 8-bit integer activations, the intermediate values passed between layers. Training combines pre-training on text and code, supervised fine-tuning (SFT) on instruction and conversation examples, and Direct Preference Optimization (DPO), which learns from response preferences without a separate reward model.
• The model has a favorable performance–memory trade-off within the tested small-model comparisons, including higher reported averages than the evaluated 4-bit Qwen2.5 variants and other near-1-bit models. The 0.4GB figure excludes embeddings, the learned token representations; energy is estimated from arithmetic operations, and decoding latency is measured on one CPU configuration using different inference libraries for BitNet and its competitors.

1 Introduction

The Introduction identifies deployment resources as a barrier to using open language models on edge devices, which perform computation locally, and in applications requiring low response latency. It distinguishes post-training quantization (PTQ), which reduces numerical precision after training, from native low-bit training, which incorporates quantized weights from the start.

• The stated contribution is a native 1-bit model at 2 billion parameters and 4 trillion training tokens that approaches similarly sized full-precision models across language, mathematics, code, and conversation evaluations. Here, the paper’s 1-bit terminology includes ternary weights represented at 1.58 bits.
• Figure 1 plots average performance on 11 benchmarks against memory footprint. Its benchmark aggregate differs from the broader evaluation in Table 1.

BitNet appears at approximately the same 11-benchmark performance level as Qwen2.5 1.5B while using substantially less memory. The plot supports a favorable performance–memory position; its 11-task aggregate differs from the broader reported average in Table 1.

2 Architecture

The architecture derives from the Transformer, the report’s standard language-model architecture, and replaces torch.nn.Linear with BitLinear, custom linear layers that quantize weights during the forward pass. Weight quantization uses absmean, an absolute-mean-based scheme mapping weights to {−1, 0, +1}; activations use per-token absmax, an absolute-maximum-based scheme producing 8-bit integers. The model also uses subln, a normalization method included for training stability.

• The feed-forward network (FFN), a transformation sublayer, uses squared ReLU (ReLU²), a squared rectified-linear activation, for its potential sparsity and computational benefits. Rotary Position Embeddings (RoPE) supply positional information, and linear and normalization layers omit bias terms.
• The model adopts the LLaMA 3 tokenizer, which converts text into tokens using byte-level Byte-Pair Encoding (BPE), with a vocabulary of 128,256 tokens.
• The 1.58-bit designation describes ternary weights, not every value used during training or inference. The paper separately releases bf16, a 16-bit floating-point format, master weights for training and packed weights for inference.

3 Training

Training proceeds through pre-training, SFT, and DPO. Additional reinforcement learning, which optimizes behavior using rewards, remains a proposed extension for this release.

3.1 Pre-training

Pre-training uses a two-stage schedule over 4 trillion tokens. The first stage combines a relatively high peak learning rate with cosine decay, a smooth cosine-shaped reduction; approximately midway through the planned token count, a cooldown stage abruptly reduces the learning rate and continues with a lower-peak cosine schedule.

• 3.1.1 Learning Rate Schedule attributes the higher initial learning rate to the authors’ observation of greater stability in 1-bit training. Exact peak learning rates are not provided.
• 3.1.2 Weight Decay Schedule uses weight decay, a regularization setting, with a cosine schedule peaking at 0.1 in Stage 1 and sets it to zero in Stage 2.
• 3.1.3 Pre-training Data combines public text and code, including DCLM web data, FineWeb-EDU educational pages, and synthetic mathematics data. General web data dominates Stage 1, while higher-quality curated data receives greater emphasis during cooldown.

3.2 Supervised Fine-tuning (SFT)

SFT develops instruction following and conversational behavior using WildChat, LMSYS-Chat-1M, WizardLM Evol-Instruct, and SlimOrca, public instruction and conversation datasets. These are supplemented with synthetic examples generated using GLAN and MathScale, methods for constructing instruction-training data. The authors report better convergence from summing token cross-entropy losses, the token-prediction training losses, rather than averaging them, alongside relatively larger learning rates and more training epochs, or passes through the data, than typical full-precision fine-tuning.

• 3.2.1 SFT Data combines public conversation data with synthetic examples targeting reasoning and complex instruction adherence.
• 3.2.2 Chat Template uses the following literal structure during SFT and conversational inference: <|begin_of_text|>System: {system_message}<|eot_id|> User: {user_message_1}<|eot_id|> Assistant: {assistant_message_1}<|eot_id|> User: {user_message_2}<|eot_id|> Assistant: {assistant_message_2}<|eot_id|>…
• 3.2.3 Optimization Details describes these loss and hyperparameter choices qualitatively. It provides no numerical comparison isolating their effects, exact SFT learning rate, or exact epoch count.

3.3 Direct Preference Optimization (DPO)

DPO follows SFT to align responses with preferences concerning helpfulness and safety. It uses UltraFeedback and MagPie, the two reported preference-data resources, for 2 epochs with a learning rate of 2 × 10⁻⁷ and beta of 0.1, the parameter controlling divergence from the reference policy, or reference model.

• 3.3.1 Training Data combines the two public preference resources.
• 3.3.2 Training Details uses optimized computation routines from the Liger Kernel library. The authors qualitatively describe preferred response styles without substantial loss of core capabilities, but provide no quantitative before-and-after DPO comparison or dedicated safety evaluation.

4 Evaluation

Evaluation covers language understanding and reasoning, world knowledge, reading comprehension, mathematics, code generation, instruction following, and conversation. The authors rerun instruction-tuned versions of LLaMA 3.2 1B, Gemma-3 1B, Qwen2.5 1.5B, SmolLM2 1.7B, and MiniCPM 2B through public evaluation pipelines; instruction tuning trains models to respond to instructions.

• Language and reasoning benchmarks are ARC-Easy, ARC-Challenge, HellaSwag, WinoGrande, PIQA, OpenbookQA, and CommonsenseQA. World knowledge uses TruthfulQA and MMLU; reading comprehension uses TriviaQA and BoolQ.
• Mathematics benchmarks are GSM8K and MATH-500, and code generation uses HumanEval+. IFEval measures instruction following, while MT-bench evaluates conversation.
• The compared systems differ in parameter counts, pre-training token budgets, and training procedures. The evaluation compares released systems without holding training conditions constant to isolate weight precision.

4.1 Main Results

Table 1 reports 0.4GB of non-embedding weight memory, 29ms CPU time per output token, and 0.028J estimated arithmetic energy for BitNet, compared with 1.4GB–4.8GB, 41ms–124ms, and 0.186J–0.649J across the full-precision baselines. Its reported average is 54.19 for BitNet and 55.23 for Qwen2.5 1.5B.

• BitNet leads the compared models on ARC-Challenge at 49.91, PIQA at 77.09, WinoGrande at 71.90, and GSM8K at 58.38. The first three use zero-example prompting, and GSM8K uses four-example prompting.
• BitNet scores 53.17 versus Qwen2.5’s 60.25 on MMLU with five-example prompting, 38.40 versus 50.60 on HumanEval+ with zero-example prompting, and 43.40 versus 53.00 on MATH-500 with zero-example prompting.
• CPU latency uses a Surface Laptop Studio 2 with a 13th Gen Intel Core i7-13800H and 8 CPU threads, averaging time per token over 128 generated tokens. BitNet uses bitnet.cpp; other models use llama.cpp, a separate inference framework.
• Energy estimates cover matrix-multiplication arithmetic at a sequence length of 512 tokens. Table 4 supplies 7nm operation costs: FP16, a 16-bit floating-point format, costs 0.16pJ for addition and 0.34pJ for multiplication; INT8, an 8-bit integer format, costs 0.007pJ and 0.07pJ respectively. These are operation-energy coefficients, not whole-device measurements.

BitNet has the lowest reported non-embedding memory, CPU time per output token, and estimated arithmetic energy among the listed systems. Its reported average is close to Qwen2.5 1.5B, with task-specific strengths in reasoning and mathematics and lower scores on several knowledge, code, and instruction-following measures.

The energy model assigns lower arithmetic costs to INT8 than FP16: 0.007pJ versus 0.16pJ for addition and 0.07pJ versus 0.34pJ for multiplication at 7nm. These coefficients support operation-energy estimates and do not measure laptop power consumption.

4.2 Comparison with Post-training Quantized Models

The PTQ comparison evaluates Qwen2.5 1.5B in bf16 and after GPTQ or AWQ, the two evaluated methods for quantizing weights to 4-bit integers. Across MMLU, GSM8K, and IFEval, Table 2 reports averages of 55.72 for bf16 Qwen2.5, 52.15 for GPTQ, 51.17 for AWQ, and 55.01 for BitNet.

• BitNet uses 0.4GB of non-embedding memory and int8 activations, compared with 0.7GB and bf16 activations for both 4-bit Qwen2.5 variants.
• BitNet exceeds both quantized variants on GSM8K and IFEval but trails them on MMLU: 53.17 versus 58.06 for GPTQ and 57.43 for AWQ. Its higher reported average therefore depends on the evaluated task mix.

BitNet’s reported three-task average of 55.01 exceeds GPTQ’s 52.15 and AWQ’s 51.17 while using 0.4GB rather than 0.7GB of non-embedding memory. Both quantized Qwen2.5 variants retain higher MMLU scores, so the aggregate advantage depends on the task mix.

4.3 Comparison with Open-weight 1-bit Models

The near-1-bit comparison includes native Bonsai 0.5B and OLMo-Bitnet 1B, together with post-training-quantized Falcon3-1.58bit 7B and Llama3-8B-1.58 8B. Across the 11 listed benchmarks, Table 3 reports averages of 60.68 for BitNet, 41.06 for Bonsai, 32.22 for OLMo-Bitnet, 50.76 for Falcon3, and 49.75 for Llama3.

• BitNet has the highest score on 9 of the 11 tasks. Exceptions are HellaSwag, where Llama3 scores 68.56 versus 68.44, and TruthfulQA, where OLMo-Bitnet scores 49.05 versus 45.31.
• The results show stronger performance among the evaluated checkpoints, including larger quantized models. Differences in model size, data, and training prevent attributing the full gap solely to native low-bit training.

BitNet leads 9 of the 11 listed tasks and has the highest reported average among the evaluated native and post-training-quantized near-1-bit models. The 7B and 8B competitors score lower overall, although the comparison does not control their data or training procedures.

5 Inference Implementation

Inference Implementation addresses W1.58A8, the combination of 1.58-bit weights and 8-bit activations, through specialized implementations for graphics processing units (GPUs) and CPUs. The code is released at https://aka.ms/bitnet to provide computation routines for a numerical format that standard libraries generally do not optimize.

5.1 GPU Inference

GPU inference uses a custom CUDA kernel, a GPU computation routine, that packs four ternary weights into one int8 value for storage. It loads packed weights from High Bandwidth Memory (HBM), the GPU’s main high-bandwidth storage, into faster on-chip shared memory, unpacks them, and computes with 8-bit activations.

• The pack-store-load-unpack-compute strategy reduces weight-transfer bandwidth while enabling custom ternary computation. The report refers to Ladder, a tensor-compilation framework, for further implementation details.
• The authors identify limited native W1.58A8 support on commodity GPUs as a constraint. This report provides no numerical GPU latency or throughput comparison.

5.2 CPU Inference

CPU inference uses bitnet.cpp, the official C++ reference library with computation routines tailored to BitNet’s quantization scheme. The authors describe inference as lossless relative to the training procedure, meaning numerically consistent with the trained quantized computation.

• The implementation targets CPUs in edge devices, laptops, and servers. The measured 29ms time per output token applies to the specific laptop and 8-thread configuration described in the evaluation appendix.

6 Conclusion

The Conclusion presents the 2-billion-parameter, 4-trillion-token model as evidence that extreme weight quantization can be incorporated directly into training while retaining competitive small-model performance. The release includes model weights and specialized GPU and CPU inference implementations.

7 Future Directions

Future Directions identifies larger native models, hardware optimization, longer sequences, broader language coverage, multimodal integration, and theoretical explanations for low-bit training as open research areas. These are proposed extensions, not demonstrated capabilities of the released model.

• Scaling Laws and Larger Models proposes testing 7B, 13B, and larger parameter scales to determine whether competitive performance persists.
• Hardware Co-Design and Optimization proposes improved computation routines and hardware designed around low-bit operations and data movement. Suggested future gains are not measured here.
• Extended Sequence Length proposes better support for long documents and long reasoning traces. Multilingual Capabilities notes that current training data is primarily English-centric.
• Multimodal Integration proposes combining different input types, such as text and images. Theoretical Understanding seeks explanations through training dynamics, loss landscapes, and learned representations.

Appendix

• A Open-weight Baselines identifies the evaluated full-precision checkpoints as meta-llama/Llama-3.2-1B-Instruct, google/gemma-3-1b-it, Qwen/Qwen2.5-0.5B-Instruct, Qwen/Qwen2.5-1.5B-Instruct, Qwen/Qwen2.5-3B-Instruct, HuggingFaceTB/SmolLM2-1.7B-Instruct, and openbmb/MiniCPM-2B-dpo-bf16. Quantized and near-1-bit baselines are Qwen/Qwen2.5-1.5B-Instruct-GPTQ-Int4, Qwen/Qwen2.5-1.5B-Instruct-AWQ, deepgrove/Bonsai, NousResearch/OLMo-Bitnet-1B, tiiuae/Falcon3-7B-Instruct-1.58bit, and HF1BitLLM/Llama3-8B-1.58-100B-tokens.
• B Evaluation Pipeline Details uses evalplus for HumanEval+, a customized math-evaluation-harness for MATH-500, the official LLM Judge codebase for MT-Bench, and lm-evaluation-harness for the remaining tasks. These are the evaluation toolkits used to run the benchmarks. Generative tasks use chat prompts, while other tasks use toolkit defaults; CPU latency and estimated arithmetic energy follow the hardware and numerical conditions stated in Main Results.
• The first-page release links distinguish packed inference weights at https://huggingface.co/microsoft/bitnet-b1.58-2B-4T, bf16 training master weights at https://huggingface.co/microsoft/bitnet-b1.58-2B-4T-bf16, and the GGUF-format artifact, a deployment file for bitnet.cpp, at https://huggingface.co/microsoft/bitnet-b1.58-2B-4T-gguf. Source code is available at https://github.com/microsoft/BitNet and the demonstration at https://aka.ms/bitnet-demo.

Brief Thoughts

The strongest deployment evidence combines competitive task scores, reduced non-embedding weight memory, and a measured CPU decoding result. Assessing a deployment still requires total runtime memory and latency measurements on the intended hardware. The comparisons include no full-precision counterpart trained with the same architecture, data, and token budget. They demonstrate the released system’s performance without isolating individual architectural or training contributions. Recommendations concerning learning rates, loss summation, and additional SFT epochs lack quantitative ablations, comparisons isolating individual choices, and several exact hyperparameters. This limits reproducibility of the training recipe. Estimated arithmetic energy does not establish whole-device energy savings, and the single CPU benchmark leaves GPU performance and other hardware and serving workloads unmeasured.