From Bytes to Ideas: Language Modeling with Autoregressive U-Nets | Summary

Contents

Summary
1 Introduction
2 Method
3 Experimental Results
4 Related Work
5 Conclusion
- Limitations and further work
Appendix
Brief Thoughts

This article explains the key points of From Bytes to Ideas: Language Modeling with Autoregressive U-Nets.

2025-06-17 (arXiv)
Videau, Mathurin, Idrissi, Badr Youbi, Leite, Alessandro, Schoenauer, Marc, Teytaud, Olivier, Lopez-Paz, David.
FAIR at Meta, TAU, INRIA and LISN, CNRS & Université Paris-Saclay, INSA Rouen Normandy, LITIS, Rouen, France
Paper
Github

Summary

Fixed text tokenization, the division of text into input units, restricts language models to a predetermined processing and prediction granularity. With pretraining compute carefully controlled, Autoregressive U-Net (AU-Net), which compresses raw bytes into coarser representations and expands them for next-byte prediction, broadly matches a strong Byte-Pair Encoding (BPE) baseline, whose vocabulary is built by repeatedly merging frequent byte sequences.
AU-Net learns contextual representations at byte, word, two-word, and four-word scales while retaining predefined splitting rules. With training data equivalent to 370B LLaMa 3 tokens, AU-Net 3 achieves 72.9 ± 0.9 on Hellaswag, a multiple-choice benchmark, versus 70.2 ± 0.9 for BPE; Table 2 reports 4e21 floating-point operations (FLOPs), a measure of computation, for both, although the detailed configuration table gives different costs.
The byte-based architecture improves selected character manipulation and low-resource translation results, but its advantages depend on the task and writing system. Deeper hierarchies increase parameter counts and reduce training throughput; compute-controlled experiments still show weaker results on knowledge and mathematical reasoning benchmarks at some scales.

1 Introduction

The introduction distinguishes text splitting from embedding, the mapping of input units to vectors. Independent lookup-table embeddings obscure shared character structure, while a single fixed tokenization prevents later layers from changing the processing granularity.

AU-Net replaces large token lookup tables with contextual representations learned directly from bytes.
Its deeper stages guide predictions over multiple words without an additional prediction loss. Skip connections, which carry earlier representations directly to later layers, preserve fine detail.
The contributions combine hierarchical representations, freedom from a fixed word vocabulary, competitive downstream performance, measured training throughput, and architecture-specific optimization scaling.

2 Method

The method combines a hierarchy that shortens and then restores sequences, a splitting function that determines contraction boundaries, and compute-based comparisons across model scales. The architecture learns segment representations; the experiments retain manually specified segmentation rules.

2.1 Autoregressive U-Net

AU-Net uses decoder-only Transformer layers, which use attention to combine information from preceding inputs, along contracting and expanding paths. Contracted stages process shorter sequences with larger hidden dimensions, the widths of their representation vectors, while skip connections carry detailed representations into the expanding path; the longest byte-level stage restricts attention to a sliding window.

2.1.1 Pooling and Upsampling: pooling reduces sequence length by selecting contextual vectors at splitting boundaries and projecting them to the next stage’s dimension. Multi-Linear Upsampling restores finer positions by duplicating a coarse vector across the following segment and applying a distinct linear transformation at each relative position.
Using the following segment preserves the prediction offset: a representation of “SAT ” helps predict “ON ”, and the deepest stage of a four-stage model helps predict the next four words.
2.1.2 Generation: training processes all stages in parallel. During autoregressive generation, which predicts successive bytes from preceding bytes, deeper stages activate according to their pooling boundaries, requiring the latest output vector from each stage to be cached.

Three-stage AU-Net with byte, word, and two-word representations, contracting and expanding paths, and skip connections.

Boundary-based pooling and position-specific expansion into the following segment.

2.2 Splitting Function

A splitting function must be stable to rightward insertion: appending bytes must leave earlier pooling decisions unchanged. The experiments use regular expressions, rules for matching text patterns, to identify word boundaries, then pool after two words and four words, or at sentence endings.

The framework permits variable segment lengths and alternative splitting functions, but the evaluated rules rely on spaces and suit Latin scripts.
Robust segmentation for languages without clear delimiters remains future work.

2.3 Evaluating on different scales

The paper measures scale by total pretraining compute, C = Fmodel/input-unit × Ninput-unit, where F is forward-and-backward compute per input unit and N is the number of training units. For a decoder-only Transformer, it estimates per-unit compute as Fmodel/input-unit = 6Nparams(no-embed) + 6dLS, where Nparams(no-embed) counts non-embedding parameters, d is representation width, L is layer count, and S is sequence length.

The data-to-model ratio is γinput-unit = Ninput-unit / Fmodel/input-unit. On DataComp-LM (DCLM), the pretraining corpus, LLaMa 3 token sequences are approximately k ≈ 4.56 times shorter than byte sequences, giving γbyte = k²γtoken.
AU-Net’s per-byte compute sums stage costs after division by each stage’s contraction factor. Deeper stages process fewer positions, allowing more parameters within a comparable computation budget.
Small-model searches retain configurations within 1% of the best validation loss, a prediction-error measure on held-out data, before fitting batch-size and learning-rate curves.

The fitted AU-Net rules are BSZ_AU-Net(C) = 0.66C^0.321 and LR_AU-Net(C) = 6.6 × C^−0.176, where BSZ denotes batch size, the amount of input processed per update, and LR denotes learning rate, the optimization step-size setting. The corresponding BPE rules are BSZ_BPE(C) = 29.9C^0.231 and LR_BPE(C) = 19.3 × C^−0.177.

The different batch-size exponents mean that a constant byte-to-token conversion cannot reproduce both fitted batch-size curves across compute budgets.

3 Experimental Results

The experiments examine equal training-data budgets, performance as computation increases, and multilingual and character manipulation tasks. Equal-data tables permit different computation costs, whereas the scaling study compares performance against computation directly.

3.1 Experimental Setup

All experiments use DCLM, described as roughly 4T tokens under GPTNeoXTokenizer, the tokenizer used to count the corpus, and predominantly English, with little code or mathematics. Baselines include a LLaMa 3 BPE Transformer, a byte-level Transformer, and byte-level Mamba, a sequence model based on selective state spaces that update a running representation of the input; 273B training bytes correspond to approximately 60B LLaMa 3 tokens.

Batch size and learning rate are tuned using models from 25M to 500M parameters, then extrapolated to larger budgets.
Multiple-choice answers minimize negative log-likelihood, a score that decreases as an answer becomes more probable, normalized by character count; open-ended tasks use generated answers. The paper describes evaluation as zero-shot, meaning without prompt examples, while acknowledging that some prompts contain a few examples.
Tables report bootstrap 95% confidence intervals where applicable, estimating uncertainty through resampling. Training throughput is measured in bytes per second per graphics processing unit on H100 80GB hardware.

The implementation uses sequence packing, which combines examples into training sequences, Fully Sharded Data Parallelism (FSDP), which distributes model state across devices, and torch.compile, a model compilation facility. To compile the variable-length hierarchy, each stage has a fixed maximum sequence length: longer outputs are truncated and shorter outputs padded.

These length limits allow a static computation graph, whose operation shapes remain fixed for compilation, while retaining variable pooling boundaries.

3.2 Equal Data Budget Results

At the nominal 1B scale and 60B token-equivalent data, AU-Net 3 improves Hellaswag from 63.6 ± 1.0 to 67.4 ± 0.9 at the same rounded 7e20 FLOPs as BPE in Table 2. Its actual parameter count is 2.5B versus 1.8B for BPE, and throughput falls from 210k to 180k bytes per second per device.

At 370B token-equivalent data, AU-Net 3 improves ARC-Challenge, a multiple-choice reasoning benchmark, from 38.5 ± 2.8 to 43.3 ± 2.8, and TQA, a knowledge-intensive answer-generation benchmark, from 37.2 ± 0.9 to 39.1 ± 0.9. Table 2 rounds both models to 4e21 FLOPs; Table 10 lists 4.0e21 for AU-Net 3 and 3.6e21 for BPE.
AU-Net 4 reaches 31.7 ± 0.7 on MMLU, a knowledge benchmark evaluated by answer accuracy, versus 27.0 ± 0.7 for BPE, but uses 5e21 rather than 4e21 FLOPs in Table 2.
At 60B tokens, AU-Net 4 falls to 15.5 ± 0.7 on TQA versus 26.3 ± 0.8 for BPE, showing that additional hierarchy does not produce uniform gains.

At the nominal 8B scale with 200B token-equivalent data, AU-Net 2 reaches 79.1 ± 0.8 on Hellaswag and 80.0 ± 1.6 on ARC-Easy, the easier ARC multiple-choice benchmark, versus 77.2 ± 0.8 and 74.5 ± 1.8 for BPE. The comparison uses 7.9B versus 7.5B parameters and rounded compute of 1e22 versus 9e21 FLOPs, with throughput of 41k versus 43k bytes per second per device.

TQA results are similar at this scale, 50.9 ± 0.9 versus 51.1 ± 0.9, while GSM8k, a mathematical reasoning benchmark, remains slightly lower for AU-Net.
Table 10 gives more precise compute values of 1.2e22 for AU-Net 2 and 9.5e21 for BPE. Literature comparisons also vary in training-data volume and corpus composition, limiting architectural attribution.

Downstream accuracy, parameter counts, training compute, and measured throughput for AU-Net and baseline models at several data budgets.

3.3 Scaling laws

Across the stated 1e19–1e22 FLOP range, two-stage and three-stage AU-Net generally track BPE on Hellaswag, ARC-Easy, and NQ, an open-ended question-answering benchmark. The three-stage model closes its TQA gap near 1e22 FLOPs, but both variants remain behind BPE on MMLU and GSM8K at the upper end.

The authors interpret the later rise on MMLU and GSM8K as a possible delayed performance transition; benefits beyond the tested range remain unverified.
The paper attributes generally weak mathematical reasoning partly to DCLM’s limited mathematics content, an interpretation without a controlled data ablation.
The paragraph preceding Section 3.3 states a data-to-model ratio of 2, whereas Section 3.3 states 10. The narrative does not resolve this inconsistency.

Accuracy versus training compute for BPE and two-stage and three-stage AU-Net across six downstream tasks.

3.4 Extended Evaluations

On FLORES-200, a multilingual translation benchmark, AU-Net 2 improves average translation into English from 20.9 ± 0.2 to 24.6 ± 0.2 BLEU, the reported translation-quality score for which higher is better. Translation from English improves less, from 8.0 ± 0.1 to 8.7 ± 0.1, while multilingual MMLU average accuracy rises from 31.4 ± 0.1 to 32.4 ± 0.1.

Gains concentrate in related languages using Latin scripts, consistent with the authors’ proposed transfer through shared spelling and word structure. These comparisons do not isolate that mechanism.
The benefit is not universal: Chinese MMLU drops from 33.0 ± 0.8 to 28.0 ± 0.7.
On CUTE, a benchmark of word and character manipulation, average character-level exact-match accuracy, the fraction of answers matching the target exactly, rises from 39.3 to 42.7, while word-level accuracy falls from 36.9 to 33.1.

FLORES-200 translation scores and multilingual MMLU accuracy for BPE and AU-Net 2.

Word-level and character-level exact-match accuracy on CUTE for BPE and AU-Net 2.

The related-work discussion places AU-Net among models that replace fixed vocabularies with byte processing, adaptive segmentation, or learned text compression. Its distinguishing combination is multiple variable-length processing stages, contextual boundary representations, and a contracting and expanding architecture with skip connections.

MEGABYTE, a multiscale byte model, uses fixed-size blocks and local models. The paper also contrasts AU-Net’s deeper hierarchy with Byte Latent Transformers and SpaceByte, models that process bytes using specialized splitting functions.
Earlier hierarchical U-Net-style language models used fixed pooling and smaller-scale evaluations focused mainly on perplexity, a measure of next-unit prediction quality.

5 Conclusion

The conclusion presents hierarchical byte processing as an alternative to fixed-vocabulary embeddings. The experiments support broad competitiveness under controlled compute and selected gains in character processing and multilingual transfer, with task-dependent trade-offs as hierarchy deepens.

The implementation is available at https://github.com/facebookresearch/lingua/tree/main/apps/aunet.

Limitations and further work

The evaluated system requires a predefined splitting function and does not robustly handle languages without space-based boundaries. The authors also report that FSDP struggles to overlap communication with computation at three or four stages as parameter counts increase and the number of processed positions contracts.

Learning the splitting function is proposed as future work.
Measured training throughput supports practical training comparisons, but the paper does not provide corresponding measured generation-speed results.
The predominantly English corpus and limited mathematics content constrain conclusions about multilingual coverage and mathematical reasoning.

Appendix

A Scaling Laws derives the parameter and attention computation terms and discusses a stable range of coupled batch sizes and learning rates. B Regular expression documents segmentation components for letters of 1–16 characters, numbers of 1–3 digits, punctuation groups, line breaks, and whitespace.
C Ablation, including C.1 Pooling and Upsampling, compares boundary selection, averaging, cross-attention, which aggregates information between two representation sequences, memory layers that retrieve learned embeddings, and several expansion strategies. Table 4 shows that replacing simple upsampling with Multi-Linear Upsampling raises three-stage Hellaswag accuracy from 60.6 ± 1.0 to 66.0 ± 0.9, supporting the importance of expansion design for deeper hierarchies.
C.2 Layer Allocations selects the configuration assigning 75% of layers to the deepest stage; Table 5 reports Hellaswag scores of 65.3, 66.0, and 67.4 for 25%, 50%, and 75% allocations. D Hyperparameters summarizes optimization rules, including sequence lengths of 2048 for BPE and 8192 for AU-Net, weight decay of 0.1 to regularize parameters, gradient clipping of 0.2 to limit gradient magnitude, and a cosine learning-rate schedule that decreases the rate along a cosine curve.
E CUTE Benchmark Detailed results documents the character-versus-word trade-off: spelling accuracy improves from 91.5 to 97.3 and inverse spelling from 80.6 to 91.7, while several word manipulation tasks favor BPE. F Evaluation Benchmarks Details and G List of Models appear as headings without accompanying explanatory prose on page 16; expanded benchmark results appear in Table 8 on page 15.
H Model Configuration Tables supplies architecture dimensions and layer allocations in Table 9, training costs and execution settings in Table 10, and optimizer settings in Table 11. These records contain variations from the summary defaults, including BPE sequence lengths of 4096 for some equal-data runs, different weight-decay values, and model-specific warmup counts, the initial steps over which learning rate increases.

Brief Thoughts

The clearest contribution is showing that contextual byte representations at several processing scales can remain competitive with a carefully tuned BPE baseline. The compute-scaling curves support this assessment more directly than comparisons with literature models trained under different conditions; selected equal-data gains also involve larger parameter counts or higher compute. The method retains manual segmentation, so the learned component is the representation hierarchy. The weaker Chinese results, uneven character manipulation gains, and slower training for deeper hierarchies identify concrete trade-offs. Larger-scale gains remain a hypothesis. The inconsistent stated scaling ratio and differences between rounded result-table compute and detailed configuration costs require care when reconstructing comparisons.