Next-Concept Modeling: Fast, Parallel Coarse-to-Fine Text Generation | Summary

This article explains the key points of Next-Concept Modeling: Fast, Parallel Coarse-to-Fine Text Generation.

• 2026-10-09 (arXiv)
• Kumbong, Hermann, Khare, Ishan S., Mlauzi, Iddah, Woen, Andrew, Ding, Charles, Buchanan, E. Kelly, Arora, Simran, Mirhoseini, Azalia, Ré, Christopher.
• Department of Computer Science, Stanford University, Department of Electrical Engineering, Stanford University, Together AI
• Paper
• Github
• Project Page

Read this article in Korean

Summary

• Parallel text generation can remain costly when iterative language models repeatedly process an entire sequence. At comparable generator sizes, Next-Concept Modeling (NCM) achieves better generative perplexity, a reference model’s score of generated text, than the strongest diffusion baseline on One Billion Word (LM1B) while running 21× faster under the paper’s timing setup.
• NCM learns a hierarchy of discrete representations and generates its levels sequentially, predicting all codes within each level in parallel. It also reports competitive translation and strong summarization and Sudoku results; its fixed generation schedules and reconstruction errors remain limitations.

The comparisons relate speed to generation quality and show that the Sudoku advantage persists as clue counts fall.

1 Introduction

Diffusion language models produce multiple tokens through iterative denoising, but fewer passes can reduce quality and each pass commonly processes the full sequence. NCM reduces the number and cost of passes by generating shorter representations at early levels. For Diffusion language models, see TiDAR: Think in Diffusion, Talk in Autoregression | Summary. For Diffusion language models, see Introspective Diffusion Language Models | Summary. For Diffusion language models, see Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decoding | Summary.

NCM predicts a whole code level at once, with early levels shorter than the output, reducing sequential passes and early computation.

2 Background & Related Work

NCM adapts residual quantization, which successively represents approximation errors, and next-scale prediction, which generates one resolution at a time, from image generation to text [20, 21]. Its hierarchy represents a whole sequence at increasing resolutions.

One translation develops from repetitive partial text into a coherent final sentence; the complete trajectory shows intermediate quality is not monotonic.

3 Next-Concept Modeling

The two-stage method trains a multiscale residual vector-quantized variational autoencoder (VQ-VAE), an encoder-decoder that represents text with discrete codes, then freezes it and trains a transformer to generate those codes. The paper uses “concept” for information carried by coarse levels; the decoder converts the completed representation back to text.

The first stage learns to reconstruct text from discrete levels; the second predicts those levels using reconstructions of preceding levels.

3.1 Preliminaries

Vector quantization replaces each latent vector with its nearest entry in a learned codebook. Residual quantization adds further codebook entries to approximate remaining error, while the multiscale version maps progressively longer code sequences back to the full sequence length. For Vector quantization, see Scaling Image Tokenizers with Grouped Spherical Quantization | Summary.

3.2 Formulation

For a sequence of length L, the tokenizer produces K code levels of increasing lengths, ending at L. The generator conditions each level on the cumulative reconstruction of preceding levels and samples its codes independently in parallel, requiring K sequential generation steps.

3.3 Multiscale Residual Tokenization

Each level downsamples the remaining latent error, selects nearest codebook entries, and adds their reconstructed contribution to the running representation. Nested dropout retains a sampled prefix of levels during tokenizer training, encouraging early levels to support useful partial reconstructions.

3.4 Coarse-to-Fine Generation

The frozen tokenizer supplies targets for a transformer trained on all levels in one forward pass; at inference, levels are generated in order. Training replaces some conditioning codes with other entries from the same codebook to reduce sensitivity to generated-code errors, while prompt-conditioned tasks use a prefix attention pattern.

4 Experiments

The authors train a separate tokenizer and generator for each of six datasets and assess reconstruction, unconditional and conditional generation, Sudoku, and scaling. Most generators have approximately 100M parameters; the Sudoku generator has 25M.

4.1 Tokenizer Quality

Held-out exact token reconstruction is 99.99% on OpenWebText, 99.86% on LM1B, 99.79% on XSum, 98.79% on WMT14, and 100.00% across the three Sudoku difficulties. Mean codebook utilization, the percentage of entries used across levels, ranges from 61.89% to 90.12%; errors can change words or entities.

WMT14 has the lowest listed reconstruction accuracy, while all three Sudoku settings recover every held-out token.

4.2 Unconditional Generation

Generative perplexity is reported with unigram entropy, a measure of token diversity. On LM1B, NCM scores 93.05 at 12 function evaluations, or generator calls, versus DUO’s 97.30 at 128, with entropy 4.31 versus 4.30; on OpenWebText NCM scores 83.12 versus DUO’s 77.69, with entropy closer to the data.

The curves compare quality against function evaluations or time, with entropy labels providing context for sample diversity.

4.3 Conditional Generation

On XSum, unguided NCM obtains ROUGE-1/2/L scores of 35.0/12.8/28.0 at 8 generator calls, versus AURORA-LM’s 34.7/12.2/26.6 at 128; ROUGE measures overlap with reference summaries. On WMT14 German–English, its BLEU translation score is 22.6 at 11 calls versus E2D2’s 24.8 at 64.

Unguided NCM leads the listed XSum systems on all three ROUGE measures, while E2D2 achieves higher unguided WMT14 BLEU.

4.4 Reasoning

For Sudoku with 40, 35, and 30 clues, NCM solves 99.5%, 96.5%, and 86.1% of boards at 12 function evaluations. The strongest reported baseline, FMLM+, solves 97.7%, 92.8%, and 81.3% at 16 evaluations.

NCM’s largest absolute advantage over FMLM+ appears on hard boards, at 86.1% versus 81.3%.

4.5 Scaling

On LM1B, 97M, 317M, and 725M generators sharing a frozen tokenizer show improved generative perplexity as size increases at a fixed generator training-token budget. Minimum Bayes risk decoding, which selects the candidate most similar on average to other sampled candidates, improves XSum and WMT14 scores as more candidates are generated.

The left panel compares model sizes at fixed generator token exposure; the right panels show gains from selecting among multiple samples.

4.6 Efficiency

Timing uses 1,024 generated sequences, batch size 16, and one NVIDIA H100 SXM 80GB GPU: NCM is 21× faster than DUO on LM1B at better perplexity and 31× faster than AURORA-LM on XSum at comparable ROUGE. LM1B tokenizer and generator training together expose 13.36B full-length token positions versus 66B for the baselines; this measures token exposure, not training floating-point operations.

The bars compare training-token exposure and annotate generative perplexity; they do not measure training floating-point operations.

4.7 Qualitative Samples

The qualitative examples include fluent translations and context-related summaries, but also factual substitutions: one generated summary names James Taylor-Fletcher where the reference says Gary Taylor-Fletcher and changes the loan duration. Such examples limit what overlap scores alone establish about factual fidelity.

The examples show plausible language alongside factual mistakes, including an altered name and loan duration in the summary.

5 Conclusion

The authors conclude that a learned hierarchy provides a short parallel generation path and competitive quality with fewer function evaluations and training tokens than the evaluated diffusion baselines. The reported scaling evidence extends to a 725M generator on LM1B.

Appendix

• A derives an evidence lower bound (ELBO), a lower bound on text log-likelihood, for deterministic nearest-neighbor tokenizer assignments. With a frozen tokenizer and decoder, clean conditioning, and equal code-position weights, minimizing generator loss maximizes the ELBO over generator parameters; corruption and nonuniform weighting modify the objective.
• B and C specify target lengths of 1,024 for OpenWebText, 128 for LM1B and TinyStories, 48 for XSum, 64 for WMT14, and 89 for Sudoku. The tokenizer has respectively 32, 12, 12, 8, 11, and 12 levels; XSum and WMT14 generators train first on unpaired text and then on source–target pairs. Experiments use seed 42, BF16 mixed precision, and torch.compile.
• D identifies published results and checkpoints for unconditional, conditional, and Sudoku baselines; conditional baselines without released checkpoints use randomly initialized models for timing only. E reports NCM’s mean LM1B perplexity of 83.24 across four additional reference models versus DUO’s 92.52, although DUO scores better under Mistral-7B alone; it also tabulates unconditional scores and entropy.
• F finds that more hierarchy levels improve TinyStories perplexity with diminishing returns, and that the level schedule matters. Block-diagonal attention achieves perplexity 15.41 in 5.65 seconds versus 19.63 in 7.90 seconds for block-causal attention under the stated 1,024-sample setup; a pretrained T5-small context encoder gives small ROUGE gains with 35.3M additional parameters. Loss weighting has little effect on perplexity up to the tested equal-level weighting, moderate code corruption improves quality, and guidance or candidate selection can improve sampling scores.
• G documents tokenizer errors involving entities, numerals, markup, and word boundaries. H shows selected TinyStories examples where fixed coarse codes preserve broad themes while finer codes change details, and where mixing or perturbing coarse codes alters the generated stories.
• I notes fixed hierarchy depth and lengths, separately trained tokenizers, limited model scale, and occasional transition artifacts. J provides translation, XSum, TinyStories, LM1B, and OpenWebText samples; its translation trajectory drops from BLEU 51.7 at level 8 to 38.9 at level 9 before reaching 77.8 at level 11, and its hard Sudoku trajectory is completed at level 12.

Brief Thoughts

Wall-clock measurements, entropy reporting, and tokenizer-inclusive token counts make the efficiency comparison more informative than function evaluations alone. Generalization to larger models and tasks remains uncertain: the main generators are roughly 100M parameters, each dataset has its own tokenizer, and the examples expose entity and factual errors.