TiDAR: Think in Diffusion, Talk in Autoregression | Summary

Contents

Summary
1. Introduction
2. Background and Related Work
- 2.1. Diffusion Language Models
- 2.2. Speculative Decoding
3. Method
Experimental Setup
4.2. Main Results
- 4.2.1. Generative Task Evaluation
- 4.2.2. Likelihood Task Evaluation
4.3. Efficiency Benchmarking
4.4. Ablation Studies
5. Limitations
6. Conclusions
Appendix
Brief Thoughts

This article explains the key points of TiDAR: Think in Diffusion, Talk in Autoregression.

2025-11-12 (arXiv)
Liu, Jingyu, Dong, Xin, Ye, Zhifan, Mehta, Rishabh, Fu, Yonggan, Singh, Vartika, Kautz, Jan, Zhang, Ce, Molchanov, Pavlo.
NVIDIA, University of Chicago, Georgia Institute of Technology
Paper
Project Page

Summary

TiDAR addresses how to generate multiple language tokens in parallel while retaining the quality of autoregressive (AR) models, which predict each token from preceding tokens. On a single NVIDIA H100 graphics processing unit (GPU) with batch size = 1, it reports 4.71× higher decoding throughput than Qwen2.5 1.5B and 5.91× higher throughput than Qwen3 8B, with competitive downstream task scores.
The same model drafts tokens through one-step masked diffusion, which predicts masked positions in parallel, and verifies earlier drafts through causal attention, which restricts access to preceding positions, in a single forward pass. Structured attention masks also support an exact key-value (KV) cache, which stores reusable attention states for accepted prefix tokens.
Continual pretraining, further language-model training of an existing model, uses 50B tokens for TiDAR 1.5B and 150B tokens for TiDAR 8B; their generative evaluations average 7.45 and 8.25 tokens per network function evaluation (NFE), meaning one model forward pass. Full masking improves the measured quality-efficiency trade-off, but quality remains below an equally trained AR model in the controlled comparison, and the headline throughput measurements cover batch size = 1.

1. Introduction

Small-batch AR decoding is memory-bound: loading model weights and cached attention states dominates latency. Figure 1 profiles Qwen3-32B on H100 and identifies “free token slots,” additional positions that incur little latency before computation becomes the bottleneck.

Parallel diffusion sampling treats tokens generated in the same step as conditionally independent, whereas AR factorization conditions each prediction on preceding tokens.
TiDAR uses parallel predictions as proposals and causal verification to determine the output, while simultaneously preparing drafts for the next step. Low overhead depends on fitting these active positions into the available free token slots.

The paper connects TiDAR to diffusion language models and speculative decoding, a procedure that proposes candidate tokens and verifies them against a target model. Table 1 compares drafting capacity, sharing with the base model, parallel token drafting, and overlap between drafting and verification.

TiDAR uses the full model for drafting and performs both drafting and verification in one forward pass; the table characterizes architectural properties rather than measured speed.

2.1. Diffusion Language Models

Diffusion language models can predict several positions simultaneously, but independent within-step sampling can reduce coherence and correctness. The paper cites a 10% accuracy drop on GSM8K, a mathematical word-problem benchmark, for Dream-7B when increasing entropy-based decoding, which uses prediction uncertainty to guide token selection, from 1 to 2 tokens per step; this result comes from prior work.

Bidirectional attention, which lets positions attend to both earlier and later positions, complicates exact reuse of cached states. Related approaches separate clean-token encoding from noisy-token decoding, correct training-sampling mismatch, or selectively cache states.
Block Diffusion uses causal dependence across blocks and bidirectional attention within each block, supporting exact caching while retaining the within-block quality-parallelism trade-off.

2.2. Speculative Decoding

Conventional speculative decoding trades inexpensive drafting against candidate acceptance: a weak drafter can reduce the number of accepted tokens enough to erase its speed advantage. Medusa adds prediction heads, while EAGLE and DeepSeek-V3 Multi-Token Prediction (MTP), which predicts future tokens using additional modules, reuse base-model hidden states.

The paper argues that these methods limit drafting capacity or retain sequential drafting, whereas TiDAR shares the full backbone and overlaps parallel drafting with verification.
Conventional speculative decoding preserves the base model’s target distribution; adapting the backbone into TiDAR changes the model being sampled.

3. Method

TiDAR combines causal and blockwise bidirectional attention within one sequence. Its method consists of dual-mode backbone training, parallel self-speculative generation, in which one model drafts and verifies its own proposals, and optimizations for mask reuse and exact caching.

Figure 2 illustrates how an accepted prefix selects one of the next-step proposals computed during the same forward pass.
Figure 3 shows the training and decoding masks that separate causal verification from bidirectional drafting.

3.1. Diffusion-AR Dual-mode Backbone Training

Training appends a fully masked copy of the input, doubling sequence length. Clean tokens use causal attention with labels shifted by one position for next-token prediction (NTP), while masked positions use within-block bidirectional attention with labels aligned to their input positions.

The causal prefix permits AR loss without label leakage, which would occur if a position could attend to its future target.
Full masking supplies a diffusion loss at every masked position, equalizes the number of loss terms across modes, and matches one-step drafting at inference.
The objective is \(\mathcal{L}_{TiDAR}(\theta)=\frac{1}{1+\alpha}\left[\sum_{i=1}^{S-1}\frac{\alpha}{S-1}\mathcal{L}_{AR}(x_i,x_{i+1};\theta)+\sum_{i=1}^{S-1}\frac{1}{S-1}\mathcal{L}_{Diff}([mask],x_i;\theta)\right]\), where S is input sequence length and the two losses are cross-entropy losses, which penalize low predicted probability for the target token. The equation’s accompanying text specifies \(\alpha\in[0,1]\), although Figure 6 also evaluates an AR loss weight of 1.2.

3.2. Fully Parallelizable Self-Speculative Generation

Generation begins by causally encoding the prompt and drafting a block from masked positions. Each later forward pass verifies the previous draft and computes next-step proposals conditioned on the possible accepted prefixes; after verification, the matching proposal is selected.

The paper describes rejection sampling as checking candidate tokens against the verifier’s predictions and retaining the accepted prefix.
One-step diffusion avoids repeated denoising passes, which progressively replace masked positions with predictions. Draft quality matters through acceptance rather than direct use of an independently sampled final block.
The architecture example uses draft length 3 and accepted length 2, showing verification and proposal selection together.

3.3. Training and Inference Optimization

Most experiments use α = 1 to balance AR and diffusion losses. During decoding, the implementation reorders draft positions and prefix positions so Flex Attention, the attention implementation used here, can reuse slices of an initialized mask instead of rebuilding it each step.

The decoding mask has size (q_len, q_len + max_sequence_len), with \(\mathrm{q\_len}=\mathrm{block\_len}\cdot(1+\mathrm{block\_len})\). The number of active positions therefore grows quadratically with draft length.
Causally computed states are cached, and states corresponding to rejected tokens are evicted; accepted states do not need recomputation.
Figure 4 evaluates whether tokens per forward translate into wall-clock throughput. Appendix B applies mask slicing to prompt prefill, the initial prompt-processing pass, as shown in Figure 7.

Experimental Setup

The experiments initialize models from Qwen2.5 1.5B, Qwen3 4B, and Qwen3 8B, and evaluate coding, mathematics, factual knowledge, and commonsense reasoning with lm_eval_harness, a language-model evaluation framework. Main TiDAR results cover 1.5B and 8B, while ablations primarily use 1.5B.

TiDAR 1.5B receives 50B continual-pretraining tokens with block sizes 4, 8, and 16; TiDAR 8B receives 150B tokens with block size 16. Training uses H100s and a global batch of 2M tokens with distributed data parallelism, which distributes training batches across devices.
Settings include cosine learning-rate scheduling, max_lr = 1e−5, min_lr = 3e−6, 1% warm-up, maximum sequence length 4096, distributed Adam optimization, and BFloat 16 numerical precision. The 8B model uses gradient checkpointing, which recomputes intermediate activations to reduce training memory.
Baselines include Llama3.2, SmolLM2, Qwen2.5, Qwen3, Dream, LLaDA, and a separately trained Block Diffusion model. Appendix Table 6 specifies task prompts, generation limits, and scoring conditions.

4.2. Main Results

Main results distinguish generative quality from likelihood-based answer scoring, which ranks candidate responses by their model-assigned probabilities. The former measures generated code and mathematical answers; the latter uses TiDAR’s causal mode without diffusion likelihood estimation.

4.2.1. Generative Task Evaluation

Table 2 reports a six-task generative average of 44.03% for TiDAR 1.5B, compared with 41.64% for its Qwen2.5 1.5B initialization and 38.41% for Block Diffusion 1.5B. TiDAR averages 7.45 tokens per NFE, but gains vary by task: HumanEval, a code-generation benchmark, improves from 35.98% to 43.29%, while GSM8k falls from 54.74% to 53.90%.

TiDAR 8B using diffusion predictions for verification averages 65.31%, versus Qwen3 8B at 68.09%, Dream 7B at 58.74%, and LLaDA 8B at 41.78%; its average output is 8.25 tokens per NFE.
Using AR predictions instead yields 63.90% and 8.23 tokens per NFE for TiDAR 8B. Diffusion verification improves five of the six task scores and leaves one unchanged.
The non-TiDAR diffusion baselines decode one token per forward to prioritize quality. The Block Diffusion baseline in the larger-model comparison is 4B, so its comparison with TiDAR 8B does not isolate architecture at matched size.

4.2.2. Likelihood Task Evaluation

TiDAR computes response likelihood with causal attention in one NFE, following the AR scoring procedure. Diffusion baselines estimate likelihood through Monte Carlo (MC) sampling, which averages scores over sampled corruptions; Table 3 identifies models using 128 MC steps.

The factual-knowledge and commonsense averages are 64.43% for TiDAR 1.5B versus 65.16% for Qwen2.5 1.5B, and 75.40% for TiDAR 8B versus 74.25% for Qwen3 8B.
The table covers MMLU, a factual-knowledge benchmark, and ARC, Hellaswag, PIQA, and Winogrande, benchmarks grouped as commonsense reasoning in the paper. The authors acknowledge continuing debate over direct comparisons between different likelihood estimators.

4.3. Efficiency Benchmarking

Wall-clock measurements use one H100, batch size = 1, downstream generative prompts, exact caching, and native PyTorch implementations. TiDAR reports average decoding-throughput gains of 4.71× over Qwen2.5 1.5B and 5.91× over Qwen3 8B, exceeding the tested Block Diffusion and EAGLE-3 configurations.

EAGLE-3 uses Qwen3-8B Instruct, the instruction-tuned model variant, because corresponding base-model draft weights were unavailable. Its checkpoints are https://huggingface.co/AngelSlim/Qwen3-8B_eagle3 and https://huggingface.co/Tengyunw/qwen3_8b_eagle3.
The EAGLE-3 comparison therefore involves a different model variant and different quality levels. Figure 4 normalizes throughput against each configuration’s corresponding AR baseline.
The authors attribute throughput to both more accepted tokens per NFE and better conversion from tokens per forward to tokens per second. Kernels, cache management, and request scheduling remain opportunities for further optimization across all methods.

4.4. Ablation Studies

Ablations examine matched training recipes, decoding strategies, the choice of verification logits, which are token scores before selection, and full versus random masking. These experiments test individual design choices using the 1.5B setting.

4.4.1. Pareto Frontier under the Same Training Recipe

Figure 5 compares models initialized from Qwen2.5 1.5B under the same training setting and 50B-token budget. TiDAR provides a stronger quality-efficiency frontier than Block Diffusion and the original AR model, but an additionally trained AR model retains higher average quality.

A Pareto frontier consists of configurations for which neither measured objective can improve without sacrificing the other; here the objectives are task score and tokens per NFE.
Larger draft blocks increase tokens per forward without uniformly improving accuracy. The authors’ suggestion that more training data could close the remaining AR quality gap is untested.

4.4.2. Comparing TiDAR with Other Decoding Strategy

Table 4 compares TiDAR with fixed-count confidence decoding, which selects a specified number of high-confidence predictions, confidence-threshold decoding, and within-block left-to-right decoding on full-mask models. TiDAR with 8 drafts achieves 5.49 tokens per NFE, 39.94% HumanEval Avg, 52.13% MBPP Avg, and 54.74% GSM8k; MBPP is another code-generation benchmark.

Confidence > 0.6 yields 3.81 tokens per NFE but scores of 22.56%, 26.47%, and 37.60% on the same task groups.
Increasing Confidence Max decoding from 1.00 to 2.00 tokens per NFE reduces HumanEval Avg from 34.45% to 23.78%. TiDAR’s verification achieves a better measured quality-efficiency trade-off than these alternatives.

4.4.3. Sampling with AR v.s. Diffusion Prediction

AR predictions and the first position of each diffusion proposal can predict the same output position. The paper mixes their logits using argmax_i{β * logits_i^ar + (1−β) * logits_i^diff, i ∈ |V|}, where argmax selects the highest-scoring token, β controls the AR contribution, and |V| denotes vocabulary size.

Figure 6 varies β from 0.0 to 1.0 and compares AR-to-diffusion loss weights of 0.8:1.0, 1.0:1.0, and 1.2:1.0. Average scores remain approximately 47%–49%.
The ablation shows limited sensitivity to the weighting in this setting. It does not establish identical AR and diffusion predictions; Table 2 shows measurable differences between the two verification choices at 8B.

Trusting AR v.s. Diffusion Outputs for Sampling

4.4.4. How Useful is the Full Mask Strategy?

Full masking matches one-step inference, where draft positions contain no partially revealed clean tokens, and supplies denser diffusion supervision. Table 5 shows that replacing random masking with full masking raises the average score from 45.45% to 48.42% for 4 drafts and from 45.68% to 48.94% for 8 drafts.

Average tokens per NFE rise from 3.42 to 3.47 for 4 drafts and from 5.22 to 5.49 for 8 drafts.
For 8 drafts, HumanEval Avg increases from 33.85% to 39.94%, while GSM8k increases from 54.43% to 54.74%. The larger quality improvements occur in coding.

Quality-efficiency Improvement from Full Masking

5. Limitations

The throughput evidence focuses on batch size = 1 and does not establish equivalent gains for heavily batched serving. The authors propose adjusting draft length for other compute profiles, but provide no corresponding large-batch throughput evaluation.

Appending mask tokens doubles training sequence length, and efficient long-context extension remains future work. The reported maximum training sequence length is 4096.
Measurements use native PyTorch with Flex Attention; custom kernels and scheduling are proposed improvements rather than demonstrated results.
TiDAR requires backbone adaptation and does not preserve the original initialization’s target distribution in the manner of conventional speculative decoding.

6. Conclusions

The paper concludes that jointly computing one-step diffusion drafts and causal verification can improve generation efficiency while maintaining competitive downstream quality. Its central evidence is 7.45 and 8.25 tokens per NFE and 4.71× and 5.91× decoding-throughput gains at 1.5B and 8B, respectively.

The throughput gains apply to the evaluated models, tasks, H100 hardware, and batch size = 1. Larger batches, longer contexts, and more optimized serving implementations remain unevaluated.

Appendix

A. Evaluation Task Configuration specifies lm_eval_harness version 0.4.8 and standard HumanEval/HumanEval+ post-processing. HumanEval and HumanEval+ use 0 prompt examples and generation length 512; MBPP and MBPP+ use 3 examples and length 512, all scored by Pass@1, which measures whether one generated solution passes the tests. GSM8K-CoT, the mathematics task with chain-of-thought prompting that includes intermediate reasoning, uses 8 examples, length 256, and strict-match; Minerva Math uses 4 examples, length 512, and exact-match.
Table 6 also specifies 5 examples for MMLU and Winogrande and 0 for ARC-Easy, ARC-Challenge, Hellaswag, and PIQA; these tasks use accuracy or normalized accuracy, which adjusts candidate scoring for response length. B. Inference Prefill Mask reorders draft and prefix positions and slices a mask initialized with size (max_seq_len + block_size, max_seq_len + block_size), allowing reuse across prompt lengths while retaining causal prefix attention and bidirectional draft attention.

Brief Thoughts

The main contribution is the overlap of full-backbone drafting and causal verification, supported by wall-clock measurements and the controlled masking ablation. Deployment assessment should account for the quality trade-off: TiDAR 8B trails Qwen3 8B on the generative average, and the matched-training experiment retains an AR quality gap. The reported speedups apply to single-request H100 decoding. Larger-batch measurements and optimized implementations of all baselines would be needed to assess the advantage under other serving conditions.