Quantization Meets dLLMs: A Systematic Study of Post-training Quantization for Diffusion LLMs | Summary

Contents

Summary
1 Introduction
2 Related Work
- 2.1 Diffusion Language Model
- 2.2 Network Quantization
3 Preliminary and Observation
4 Quantizing Diffusion LLM
5 Limitation and Future Work
6 Conclusion
Appendix
Brief Thoughts

This article explains the key points of Quantization Meets dLLMs: A Systematic Study of Post-training Quantization for Diffusion LLMs.

2025-08-20 (arXiv), Technical Report
Lin, Haokun, Xu, Haobo, Wu, Yichen, Guo, Ziyu, Zhang, Renrui, Lu, Zhichao, Wei, Ying, Zhang, Qingfu, Sun, Zhenan.
NLPR & MAIS, Institute of Automation, CAS, Tsinghua University, City University of Hong Kong, Harvard University, The Chinese University of Hong Kong, Zhejiang University
Paper

Read this article in Korean

Summary

Diffusion large language models (dLLMs), which generate text by repeatedly refining corrupted tokens using context from both directions, need compression methods that preserve task performance. Against the full-precision LLaDA-8B-Instruct baseline, GPTQ, a method that quantizes trained model weights, raises the reported six-task general accuracy from 65.7% to 66.0% with 4-bit weights and 16-bit intermediate activations, while the reported mathematics and code degradation is 0.6% and 6.4%, respectively.
The study applies post-training quantization (PTQ), conversion of trained model tensors to lower precision without retraining, to LLaDA-8B-Base, LLaDA-8B-Instruct, and Dream-7B-Base. It recommends 4-bit weight-only quantization and 8-bit weight-activation quantization, which reduces the precision of both weights and intermediate values; at 4-bit weight-activation precision, rotation methods, transformations that redistribute large activation values across coordinates, preserve more performance than SmoothQuant, which redistributes quantization difficulty through scaling.
Mathematics and code generation are more sensitive to aggressive quantization than general question answering, and instruction-tuned LLaDA, the variant adapted to follow instructions, generally preserves performance better than its base counterpart. These findings concern task quality: optimized low-bit inference kernels remain future work, Dream-7B weight-activation experiments are omitted, and numerical inconsistencies limit fine-grained comparisons.

1 Introduction

Quantization Meets dLLMs: A Systematic Study of Post-training Quantization for Diffusion LLMs investigates whether compression techniques developed for autoregressive language models, which generate tokens sequentially, transfer to diffusion language models. The introduction identifies activation outliers, unusually large intermediate values that dominate a tensor’s numerical range, as an obstacle and organizes the evaluation around bit-width, method, task category, and model type.

Existing acceleration work emphasizes key-value caches, which reuse stored attention states; quantization addresses tensor precision and is presented as a complementary approach.
The paper adapts existing quantization methods rather than introducing a new quantizer. Its code URL is https://github.com/FelixMessi/QDLM.

Activation distributions at selected linear-layer inputs in LLaDA-8B-Base and LLaDA-8B-Instruct, illustrating normal and massive outliers.

The related work connects diffusion-based language generation with neural network quantization. It motivates testing whether compression methods developed for sequential generation also preserve the quality of iterative text generation.

2.1 Diffusion Language Model

Masked Diffusion Models (MDMs) progressively replace tokens with [MASK] during corruption and predict missing content during generation. LLaDA-8B uses a bidirectional Transformer, an attention-based network that accesses context on both sides of a token, as its mask denoiser; Dream initializes from a pretrained autoregressive model.

The section presents quantization as complementary to caching strategies for diffusion inference. Integration with quantized caches is proposed but not evaluated experimentally.

Selected attention and FFN input activation distributions in Dream-7B-Base.

2.2 Network Quantization

PTQ applies quantization after training, whereas quantization-aware training incorporates its effects during optimization. Weight-only methods compress weight matrices: GPTQ extends Optimal Brain Quantization, a weight-quantization algorithm, to language models, and AWQ, activation-aware weight quantization, uses activation statistics and reparameterization to guide weight compression.

For weight-activation quantization, SmoothQuant transfers difficulty from activations to weights through scaling. QuaRot uses Hadamard-based rotations, structured transformations that spread large values across coordinates, while DuQuant combines outlier-aware rotations with channel permutation, a reordering of activation coordinates.
SqueezeLLM, a non-uniform quantization method, and OmniQuant, which jointly optimizes clipping thresholds and scaling factors, provide context but are not experimental baselines.

3 Preliminary and Observation

The preliminary analysis introduces masked denoising and uniform quantization before inspecting activation distributions. Its empirical component examines whether the outlier patterns that obstruct conventional language-model quantization also occur in the evaluated diffusion models.

3.1 Masked Diffusion Model

This subsection describes a binary mask that preserves observed regions while a neural network predicts Gaussian noise, the normally distributed perturbation added during corruption. It gives the training objective L_MDM = E_{x,m,ε,t}[||ε − ε_θ(x_t,m,t)||²], where x_t is the noisy input at step t, m is the mask, ε is Gaussian noise, and ε_θ is the noise predictor.

This Gaussian-noise formulation differs from the discrete [MASK]-replacement account in Section 2.1. The paper does not reconcile the formulations or establish that this objective is the training loss of the evaluated checkpoints.

3.2 Quantization

Uniform b-bit quantization maps a floating-point tensor X to integer values using X_q = clamp(⌊X/s⌉ + z, 0, 2^b − 1), where \(s=\frac{\max(\mathbf{X})-\min(\mathbf{X})}{2^b-1}\) and z = −⌊min(X)/s⌉. Here clamp restricts values to the allowed interval, ⌊·⌉ denotes nearest rounding, s is the quantization step size, and z is the zero point, the integer offset; outliers enlarge the range and therefore increase the step size at a fixed bit-width.

3.3 Outliers in dLLMs

A randomly sampled WikiText-2 calibration batch, representative inputs used to configure quantization, is passed through each model once to visualize layer inputs. The authors distinguish normal outliers, elevated channel values across tokens, from massive outliers, much larger values concentrated at fewer token positions; both occur in LLaDA and Dream.

In LLaDA, massive outliers appear at inputs to the second linear layer of the feed-forward network (FFN), the Transformer component that transforms each token representation.
The authors report that massive outliers span more tokens than in conventional language models and associate this with reduced effectiveness of global clipping, truncation of extreme values, or scaling at low precision. The study does not isolate that explanation through a causal ablation.
Figure 2 shows strong normal outliers at Dream’s FFN up-projection input, where representations are expanded, and smaller massive-outlier peaks than the illustrated LLaDA examples.

4 Quantizing Diffusion LLM

The experimental section asks four research questions (RQs): RQ1 concerns preferred bit-widths, RQ2 compares quantization methods, RQ3 examines task sensitivity, and RQ4 compares model types. Performance is assessed against each model’s full-precision baseline.

4.1 Experimental Setup

The evaluated models are LLaDA-8B-Base, LLaDA-8B-Instruct, and Dream-7B-Base. GPTQ and AWQ use group-wise weight quantization, sharing quantization parameters within groups of 128 values; SmoothQuant, QuaRot, and DuQuant are evaluated for LLaDA weight-activation quantization using 128 calibration samples, generally from WikiText-2, with Pile data used for AWQ.

General benchmarks are MMLU, a multitask language-understanding benchmark; ARC-E and ARC-C, easy and challenge question-answering subsets; Hellaswag, a sentence-completion benchmark; WinoGrande, a pronoun-resolution benchmark; and PIQA, a physical-commonsense benchmark. Mathematics uses GSM8K, a math word-problem benchmark, and Math, a mathematical problem-solving benchmark; code uses HumanEval and MBPP, program-generation benchmarks.
Question-answering and mathematics scores use accuracy. Code uses Pass@1, success of a single generated program; reported degradation relative to full precision is categorized as negligible (<1%), moderate (1–4%), or significant (>4%). These degradation percentages are distinct from percentage-point score differences.
W4A16 denotes 4-bit weights and 16-bit activations, W3A16 denotes 3-bit weights and 16-bit activations, and W8A8 and W4A4 quantize both to 8 and 4 bits, respectively. The suffix g128 denotes group size 128.
The main text specifies per-channel weights and per-token activations, assigning quantization parameters separately to each channel or token. Appendix A instead describes SmoothQuant with per-tensor weights and per-channel activations, an unresolved configuration discrepancy.

4.2 Ideal Quantization Bit Precision (RQ1)

The authors recommend 4-bit weight-only quantization because general and mathematics performance is usually better preserved than at 3 bits. For LLaDA-8B-Instruct, W4A16 GPTQ gives a reported general average of 66.0 versus 65.7 at full precision and a mathematics average of 55.6 versus 56.0, although its code average falls from 37.6 to 35.2.

W8A8 is generally less damaging than W4A4. On LLaDA-8B-Instruct, all three W8A8 methods have a reported general average of 67.0 versus the Table 2 baseline of 67.1, whereas W4A4 averages are 47.7 for SmoothQuant, 64.5 for QuaRot, and 65.4 for DuQuant.
At W4A4, SmoothQuant’s LLaDA-8B mathematics average falls from 45.5 to 1.2 and its code average from 36.2 to 0.0. Rotation methods preserve substantially higher scores but still incur large losses on these tasks.
The description of W8A8 as nearly lossless requires task-specific qualification: Table 4 reports code degradation of 6.5% for base-model SmoothQuant and 4.3% for instruction-model QuaRot.

4.3 Optimal Quantization Methods (RQ2)

GPTQ exceeds AWQ on the reported general-task averages for both LLaDA variants at both tested weight-only precisions, and on Dream’s corresponding averages. The ranking changes on other tasks: at W4A16, AWQ gives LLaDA-8B-Instruct a mathematics average of 56.2 versus GPTQ’s 55.6 and a code average of 36.4 versus 35.2.

The authors hypothesize that less prominent activation outliers weaken AWQ’s activation-based selection of the top 1% of salient weights. This mechanism is not directly tested, and GPTQ and AWQ also use different calibration datasets and sequence lengths.
The largest weight-activation advantage of rotation methods occurs at W4A4. DuQuant’s reported general degradation is 5.1% versus QuaRot’s 6.6% for the base model, and 2.5% versus 3.9% for the instruction model.
DuQuant is recommended overall, but the tables contain exceptions: instruction-model W4A4 code averages are 32.6 for QuaRot and 32.0 for DuQuant, while base-model W8A8 general averages are 64.4 and 64.3, respectively. The claim that rotation methods always outperform SmoothQuant also has exceptions at W8A8.

General-task performance of LLaDA models under weight-only quantization with group size 128.

General-task performance of LLaDA models under W8A8 and W4A4 weight-activation quantization.

4.4 Influence of Task Categories on Quantization (RQ3)

Aggressive quantization generally causes greater losses on mathematics and code than on general question answering. For base-model W3A16 GPTQ, Table 1 reports 2.7% general degradation, while Table 3 reports 15.7% mathematics degradation and 14.8% code degradation.

The authors suggest that accumulated errors in multistep reasoning and disruption of long-range dependencies or code syntax contribute to this sensitivity. These explanations remain hypotheses because intermediate representations and error propagation are not directly measured.
The reported HumanEval standard deviation is approximately 3%, which complicates interpretation of small method differences. The paper does not provide uncertainty estimates throughout its result tables.
The prose says 4-bit weight-only code losses exceed 5%, but instruction-model AWQ has a reported 3.2% loss. The overall task trend does not establish that every configuration follows this statement.

Mathematics accuracy and code Pass@1 for weight-only quantized LLaDA models, with task-specific few-shot settings and generation lengths.

Mathematics accuracy and code Pass@1 for LLaDA models under weight-activation quantization.

4.5 Impact of Model Types (RQ4)

Instruction-tuned LLaDA generally shows smaller degradation than the base model, particularly at aggressive precision. Under W4A4 DuQuant, reported mathematics degradation is 3.5% for LLaDA-8B-Instruct versus 13.0% for LLaDA-8B, and general degradation is 2.5% versus 5.1%.

The pattern has exceptions: W4A16 GPTQ code degradation is reported as 6.4% for the instruction variant versus 5.9% for the base variant. The results establish a tendency within this model pair without identifying why instruction tuning changes sensitivity.
Dream-7B’s four-task general average falls from 71.2 to 70.6 with W4A16 GPTQ, a reported 0.8% drop, versus 68.2 with AWQ, a 4.3% drop. At W3A16, the reported losses are 10.1% and 10.3%, respectively.
Dream supports the preference for 4-bit GPTQ on its tested general tasks. Weight-activation quantization was not evaluated for Dream because of resource constraints, so that recommendation is supported experimentally by LLaDA.

5 Limitation and Future Work

The study focuses on downstream quality and leaves optimized low-bit inference kernels, implementations of low-precision computation, to future work. It therefore does not establish measured latency, throughput, or device-memory improvements for a deployed diffusion inference system.

Planned extensions include more models, model sizes, and tasks; analysis of interactions between quantization and generation-step counts; and comparisons of remasking strategies, which select generated tokens to mask and revise.

Dream-7B general-task performance under 4-bit and 3-bit weight-only quantization with group size 128.

6 Conclusion

The conclusion identifies activation outliers as obstacles to low-bit quantization and recommends GPTQ and DuQuant based on their overall experimental performance. The results support different precision requirements for weight-only and weight-activation quantization, with quality losses depending on the task and model variant.

Appendix

A Additional Implementation Details: GPTQ uses 128 WikiText-2 samples of sequence length 2048, asymmetric quantization, which permits a nonzero offset, and group size 128 through https://github.com/AutoGPTQ/AutoGPTQ. AWQ uses 128 PileVal samples of sequence length 512 with asymmetric quantization and group size 128 through https://github.com/mit-han-lab/llm-awq.
SmoothQuant uses \(\alpha=0.5\) and s_j = max(|X_j|)^α/max(|W_j|)^(1−α), with 128 WikiText-2 calibration samples of sequence length 2048. QuaRot preserves query states, attention inputs used to determine relevance, at 16-bit precision and uses symmetric activation quantization, which centers the quantized range around zero. DuQuant uses the same calibration setup and α as SmoothQuant, activation and weight clipping ratios of 0.9 and 0.8, rotation step 256, and block size 128.
Evaluation uses https://github.com/EleutherAI/lm-evaluation-harness. General tasks other than MMLU use 0-shot evaluation, with no demonstration examples, and 128 Monte Carlo samples, repeated samples used for estimation; MMLU uses 5-shot evaluation, with five demonstrations, and one sample. LLaDA’s general-task diffusion steps, block size, and generation length are all 1024, its classifier-free guidance (CFG) scale, a control for generation conditioning, and temperature, a sampling control, are 0.0, and it uses low confidence remasking; Dream uses at most 128 new tokens, CFG scale 1.0, temperature 0.0, and top-p threshold 0.95, described by the paper as the probability of retaining generated tokens.
Table A1 specifies LLaDA mathematics and code settings as few-shot count, generation length, diffusion steps, and block size: GSM8K uses 4, 256, 256, 32; Math uses 0, 256, 256, 64; HumanEval uses 0, 512, 512, 32; and MBPP uses 3, 512, 512, 32. B More Visualizations adds Figure B1 for LLaDA-8B-Base and Figure B2 for LLaDA-8B-Instruct, showing additional channel-wide outliers and concentrated massive peaks at FFN output-projection inputs.

Brief Thoughts

W4A16 GPTQ is a reasonable initial configuration to test for these checkpoints, with mathematics and code validated separately before accepting a general-task average. The W4A4 SmoothQuant failures show that performance preserved at higher precision can collapse under aggressive activation quantization. Deployment decisions require additional evidence because optimized kernels, measured system efficiency, and Dream weight-activation results are absent. The single-forward-pass outlier analysis also does not characterize changes throughout iterative decoding. Several table entries require verification. Table 1 reports a 0.1% drop for instruction-model W4A16 AWQ despite averages of 65.7 and 64.9; Table 4 lists an instruction-model W8A8 SmoothQuant code average of 38.6 despite component scores of 37.2 and 37.1. Table 2’s base-model W4A4 QuaRot average of 59.2 does not match its five displayed task scores, and its reported 6.6% drop does not match the displayed baseline and average. These inconsistencies limit confidence in small gains and fine method rankings.