Decoding Looped Transformers Better for (Almost) Free | Summary

This article explains the key points of Decoding Looped Transformers Better for (Almost) Free.

• 2026-10-01 (arXiv)
• Liu, Weihao, Zheng, Huangjie, Chen, Tianrong, Dilip, Rohit, Bai, Richard He, Jiao, Yizhu, Wang, Yuyang, Zhang, Ruixiang.
• Apple
• Paper

Read this article in Korean

Summary

• Looped Transformers, language models that repeatedly apply shared layers, normally discard earlier predictions that could help select the next token without additional training. At the same number of recurrent iterations, adaptive LoopCD-Logits raises Ouro-2.6B-Thinking’s AIME 2024 mathematical-reasoning pass@1, the estimated success rate of one sampled solution, from 61.88% to 73.33% against an unguided baseline with the same checkpoint and prompt.
• LoopCD applies contrastive decoding, which guides a stronger prediction away from a weaker reference, using states already produced by earlier iterations. LoopCD-Logits combines logits, token scores before conversion to probabilities, after the output layers and requires an additional output pass; LoopCD-Hidden combines hidden states, the model’s internal representations, before those layers and preserves a single output pass.
• Across six reduced-depth settings on seven multiple-choice benchmarks, LoopCD matches or exceeds the full-depth unguided mean while halving recurrent iterations and reducing theoretical forward floating-point operations by 22.5% to 48.2%. These savings are arithmetic estimates for a 512-token prefill, the processing of an input prefix, rather than measured end-to-end speedups; generation also requires guidance tuning and includes individual benchmark regressions.

1 Introduction

Standard decoding discards the predictions available after earlier recurrent iterations. Because each iteration applies the same parameter block to the same input prefix, these intermediate predictions supply a structurally aligned weaker reference without an auxiliary model or modified prompt. LoopCD extrapolates the change from an earlier prediction to the final one, requiring neither additional training nor additional recurrent iterations.

The overview connects two findings: earlier states guide the final prediction at unchanged depth, and the resulting gains can compensate for fewer iterations. Its reduced-compute panel summarizes multiple-choice means; it does not demonstrate half-depth preservation of the mathematical-reasoning and code scores shown in the full-depth panel.

2 Preliminaries: Looped Transformers

A recurrent iteration applies a shared block once, increasing effective depth and computation while keeping stored parameters fixed. Ouro repeats its entire decoder stack; Huginn, Parcae, and Looped-Qwen3 place recurrence between a prelude, the input-processing layers, and a coda, the layers after the loop. Huginn conditions each update on the prelude representation, Parcae learns recurrent dynamics, and Looped-Qwen3 repeatedly applies a frozen middle-layer window with damped updates that mix its output with the previous state.

• After R iterations, the final state h_R passes through the coda and lm_head, the final normalization and vocabulary projection, to produce token logits. Standard decoding converts these scores to probabilities through softmax, which exponentiates and normalizes them.
• The evaluated architectures differ substantially in output cost: Ouro has no coda Transformer layers, whereas Looped-Qwen3 has seventeen layers after its recurrent window.

The architecture comparison explains why an additional output pass has different costs across models. Ouro needs only its final normalization and head for an earlier readout, while the other families must also process layers after the recurrent block; Looped-Qwen3 has the longest such tail.

3 LoopCD: Contrastive Decoding from Recurrent States

LoopCD ordinarily uses the first completed recurrent state h_1 as the weak reference for the final state h_R. It extends the direction of recurrent refinement in either hidden-state space or logit space. The guidance space and the nonnegative coefficient ω determine where and how strongly this extrapolation acts.

3.1 Guidance Space

LoopCD-Hidden computes h′ = h_R + ω(h_R − h_1), then obtains z′ = lm_head(coda(h′)). LoopCD-Logits separately computes z_1 = lm_head(coda(h_1)) and z_R = lm_head(coda(h_R)), then forms z′ = z_R + ω(z_R − z_1). Both obtain next-token probabilities from softmax(z′), with ω = 0 recovering ordinary decoding.

• Hidden-state guidance adds lightweight vector operations while executing the coda and head once. Logit guidance executes those output layers twice, with architecture-dependent overhead.
• The h_1 reference is not universal: Huginn’s hidden-state experiments use later references to avoid amplifying initialization noise.

Moving the contrast before the output layers removes the second output evaluation, but also sends the guided representation through the model’s nonlinear coda. Consequently, the variants differ in both computation and how their updates affect token scores.

3.2 Guidance Strength

Fixed guidance uses a constant coefficient for every token in an evaluation setting. Adaptive guidance uses ω = ω_max[1 − (p_R,(1) − p_R,(2))], where p_R,(1) and p_R,(2) are the two largest unguided token probabilities and ω_max is the maximum strength. Guidance approaches this maximum when those probabilities are close and decreases when one token dominates.

• The adaptive rule uses the top-two probability margin, the difference between the leading probabilities, without computing entropy over the entire vocabulary.
• LoopCD-Logits already computes the unguided probabilities needed by this rule. Applying it to LoopCD-Hidden would require another output pass, so the reported hidden-state configurations use fixed guidance.

4 Experiments

Experiments cover Ouro, Huginn, Parcae, and Looped-Qwen3 under likelihood-based multiple-choice scoring, which ranks candidate answers by model-assigned probability; greedy generation, which chooses the highest-probability token at each step; and sampled mathematical reasoning. Guided runs and their baselines share checkpoints, prompts, demonstration counts, generation limits, stopping rules, answer extractors, and evaluators; full-depth comparisons also match recurrent depth. The reduced-depth experiments explicitly compare half-depth guided models with full-depth unguided models.

4.1 Gains on Mathematical Reasoning and Code Generation

On the mathematical-reasoning benchmarks AIME 2024, AIME 2025, and OlympiadBench, both Ouro-Thinking models improve pass@1 and pass@10, the estimated probability of obtaining at least one correct solution among ten samples, under fixed and adaptive LoopCD-Logits. Across the three benchmarks, mean pass@1 gains range from 5.27 to 7.33 percentage points, and mean pass@10 gains range from 2.53 to 4.12 points. For Ouro-2.6B-Thinking, adaptive guidance raises AIME 2025 pass@1 from 49.58% to 56.88% and OlympiadBench pass@1 from 64.05% to 67.29%.

• HumanEval and MBPP assess generated code through execution, using base and extended tests from EvalPlus, a code-evaluation suite. Every reported logit configuration improves its mean across these four test columns, although individual columns sometimes decline.
• With its own matched baseline, Huginn at R = 32 improves HumanEval pass@1 from 22.56% to 31.71% under LoopCD-Hidden. Its four-column code mean increases by 5.13 points; at R = 16, the mean increases by 3.76 points.

The mathematical results improve both sampling metrics for the two Ouro-Thinking models. Code-generation means also improve in every listed configuration, but individual columns reveal exceptions, including lower MBPP scores for fixed-guidance Ouro-1.4B and lower HumanEval scores for fixed-guidance Looped-Qwen3.

4.2 Consistent Improvements Across Architectures and Suites

Every evaluated LoopCD-Logits configuration improves its equally weighted mean across ARC-Challenge, ARC-Easy, SciQ, MMLU, HellaSwag, WinoGrande, and PIQA, seven multiple-choice evaluation datasets scored by answer likelihood. Fixed-guidance mean gains range from 0.32 to 1.38 points, while adaptive-guidance gains range from 0.29 to 1.59 points. The gains span Ouro’s training with supervised intermediate predictions, Huginn and Parcae’s training with sampled recurrent depths, and recurrence added to the frozen model underlying Looped-Qwen3.

• Consistency applies to suite means, not every benchmark: Ouro’s WinoGrande scores decline under both guidance rules, and Parcae-1.3B’s PIQA score also declines.
• LoopCD-Hidden raises the seven-benchmark mean by 0.83 and 0.75 points for Huginn at R = 32 and R = 16, respectively, and by 0.85 points for Parcae-1.3B.
• Prompt formats differ across architectures and sometimes across guidance forms. Each reported improvement is relative to its own matched baseline, so absolute scores do not always support direct comparisons between forms.

All seven listed model/depth configurations have positive suite means under both guidance rules. Adaptive guidance produces the larger mean gain in six configurations, but Ouro-1.4B favors fixed guidance and several individual datasets regress.

Hidden-state guidance improves all three listed multiple-choice means and every code-generation column at both Huginn depths. Its baseline values differ from those in the logit experiments, making within-form changes the appropriate evidence for comparing benefits.

4.3 Fewer Iterations, Matching or Superior Performance

Halving recurrence lowers the unguided seven-benchmark mean by 0.17 to 1.29 points. Guidance recovers this deficit in all six evaluated settings: relative to full-depth unguided baselines, half-depth Huginn gains 1.02 points with logit guidance and 0.51 with hidden-state guidance; Parcae-370M gains 0.77 with logit guidance; Parcae-1.3B gains 0.57 with logit guidance and 0.11 with hidden-state guidance; Looped-Qwen3 matches its baseline with hidden-state guidance.

• Including guidance overhead, the respective forward-operation reductions are 46.0%, 48.2%, 22.5%, 27.3%, 39.2%, and 23.6%.
• The reduced-depth evidence covers the multiple-choice suite exclusively. It does not establish equivalent savings at preserved mathematical-reasoning or code-generation accuracy.
• Reduced-depth settings use separately selected reference states and strengths. Huginn’s reduced-depth logit experiment uses h_5 rather than the h_1 reference used at full depth.

All six half-depth guided results reach or exceed their full-depth baseline means, while the computation panel includes the extra readout required by logit guidance. Huginn with hidden-state guidance has the largest saving; substantial fixed layers limit the saving for Looped-Qwen3.

5 How and Why LoopCD Works

The mechanism analysis examines whether recurrence supplies useful weak and strong predictions, which part of the contrast changes decisions, and how guidance strength affects stability. Its evidence combines reference-depth sweeps, comparisons between prediction distributions, geometric decomposition of the logit update, and accuracy changes grouped by prediction confidence.

5.1 Built-in Weak and Strong Predictions Across Recurrence

Used alone, the first recurrent prediction trails the final prediction by 4.0 to 23.7 points on the seven-benchmark mean. Later predictions generally converge toward the final distribution, reducing the contrast available for guidance. On ARC-Challenge, Huginn’s first reference disagrees with the final choice on 51.3% of questions and yields a 3.07-point gain at ω = 0.5; its sixteenth reference disagrees on 7.3% and yields 0.17 points.

• Jensen–Shannon divergence, a symmetric measure of distributional difference, falls from 0.71 bits after Ouro-1.4B’s first pass to 0.013 bits after its third. Huginn’s Kullback–Leibler divergence, a directional measure of distributional difference, falls from 3.04 nats after its first iteration to about 0.04 nats after its sixteenth; bits use base-two logarithms, while nats use natural logarithms.
• The first completed iteration gives the largest logit-guidance gain in the reported ARC-Challenge reference sweeps. Huginn’s hidden-state reference requires burn-in, the initial iterations that remove initialization noise: at R = 32 and ω = 0.5, h_1 reduces the suite mean by 0.58 points, whereas h_6 increases it by 0.83 points.

First-iteration predictions are weaker on the aggregate suite, supplying the reference asymmetry used by LoopCD. The iteration curves also show diminishing returns at later depths, although Huginn’s curve fluctuates rather than increasing at every iteration.

Distributional differences shrink as recurrence approaches the final prediction, and references with greater answer disagreement generally produce larger guidance gains. The scatter supports an empirical association between disagreement and gain rather than a universal scaling law.

The logit sweeps favor the first completed iteration on ARC-Challenge, but the hidden-state sweep exposes Huginn’s initialization exception. A later reference improves the seven-benchmark mean after the earliest hidden-state reference reduces it, showing why reference selection must account for guidance space.

5.2 Re-ranking Close Decisions Under Uncertainty

The logit contrast separates into a parallel component along the final-logit vector, which rescales scores like a temperature adjustment, and an orthogonal component perpendicular to it, which can change token rankings. On HellaSwag at ω = 0.5, using only the ranking component gains 1.72 points for Ouro-1.4B and 2.27 for Ouro-2.6B, compared with 0.76 and 1.56 points for the full update. This supports ranking changes as the source of improvement in those experiments.

• For four models on ARC-Challenge, gains in the least confident fifth of questions range from approximately 6.4 to 13.3 points, while the most confident fifth gains at most approximately 0.4 points.
• Only 5.8% to 14.9% of ARC-Challenge answers change at ω = 0.5. The resulting net gains of 2.1 to 3.5 points reflect more corrected answers than newly introduced errors.
• These findings motivate probability-margin adaptation, but they do not establish that every uncertain prediction benefits from guidance.

The decomposition separates score rescaling from changes in ranking, and the HellaSwag ablation shows that ranking changes can outperform the full update. The confidence-group panel places most ARC-Challenge gains among close decisions, motivating adaptive strength.

5.3 Tuning Guidance Strength Across Scoring and Generation

Multiple-choice scoring supports a broader guidance range, with a fixed value near ω = 0.5 providing a useful shared setting. Autoregressive generation, which repeatedly predicts the next token from the generated prefix, typically uses smaller fixed values in [0.2, 0.3], because early changes can compound across later tokens. The reported generation sweeps show steep degradation beyond approximately ω = 0.6.

• Adaptive guidance retains positive mean gains over a wider coefficient range by reducing updates when the unguided top-two probability margin is large. Caps in [0.5, 1.0] provide the principal operating range examined for multiple-choice scoring.
• Negative coefficients move predictions toward the weaker reference and degrade the reported multiple-choice means. Positive guidance also requires tuning: larger coefficients change more answers but can reverse previously beneficial corrections.

The sweeps show that a coefficient suitable for multiple-choice scoring can be too aggressive for sequential generation. The generation panel’s steep negative region makes tuning an operating requirement even though the method requires no training.

Adaptive strength preserves the average benefit at larger coefficient settings where fixed guidance becomes harmful. The individual curves still differ across models, so the broader mean operating range does not eliminate model-specific tuning.

6 Conclusion

The paper concludes that completed recurrent states provide reusable prediction pairs for inference-time contrast. Its experiments support higher suite means in the principal fixed-depth comparisons and preserved or improved multiple-choice means at half depth in six configurations. The suggestion that training explicitly for useful intermediate predictions could strengthen this approach remains a research hypothesis.

Appendix

• A Looped Transformers and LoopCD: A.1 States and coda layers specifies that references come from completed iterations and use the model’s existing output interface. A.2 Model architectures describes Ouro’s repeated full stack, Huginn’s noise-initialized state conditioned on its prelude, Parcae’s learned recurrent updates and projection before the coda, and Looped-Qwen3’s damped frozen window. Ouro-1.4B and Ouro-2.6B use four iterations of 24 and 48 layers; Huginn uses a four-layer block at R = 16 or R = 32; Parcae-370M and Parcae-1.3B use eight iterations of four and eight layers; Looped-Qwen3 uses eight substeps over four layers between fifteen prelude and seventeen coda layers. A.3 The probability form of LoopCD-Logits gives p′(v) ∝ p_s(v)(p_s(v)/p_w(v))^ω, where v is a vocabulary token and p_s and p_w denote final and earlier token probabilities. This identity holds before any evaluation-specific temperature adjustment or probability truncation; nonlinear coda layers prevent directly extending it to LoopCD-Hidden.
• B Detailed Experimental Setup: B.1 Model configurations and paired comparisons documents matched checkpoints and evaluation settings, with dedicated baselines for hidden-state runs. B.2 Benchmarks and table conventions defines equally weighted percentage-point changes: each AIME examination contains 30 problems, OlympiadBench uses its 674-problem English mathematics subset, and pass@1 and pass@10 are estimated from 16 sampled solutions per problem. B.3 Prompts and sampling specifies zero-shot evaluation, which supplies no worked examples, for mathematical sampling and code generation. MMLU-Pro, a multi-subject language-understanding benchmark evaluated by generated answers, uses five chain-of-thought demonstrations containing intermediate reasoning; GSM8K, a mathematics word-problem benchmark, uses three demonstrations by default or eight for Huginn’s chain-of-thought format. B.4 Strengths and reference states records screening sweeps and selected coefficients: standard logit multiple-choice scoring uses ω = 0.5; answer and code generation use 0.3 for Ouro and Looped-Qwen3 and 0.2 for Huginn. Huginn’s hidden-state scoring uses h_6 at R = 32 and h_7 at R = 16 with ω = 0.5; hidden-state code generation uses h_7 and h_6, respectively, with ω = 0.3.
• C Extended Results in Full Recurrent Depth: C.1 GSM8K and MMLU-Pro under LoopCD-Logits reports improvements in nine of ten MMLU-Pro rows and six of ten GSM8K rows, with GSM8K changes ranging from −1.29 to +2.04 points. C.2 Reasoning metrics: pass@1 and pass@10 across sampled trajectories identifies an exception to the main reasoning results: Looped-Qwen3’s mean pass@1 declines by 1.11 points with fixed guidance and 1.67 with adaptive guidance, while mean pass@10 rises by 1.90 and 3.18 points. On AIME 2024, fixed guidance changes pass@1 from 64.79% to 61.88% and pass@10 from 82.91% to 86.17%. The authors interpret this as consistent with broader coverage across sampled solutions, while explicitly noting that aggregate pass@k does not directly measure their diversity.
• D Computation and Cost Analysis: D.1 FLOP accounting assumes two floating-point operations, or FLOPs, per multiply-and-accumulate in matrix products during a 512-token prefill. Full-depth LoopCD-Logits cost multipliers range from 1.010× for Ouro-2.6B to 1.306× for Looped-Qwen3; LoopCD-Hidden rounds to 1.000×. D.2 Reduced-depth evaluation: performance recovery and computational tradeoffs specifies the six half-depth comparisons and their separately selected references and strengths; Looped-Qwen3 exits after four of eight substeps while retaining the damping coefficient 1/8. D.3 Representational dynamics: LoopCD-Logits versus LoopCD-Hidden associates a deeper coda with attenuation of hidden-state guidance: Parcae-1.3B retains about 62% of the fixed-logit mean gain, whereas Huginn’s shallow coda permits comparable or larger hidden-state gains. These comparisons use each form’s own matched baseline and independently selected settings, limiting causal attribution to coda depth alone.
• E The Reference Iteration: E.1 Reference selection across iterations shows that guidance gains track disagreement with the final choice more closely than standalone reference accuracy, and attributes Huginn’s early hidden-state failure to initialization noise. E.2 Geometric convergence in representation space standardizes hidden dimensions before measuring cosine similarity, the normalized alignment between representation vectors: Ouro repeatedly moves away from and back toward its final representation within each pass, while Looped-Qwen3 changes more gradually. E.3 Intra-iteration dynamics and output convergence reports that tested internal-layer references reduce ARC-Challenge accuracy by up to 20.6 points for Ouro-1.4B and 24.9 for Ouro-2.6B, supporting completed-iteration references in these architectures. Huginn’s median next-token entropy, a measure of predictive uncertainty, settles near 2.7 nats by iteration 16, yet 67% of answer positions remain above 2 nats at iteration 32.
• F Re-ranking Close Decisions: F.1 Concentration of guidance gains on uncertain decisions supplies confidence-group results and an ARC-Challenge case where guidance promotes the correct answer, “a tornado,” above “an earthquake” after the final prediction leaves them narrowly separated. F.2 Geometric equivalence of LoopCD-Hidden and orthogonal re-ranking shows that, on 10,042 HellaSwag questions at ω = 0.5, Ouro-1.4B’s hidden-state answers differ from isolated logit re-ranking on 205 questions and from the full logit update on 508; corresponding Ouro-2.6B counts are 136 and 357. These measurements support close empirical agreement rather than identical decisions. F.3 The double-edged effect of temperature sharpening finds that making the distribution more peaked through the parallel component has benchmark-dependent value: it hurts the reported HellaSwag comparison but helps ARC-Challenge reach gains at smaller coefficients.
• G Guidance Strength Dynamics: G.1 Strength boundaries: generation collapse and negative interpolation documents the narrow useful range for generation. On GSM8K, Ouro-1.4B loses 8 points at ω = 0.8 and 20 points at ω = 1.0; HumanEval loses 10 and 22 points at the same strengths. G.2 The trade-off between decision reach and overshoot shows that increasing ω from 0.5 to 1.0 raises the share of changed ARC-Challenge answers from 6.6% to 9.0% for Ouro-1.4B, while reducing the gain in its least confident fifth from 6.4 to 3.9 points. This empirical trade-off supports moderate or uncertainty-adaptive guidance.
• H Related Work positions LoopCD relative to shared-layer recurrent computation, weak-to-strong contrastive decoding, and guidance in diffusion models, which generate samples through iterative refinement. Its distinction is to contrast completed temporal iterations of the same block, obtaining a weaker reference from computation already performed. The discussion connects this design to autoguidance, which uses a weaker version of a model as a reference, while the internal-layer experiments support preferring completed iterations in the evaluated looped architectures.

Brief Thoughts

LoopCD reuses an existing recurrent trajectory through a simple inference-time operation, with matched comparisons across four architecture families and explicit accounting for output-layer cost. The practical choice between logit and hidden-state guidance depends on output depth, reference initialization, and task: Looped-Qwen3’s expensive second readout and Huginn’s noisy first hidden state make these dependencies concrete. The efficiency claim needs latency validation in a serving implementation: the paper estimates theoretical prefill arithmetic, and its half-depth accuracy evidence is restricted to multiple-choice evaluation. Positive suite means should be read alongside GSM8K regressions, Looped-Qwen3’s lower reasoning pass@1, and sensitivity to guidance strength. The reported tables do not supply confidence intervals for assessing small gains.