Gorio Tech Blog search

Fast and Simplex: 2-Simplicial Attention in Triton | Summary

한국어로 읽기

|
Contents

This article explains the key points of Fast and Simplex: 2-Simplicial Attention in Triton.

  • 2025-07-03 (arXiv)
  • Roy, Aurko, Chou, Timothy, Duvvuri, Sai Surya, Chen, Sijia, Yu, Jiecao, Wang, Xiaodong, Zaheer, Manzil, Anil, Rohan.
  • Meta, Department of Computer Science, University of Texas at Austin
  • Paper

Read this article in Korean


Summary

  • The paper studies whether language models can learn more from a limited training-token budget using 2-simplicial attention, which scores each query jointly against two keys instead of one. With equal training-token counts and matched parameter counts, its largest models, with 3.5 billion active parameters used per token, reduce negative log-likelihood, a measure of reference-answer prediction error, by a reported 0.45%–2.27% across four benchmarks relative to standard dot-product attention, which scores query–key pairs.
  • The implementation restricts the two key dimensions to sliding windows, local regions that move with each query, of (512, 32) and uses a 2-simplicial attention layer every fourth layer. Triton, the language used to implement the accelerator kernels, supports a reported throughput of 520 TFLOPS, or trillion floating-point operations per second; measured latency becomes competitive with FlashAttention v3, an optimized dot-product attention implementation, at long sequence lengths.
  • The fitted parameter-scaling exponents, which describe how loss decreases with model size, increase by a reported 6.8%–20.2%, but the fits use only three sizes and approximate the additive loss offset as small. The 1 billion active-parameter models show no meaningful gains, and the paper does not specify the training-token count, training-data composition, or repeated-run uncertainty.

1 Introduction

The introduction frames high-quality training data as a potential constraint on language-model scaling. Compute-optimal scaling, which allocates a fixed computation budget between model size and training data, typically calls for increasing parameters and tokens together; the paper asks whether a more expressive attention operation can improve performance when tokens are fixed.

  • The authors revisit the 2-simplicial Transformer introduced by Clift et al. (2019), combining higher-order attention with a kernel implementation and a theoretical construction based on rotation-invariant trilinear forms, functions that are linear in each of three inputs and unchanged by a shared rotation.
  • The suggestion that tokens could grow more slowly than parameters is an interpretation of the fixed-token experiments. The paper does not measure optimal allocations across multiple token budgets.

The related-work discussion connects methods that reduce attention’s sequence-length cost with higher-order methods that increase expressiveness per parameter. It places 2-simplicial attention alongside Edge Transformer and its triangular attention, which models three-way interactions, protein-structure triangle self-attention, and theoretical results on Match3, a task that checks whether a token participates in a triple whose sum is zero modulo a specified integer.

  • Looped Transformers, which repeatedly apply layers, provide another route to increased computation per parameter. The authors identify the greater effective depth and associated training difficulty as a concern when scaling this alternative.
  • The contribution combines an implementation and language-model scaling study of an existing attention family with a determinant-based theoretical construction. A determinant is a signed area or volume measure computed from a square matrix.

3 Overview of Neural Scaling Laws

The paper adopts \(L(N,D)=E+\frac{A}{N^\alpha}+\frac{B}{D^\beta}\), where N is the parameter count, D is the training-token count, E is irreducible loss associated with the underlying text distribution, and the remaining terms describe finite-model and finite-data contributions. It reviews earlier compute-optimal estimates in which optimal parameter and token counts scale approximately as C^0.49 and C^0.5, respectively, for computation budget C.

  • This framework distinguishes a change in a scaling exponent from a shift in the loss curve. Later experiments hold D fixed and estimate a parameter-dependent slope using active parameters rather than total parameters.

4 The 2-SIMPLICIAL Transformer

Standard dot-product attention scores each query against one key and combines the corresponding values, the content vectors retrieved by attention. The 2-simplicial version adds a second key projection K′ and value projection V′, computes a trilinear score \(A^{(2s)}_{ijk}=\frac{1}{\sqrt d}\sum_{l=1}^{d}Q_{il}K_{jl}K’_{kl}\), where d is the head dimension, and applies softmax, exponential normalization into probability weights, over both key indices.

  • The output sums S^(2s)_ijk(v_j ◦ v′_k) over key pairs, where S denotes normalized attention weights and ◦ is the Hadamard product, or elementwise multiplication. The retrieved content therefore depends on a pair of value vectors.
  • Figure 1 illustrates the change from a two-node edge to a three-node triangle. Algorithm 1 expresses the forward computation through tensor contractions, sums of products over shared indices, and a causal mask, which excludes future positions.

The edge represents an interaction between two token positions, whereas the triangle represents a joint interaction among three. This geometry explains why each query in 2-simplicial attention indexes a pair of keys rather than one key.

Geometric representations of dot-product attention and 2-simplicial attention.
Geometric representations of dot-product attention and 2-simplicial attention.

5 Determinant Based TRILINEAR FORMS

RoPE, rotary position embedding, encodes positions by rotating queries and keys; a shared rotation preserves their dot product. Because the ordinary trilinear product lacks this invariance, the authors propose summing signed determinants over three-dimensional chunks of the query and two keys.

  • Each determinant remains unchanged under a shared three-dimensional rotation. Sarrus rule, the expansion of a 3 × 3 determinant, expresses the operation as a difference of two trilinear contractions, increasing its arithmetic cost.
  • Theorem 5.1 states that a single attention head of dimension d = 7 can identify whether there exist j₁ and j₂ with x_i + x_j₁ + x_j₂ = 0 (mod M), for a polynomially bounded input range. This is an expressiveness claim about Match3, not an empirical comparison of determinant-based attention.
  • The subsequent kernel derivations use the simpler trilinear form. The experiments do not isolate the contribution of determinant-based scores or compare positional-encoding alternatives.

6 Model Design

Full 2-simplicial attention has O(n³) sequence-length complexity, so the model restricts each query to w₁ keys from K and w₂ keys from K′, reducing the dependence to \(O(nw_1w_2)\). The authors select (w₁, w₂) = (512, 32), with a reported latency of 55.1 ms in Table 1.

  • The paper’s operation-count expressions are 2n² for causal dot-product attention and 6nw₁w₂ for windowed 2-simplicial attention. For the selected windows, the authors place their computational crossover at 48k context length.
  • Grouped Query Attention, or GQA, lets multiple query heads share key and value heads. A GQA ratio of 64 enables tiling, dividing computation into hardware-sized blocks, across query heads to improve dense computation and reduce masking overhead.
  • Table 1 documents latency across seven window configurations but provides no quality measurements for the stated latency–quality tradeoff.

Latency generally grows with the product of the two window sizes: the listed 8k configuration takes 28.3 ms, 16k configurations take 55.1–59.2 ms, and 32k configurations take 104.1–110.7 ms. The selected (512, 32) configuration ties (1024, 16) at 55.1 ms; the table includes no quality measurements to distinguish them.

Measured latency for seven combinations of the two attention-window sizes.
Measured latency for seven combinations of the two attention-window sizes.

7 KERNEL Optimization

The forward kernel combines elementwise products with tiled matrix multiplication and online softmax, which incrementally normalizes attention without storing the complete score tensor. It overlaps elementwise work on CUDA cores, general arithmetic units, with matrix multiplication on Tensor Cores, accelerator units specialized for matrix operations, and reports 520 TFLOPS.

  • Figure 2 shows how the sliding windows and contraction decomposition map higher-order attention to matrix-multiplication hardware. Figure 3 shows competitive latency toward the long end of the measured sequence-length range.
  • The backward pass separates gradients for K and V from those for K′, V′, and Q. This repeats some intermediate computation but avoids the greater overhead the authors observe from atomic operations, which coordinate concurrent updates to shared memory locations.
  • For small w₂, Algorithm 2 processes even and odd query tiles in separate stages to compute gradients without atomic additions. The authors identify Triton’s coarse pipeline control as a possible obstacle to further optimization.

The left panel shows a rectangular region of key pairs moving with each query. The right panel decomposes the trilinear contraction into elementwise multiplication and tiled matrix multiplication, showing how the operation can use Tensor Cores without materializing the complete three-dimensional interaction tensor.

Sliding-window 2-simplicial attention and the tiling strategy for its trilinear computation.
Sliding-window 2-simplicial attention and the tiling strategy for its trilinear computation.

The operation-count curves cross near 48k tokens, but the latency bars show that the 2-simplicial kernel remains slower at 48k, is close at 56k, and is faster at 64k in the plotted comparison. The theoretical computation crossover therefore differs from the measured latency crossover.

Theoretical floating-point operation counts and measured execution times across sequence lengths for FlashAttention v3 and windowed 2-simplicial attention.
Theoretical floating-point operation counts and measured execution times across sequence lengths for FlashAttention v3 and windowed 2-simplicial attention.

8 Experiments & Results

The experiments train mixture-of-experts models, which activate a subset of their total parameters for each token, at active/total sizes of 1B/57B, 2B/100B, and 3.5B/176B, where B denotes billion. Every fourth layer uses sliding-window 2-simplicial attention, with the ordering chosen to distribute attention computation across pipeline stages, groups of layers assigned to different devices.

  • Training uses AdamW, an optimizer with decoupled weight decay, a peak learning rate of 4 × 10⁻³, weight decay of 0.0125, 4000 warmup steps, and cosine decay to 0.01× the peak learning rate.
  • Evaluation reports negative log-likelihood on GSM8k, a mathematics benchmark; MMLU, a multitask language-understanding benchmark; MMLU-pro, its more challenging counterpart; and MBPP, a programming benchmark. GSM8k uses 5-shot evaluation, meaning five examples are supplied; MMLU and MMLU-pro score the choice together with the entire answer.
  • The comparison fixes the training-token count across architectures, but the PDF does not report that count, the training-data mixture, total training computation, or end-to-end training time.

At 2B active parameters, Table 2 reports lower negative log-likelihood on all four benchmarks, with relative reductions of 1.51%, 1.19%, 0.71%, and 1% for GSM8k, MMLU, MMLU-pro, and MBPP, respectively. At 3.5B, the corresponding reported reductions are 2.27%, 1.06%, 2.15%, and 0.45%; at 1B, the displayed losses are worse on three benchmarks and equal to four decimal places on MMLU-pro.

  • Table 3 fits \(-\log L(N)=\alpha\log N+\beta\) using active parameter counts. The reported slopes rise from 0.1420 to 0.1683 on GSM8k, 0.1256 to 0.1364 on MMLU, 0.0901 to 0.1083 on MMLU-pro, and 0.1720 to 0.1837 on MBPP.
  • The reported relative slope increases are 18.5%, 8.5%, 20.2%, and 6.8%. Table 4 reports R², a measure of the fraction of observed variation explained by the fit, between 0.9962 and 0.9999.
  • These fits use only three model sizes per architecture. Holding tokens fixed makes the data term constant rather than zero, and the subsequent logarithmic fit approximates the additive offset; high R² over these points does not establish an asymptotic exponent. Reference-answer likelihood also does not directly measure answer accuracy or executable-code success.

The matched-size comparison shows no meaningful benefit at 1B active parameters, followed by lower losses on all four benchmarks at 2B and 3.5B. At 3.5B, GSM8k changes from 0.2781 to 0.2718 and MMLU-pro from 0.7858 to 0.7689, providing the largest reported relative reductions at that size. The displayed Transformer MBPP loss is 0.2203; the relative changes are the table's reported values and are not all exactly reproducible from its rounded losses.

Benchmark negative log-likelihood for matched Transformer and 2-simplicial models at three active and total parameter counts.
Benchmark negative log-likelihood for matched Transformer and 2-simplicial models at three active and total parameter counts.

All four fitted slopes are larger for 2-simplicial attention. The reported relative increases range from 6.8% on MBPP to 20.2% on MMLU-pro, describing a steeper empirical loss trend within the tested model-size range.

Estimated slopes and intercepts for the parameter-scaling fits on four benchmarks.
Estimated slopes and intercepts for the parameter-scaling fits on four benchmarks.

The reported R² values and small residuals indicate close fits to the three observed sizes for each architecture. They quantify agreement with those observations but provide neither confidence intervals for the slopes nor evidence about extrapolation to larger models.

R² values and residuals for the reported parameter-scaling fits.
R² values and residuals for the reported parameter-scaling fits.

9 Discussion

The discussion identifies settings where token efficiency is especially valuable as the likely practical case for the method. The authors explicitly describe the Triton implementation as a prototype that remains far from production use and call for implementation co-design with specific hardware accelerators.

10 Conclusion

The conclusion emphasizes lower benchmark negative log-likelihood at matched model sizes and fixed training tokens, together with larger fitted parameter-scaling slopes. The largest relative slope increases occur on MMLU-pro and GSM8k; gains from further scaling and specialized implementations remain prospective.

Appendix

  • A ROTATION INVARIANT TRILINEAR FORMS and A.1 PROOF FOR THEOREM 5.1 construct seven-dimensional query and key embeddings for Match3. Two three-dimensional determinant terms produce a score proportional to cos(2π(x_i + x_j₁ + x_j₂)/M), while the seventh dimension supplies a blank pair with value zero. For a large score multiplier, the attention output approaches zero without a match and β_i/(β_i + 1) with β_i matching pairs; an output multilayer perceptron, a feed-forward neural network, applies the stated threshold of 1/2.
  • B TRITON KERNEL: FORWARD PASS FOR 2-SIMPLICIAL ATTENTION provides Listing 1, two_simplicial_attn_fwd_kernel. Its displayed configuration uses BLOCK_SIZE_Q = 64, BLOCK_SIZE_KV = 32, num_stages = 1, and num_warps = 4, with bfloat16, a 16-bit floating-point format, for stored outputs and matrix-multiplication operands and float32, a 32-bit format, for accumulation. It implements local causal masks, incremental softmax normalization, value-pair products, and storage of the output and log-normalization statistic.
  • C TRITON KERNEL: BACKWARD PASS FOR 2-SIMPLICIAL ATTENTION supplies Listing 2, two_simplicial_attn_bwd_kv1_kernel, and Listing 3, two_simplicial_attn_bwd_kv2q_kernel. The first includes an optional atomic update for the query gradient; the second uses separate even- and odd-tile passes to avoid atomic additions for small w₂. Its displayed configuration uses BLOCK_SIZE_Q = 32 and BLOCK_SIZE_KV2 = 64 and requires BLOCK_SIZE_KV2 == BLOCK_SIZE_Q + w2.

Brief Thoughts

The kernel design makes higher-order attention feasible to evaluate in large, sparsely activated language models, with consistent likelihood gains at 2B and 3.5B active parameters. Evidence for a changed scaling exponent is limited by three model sizes, an approximate treatment of the loss offset, and no reported uncertainty. Comparisons across token budgets, direct task-success metrics, and end-to-end training costs would help assess whether the observed token efficiency justifies the additional attention computation.