Kimi Linear: An Expressive, Efficient Attention Architecture | Summary

This article explains the key points of Kimi Linear: An Expressive, Efficient Attention Architecture.

• 2025-10-30 (arXiv)
• Kimi Team, Zhang, Yu, Lin, Zongyu, Yao, Xingcheng, Hu, Jiaxi, Meng, Fanqing, Liu, Chengyin, Men, Xin, Yang, Songlin, Li, Zhiyuan, et al.
• Moonshot AI, Soochow University, China, The Hong Kong University of Science and Technology (Guangzhou), Massachusetts Institute of Technology, Hangzhou Institute of Medicine, CAS
• Paper

Read this article in Korean

Summary

• Processing and generating long language-model sequences incurs growing computation and memory costs. Under matched 1.4T-token training recipes and parameter counts, Kimi Linear improves MMLU-Pro, a multitask language-understanding benchmark, from 47.2 to 51.0 and the average score on 128k-context evaluations from 52.2 to 54.5 relative to full Multi-Head Latent Attention (MLA), the paper’s global-attention baseline.
• Kimi Delta Attention (KDA), a linear attention mechanism that compresses token history into a fixed-size recurrent memory, adds independently learned forgetting rates for individual feature channels to Gated DeltaNet, a model that decays memory and corrects stored key–value associations through the delta rule. Kimi Linear repeats three KDA layers followed by one full MLA layer, retaining global attention while reducing key–value (KV) cache usage, the stored attention history, by up to 75%.
• The model has 48B total parameters and 3B activated parameters, the subset used per forward pass; its chunkwise kernel, which processes blocks of tokens, is nearly 2× faster than the compared general Diagonal-Plus-Low-Rank (DPLR) kernel, whose transition matrix combines a diagonal matrix with a low-rank correction. At 1M context, the paper reports 2.9× faster prompt processing and approximately 2.2× faster batch-size-1 decoding, with up to 6.3× decoding acceleration when memory savings support larger batches. Quality gains include benchmark exceptions, and the released 5.7T-token model’s comparison with Moonlight changes total parameter count as well as attention architecture.

The quality–speed panel places Kimi Linear above MLA on both MMLU-Pro and RULER under 1.4T-token training, with 3.98× acceleration reported for the 128k RULER condition. The decoding panel shows the larger long-context advantage when reduced memory consumption enables larger batches; its 1M comparison reports 1.84 ms versus 11.48 ms per generated token and labels the acceleration 6.3×.

1 Introduction

Standard softmax attention, which normalizes query–key scores into weights over previous tokens, has quadratic sequence-processing cost and a KV cache that grows linearly with sequence length. Linear attention reduces these costs through recurrent memory, but that memory’s finite capacity limits exact copying and retrieval from long histories. The paper combines more selective recurrent updates with periodic global-attention layers to address this limitation.

Contributions

The contributions are a channel-wise gated delta rule, a hardware-efficient implementation of that rule, and a 3:1 layerwise hybrid architecture evaluated through matched 1.4T-token training runs. The released artifacts include KDA kernels, integration with vLLM, an inference framework, and pretrained and instruction-tuned checkpoints.

• Project: https://github.com/MoonshotAI/Kimi-Linear
• KDA kernels: https://github.com/fla-org/flash-linear-attention/tree/main/fla/ops/kda
• Instruction-tuned checkpoint: https://huggingface.co/moonshotai/Kimi-Linear-48B-A3B-Instruct

2 Preliminary

The preliminary material presents linear attention as online learning of a matrix-valued associative memory, a temporary mapping from keys to values. It develops the delta rule and forgetting gates as mechanisms for correcting associations and removing outdated information.

2.1 Notation

Queries, keys, values, and outputs are token-level column vectors, while \(\mathbf{S}_t\) is the memory matrix with dimensions \(d_k\times d_v\), the key and value dimensions. A sequence of length L is partitioned into L/C chunks of length C; capitalized quantities stack vectors within a chunk, and each chunk starts from the previous chunk’s final state.

• M and M− are lower-triangular masks with and without diagonal entries, respectively; the paper also calls them Tril and StrictTril.
• Cumulative decay products track how information is attenuated between positions, while Diag(α_t) applies separate forgetting rates to memory channels.

2.2 Linear Attention and the Gated Delta Rule

Ordinary linear attention adds each key–value outer product to memory, reinforcing associations without deciding what to erase. DeltaNet instead performs online gradient descent on the reconstruction loss L_t(S) = ½‖Sᵀk_t − v_t‖², producing a rank-1 corrective update that moves the stored mapping toward the current value.

• Gated DeltaNet (GDN) additionally decays the previous state using a scalar forget gate \(\alpha_t\in[0,1]\) for each attention head, an independently parameterized attention component.
• The paper interprets this gate as data-dependent weight decay on fast weights, the temporary memory parameters updated as tokens arrive.

3 Kimi Delta Attention: Improving Delta Rule with Fine-grained Gating

KDA replaces GDN’s scalar decay with a diagonal channel-wise gate, allowing individual feature dimensions to retain information for different durations. Its central recurrence is S_t = (I − β_t k_t k_tᵀ) Diag(α_t) S_{t−1} + β_t k_t v_tᵀ, followed by o_t = S_tᵀq_t. The memory is decayed before the delta-rule correction is applied.

• β_t controls the strength of the corrective update, while the vector α_t controls forgetting separately across key dimensions.
• The method preserves the corrective rank-1 structure while introducing more granular memory decay and implicit positional information.

3.1 Hardware-Efficient Chunkwise Algorithm

The chunkwise algorithm expands the recurrence into a transition acting on the incoming state and an accumulated contribution from tokens within the chunk. A WY representation, a compact representation of successive rank-1 transformations, expresses these terms using auxiliary vectors; a UT transform, a reformulation using a triangular matrix solve, reduces operations outside matrix multiplication.

• The required lower-triangular inverse is computed by forward substitution, which solves the system row by row.
• Outputs combine information from the incoming recurrent state with a causally masked intra-chunk term, which excludes future tokens. Computation is recurrent between chunks and parallel within each chunk, increasing use of accelerator matrix-multiplication hardware.

3.2 Efficiency Analysis

Channel-wise decay can make reciprocal cumulative-decay terms numerically unstable, requiring secondary chunking, smaller computational subdivisions used for numerical stability. KDA’s constrained transition structure reduces the relevant secondary-chunk matrices from four to two and eliminates roughly three additional matrix multiplications relative to general DPLR. The measured kernel comparison shows nearly 2× acceleration for sequence lengths up to 64k at batch size 1 with 16 heads.

KDA has lower kernel execution time than the compared general DPLR implementation over input lengths from 2K to 64K. With batch size 1 and 16 heads, the long-input results support the paper’s nearly 2× kernel-speed claim; this comparison measures the attention operator rather than end-to-end model decoding.

4 The Kimi Linear Model Architecture

The backbone follows Moonlight, the language-model architecture used as the starting configuration, and alternates token mixing with a mixture-of-experts (MoE) layer, which routes tokens through a subset of specialized feed-forward networks. KDA uses short convolutions to capture local token dependencies and Swish nonlinear activations for queries, keys, and values, normalizes queries and keys to unit length, and sets both key and value head dimensions to 128. Its output uses head-wise root mean square normalization (RMSNorm), a normalization based on feature magnitude, followed by a data-dependent Sigmoid gate, which scales features between 0 and 1, and an output projection.

• The forget and output gates use low-rank projections to maintain comparable parameter counts. The architecture repeats three KDA layers and one MLA layer; whole-layer alternation is chosen for infrastructure simplicity and training stability.
• All MLA layers use No Position Encoding (NoPE), meaning they receive no explicit positional encoding, while KDA supplies positional information and recency bias.
• The paper states that NoPE enables conversion of MLA to Multi-Query Attention (MQA), an inference form sharing keys and values across attention heads, and avoids rotary position embedding (RoPE) frequency adjustments during context extension. RoPE encodes position through feature rotations.

The diagram shows whole-layer alternation: three KDA token-mixing blocks precede one MLA block, and each is followed by expert-based channel mixing. The expanded KDA block shows convolutional preprocessing, normalization, forgetting and update gates, and output gating, identifying the components evaluated alongside the recurrence.

5 Experiments

The experiments evaluate synthetic memory tasks, individual architectural components, training-compute scaling, matched large-model benchmarks, and inference timing. The synthetic tests examine the recurrent operator, while the large-model comparisons evaluate the complete hybrid architecture.

5.1 Synthetic tests

Synthetic tests compare KDA, GDN, and Mamba2, a recurrent model using multiplicative decay without the delta rule in these comparisons. Every model has 2 layers, 2 attention heads, and head dimension 128, and is trained for at most 20,000 steps with learning rates searched over {5 × 10⁻⁵, 1 × 10⁻⁴, 5 × 10⁻⁴, 1 × 10⁻³}. The study reports the best training-accuracy curves, measuring peak accuracy at lengths from 256 to 2,048 tokens and convergence at a fixed length of 1,024 tokens.

• Palindrome requires exact reversal of random token sequences. Multi Query Associative Recall (MQAR) requires retrieving values for multiple keys appearing at different positions in the history.
• Stack tests state tracking over 64 independent last-in-first-out stacks through PUSH and POP operations.
• KDA learns the reversal and associative-recall tasks substantially faster than GDN, while both perform strongly on stack tracking. Reversal and recall accuracy still fall at 2,048 tokens; Mamba2’s poor results apply to the tested small-model settings.

The upper panels compare peak training accuracy across sequence lengths, while the lower panels compare convergence at 1,024 tokens. KDA learns palindrome reversal and MQAR much faster than GDN, but both lose accuracy on those tasks at 2,048 tokens; stack tracking remains strong. The Mamba2 results apply to the stated two-layer, two-head experiment.

5.2 Ablation on Key Components of Kimi Linear

Ablations use the smallest scaling-law configuration, with 16 heads and 16 layers, under the same floating-point operation (FLOPs) budget and hyperparameters. Perplexity (PPL), a language-model prediction error measure where lower is better, is evaluated on both training data and a validation distribution that differs substantially from pretraining. The 3:1 hybrid achieves training/validation PPL of 9.23/5.65, compared with 9.45/5.77 for full attention and 9.29/5.66 for a 1:1 hybrid.

• Increasing the ratio to 7:1 preserves training PPL at 9.23 but worsens validation PPL to 5.70; 15:1 reaches 9.34/5.82.
• Removing the output gate produces 9.25/5.67, replacing Sigmoid with Swish gives 9.43/5.81, and removing convolution gives 9.29/5.70. The authors use Sigmoid output gating in all experiments, including the hybrid GDN baseline.
• NoPE outperforms the RoPE hybrid on the long-context average, 54.5 versus 51.8. The authors attribute this difference to the distribution of positional bias across layers, but the comparison does not isolate that proposed mechanism.

The 3:1 ratio gives the lowest reported validation perplexity, 5.65, while 1:1 is close at 5.66 and uses more global-attention layers. Matching training perplexity at 7:1 does not preserve validation quality. Removing the output gate or convolution, or replacing Sigmoid with Swish, also increases validation perplexity.

5.3 Scaling Law of Kimi Linear

Scaling experiments train five MoE sizes with 653M, 878M, 1.1B, 1.4B, and 1.7B activated parameters, excluding embeddings, using 38.8B, 59.8B, 85.2B, 102.5B, and 128.0B tokens respectively. They activate 8 of 64 experts, use Muon, the optimizer adopted from the Moonlight training configuration, and train at context length 4,096. MLA hyperparameters are grid-searched following the Chinchilla methodology for allocating model size and training tokens under a compute budget; Kimi Linear reuses those configurations with its 3:1 hybrid ratio.

• The fitted loss curves are \(2.3092\times C^{-0.0536}\) for MLA and \(2.2879\times C^{-0.0527}\) for Kimi Linear, where C denotes training compute in this scaling analysis.
• The fitted comparison suggests approximately 1.16× compute efficiency at matched loss within the studied range. Further improvement from KDA-specific tuning is proposed but not measured.

The five scaling configurations jointly increase activated parameter count, head count, depth, hidden width, token budget, and batch size while decreasing learning rate. All use 4,096-token contexts, and activated parameter counts exclude embeddings; these conditions define the range covered by the fitted scaling comparison.

Kimi Linear’s measured points and fitted curve lie below MLA’s over the studied compute range. The horizontal separation is summarized as approximately 1.16× compute efficiency at matched loss. This measures fitted training-compute requirements rather than kernel acceleration or serving throughput.

5.4 Experimental Setup

The main comparison includes full MLA, hybrid Gated DeltaNet (GDN-H), and Kimi Linear with matched parameter counts and training settings; a Kimi Linear (RoPE) variant tests explicit positional encoding. Each model activates 8 of 256 experts, including one shared expert, for 48B total and 3B active parameters, and its first layer is dense rather than MoE.

• Evaluation covers language understanding, reasoning, mathematics, code generation, Chinese-language tasks, and long-context tasks. LiveCodeBench v6, a code-generation benchmark, uses questions from 2024.8 to 2025.5.
• Temperature, the sampling parameter controlling response randomness, is 1.0. High-variance benchmarks use Avg@k, the mean score across k sampled evaluations; GPQA-Diamond, a graduate-level question-answering benchmark, averages eight independent runs.
• Base-model evaluations use perplexity for MMLU, a multitask language-understanding benchmark, its MMLU-Redux variant, GPQA-Diamond, and C-Eval, a Chinese-language evaluation suite, with generation-based evaluation otherwise. All models use the same internal framework derived from LM-Harness-Evaluation, a language-model evaluation toolkit.

5.4.1 Pre-training recipe: The controlled runs use a 4,096-token context window, MuonClip as the optimizer, WSD as the named learning-rate schedule, and 1.4 trillion tokens from the K2 pretraining corpus, the data source used for Kimi K2. The learning rate is 1.1 × 10⁻³ and the global batch contains 32 million tokens, with shared annealing and long-context activation procedures. The released checkpoint expands training to 5.7 trillion tokens and supports up to 1 million tokens of context.

5.4.2 Post-training recipe: Supervised fine-tuning (SFT), training on instruction-response examples, extends the Kimi K2 data with reasoning tasks, first covering diverse instruction-following data and then emphasizing reasoning, mathematics, and coding. Reinforcement learning (RL), training through rewards on generated responses, uses mathematics, code, and science, technology, engineering, and mathematics (STEM) prompts selected for moderate starting difficulty.

• The auxiliary PTX loss, implemented here as concurrent supervised training, uses diverse reasoning and general-purpose tasks to limit deterioration of general capabilities. The post-training data are subsets derived from the K2 training recipe.
• The RL algorithm follows K1.5, the earlier Kimi reasoning model’s training method, with truncated importance sampling, bounded probability-ratio corrections, to address mismatches between rollout and training policies caused by numerical precision differences.
• The Kullback–Leibler (KL) penalty, a constraint on policy divergence, and minibatch size are dynamically adjusted to stabilize learning and prevent loss of response diversity.

5.5 Main results

5.5.1 Kimi Linear@1.4T results: After matched pretraining, Kimi Linear leads the reported general-knowledge benchmarks, including HellaSwag, a sentence-completion benchmark, at 82.9 versus 81.7 for MLA and 82.2 for GDN-H, and MMLU-Pro at 51.0 versus 47.2 and 47.9. It also achieves the highest reported Chinese-task scores, 79.5 on CEval and 80.8 on CMMLU, a Chinese multitask language-understanding benchmark.

• MATH, a mathematics benchmark, ties MLA at 54.7, and EvalPlus, a code-correctness benchmark, is 60.2 versus GDN-H’s 63.1.
• GSM8K, a mathematics benchmark, improves only slightly over MLA, 83.9 versus 83.7. The larger MMLU-Pro difference shows that the size of the gain varies considerably across benchmarks.

The matched 1.4T-token base-model comparison shows Kimi Linear gains across the reported general-knowledge and Chinese tasks, including MMLU-Pro at 51.0 versus MLA’s 47.2. MATH ties MLA at 54.7, and GDN-H leads EvalPlus with 63.1 versus Kimi Linear’s 60.2.

After the same SFT recipe, Kimi Linear achieves 67.4 on MMLU-Pro versus 65.7 for MLA and 64.8 for GDN-H, and 62.1 on GPQA-Diamond (Avg@8) versus 57.1 and 58.6. It also leads the mathematics benchmarks AIME 2025, HMMT 2025, and PolyMath-en, along with LiveCodeBench v6, in the reported table.

• Exceptions include LiveBench, a general language-model benchmark, where Kimi Linear scores 45.2 versus 45.7 and 46.4, and EvalPlus, where it scores 61.0 versus 62.6 and 62.5.
• MATH500, a mathematics test set, reaches 81.2, above MLA’s 80.8 but below GDN-H’s 83.0. These results establish gains on many metrics rather than a uniform ranking across tasks.

At 128k context, Kimi Linear has the highest reported average across eight long-context metrics: 54.5 versus 52.2 for MLA, 51.2 for GDN-H, and 51.8 for Kimi Linear (RoPE). Its gains over MLA include MRCR, a long-context recall benchmark, at 29.6 versus 22.6, and RepoQA, a repository-code understanding benchmark, at 68.5 versus 63.0; RULER, a long-context benchmark suite, improves from 81.3 to 84.3.

• Kimi Linear also leads HELMET-ICL, the in-context-learning evaluation within a long-context benchmark framework, at 90.0, and Long Code Arena Lib, a long-context code task, at 37.1.
• MLA remains stronger on LongBench V2, a long-context multitask benchmark, at 36.1 versus 35.0; Frames, a fact-retrieval and reasoning benchmark, at 60.5 versus 58.8; and Long Code Arena Commit, another code task, at 33.2 versus 32.7.

Kimi Linear’s average of 54.5 is highest across the eight reported metrics at 128k context. Its gains on recall and repository-code tasks coexist with weaker scores than MLA on LongBench V2, Frames, and the Commit task. The RoPE hybrid’s 51.8 average supports the selected NoPE configuration within this comparison.

Mathematical reinforcement learning with verifiable rewards (RLVR), rewards based on checking response correctness, compares the 1.4T-token Kimi Linear and MLA checkpoints using identical algorithms and hyperparameters. Kimi Linear’s training accuracy increases faster, and its MATH500 and AIME 2025 test curves generally remain above MLA’s. This comparison measures convergence during mathematical RL training, rather than wall-clock training efficiency.

Starting from similar training accuracy, Kimi Linear improves faster and generally maintains higher training and held-out mathematics accuracy during the matched RL experiment. The test curves fluctuate, particularly on AIME 2025, so the evidence is their overall separation rather than monotonic improvement at every evaluation.

5.6 Efficiency Comparison

Timing comparisons use the 48B model setting with matched layer and head counts. Prefilling, processing the input prompt before generation, is similar for Kimi Linear and GDN-H and becomes substantially faster than MLA at long lengths: 2.3× at 512k and 2.9× at 1M. At batch size 1, Figure 7 reports approximately 2.2× lower time per output token (TPOT), the latency to generate each token, at 1M.

• Figure 1’s larger-batch comparison reports 1.84 ms versus 11.48 ms TPOT at 1M, labeled 6.3× acceleration. This includes the batching benefit enabled by reduced KV memory.
• The timing plots show little advantage at short contexts and closely overlapping Kimi Linear and GDN-H curves, indicating little measured overhead from channel-wise gating in this setup.
• The PDF does not specify accelerator hardware or the exact batch sizes for the larger-batch comparison, leaving those reproduction conditions unresolved.

At batch size 1, Kimi Linear and GDN-H have nearly overlapping timing curves and gain most over MLA at long contexts. The 1M annotations show 2.9× faster prefilling and 2.2× faster decoding, a smaller single-request benefit than the 6.3× larger-batch comparison.

6 Discussions

The discussion interprets KDA as a learned positional mechanism, relates its transition to constrained DPLR, and compares training-operation counts with inference memory costs. These analyses connect the channel-wise gate, the chunkwise implementation, and the choice of NoPE in global-attention layers.

6.1 Kimi Delta Attention as learnable position embeddings

The paper expresses positional interactions as cumulative products of transition matrices between a past key and the current query. RoPE uses fixed rotations, whereas gated delta models use data-dependent transitions combining forgetting with corrective updates. KDA’s channel-wise gates add feature-specific variation, motivating NoPE in global-attention layers.

• Table 6 compares recurrent and parallel forms of attention mechanisms, showing how decay and delta-rule corrections appear within a common algebraic framework.
• The authors propose that learned transitions could improve context-length extrapolation. The NoPE-versus-RoPE comparison supports the selected design at 128k context, but does not directly establish the proposed explanation.

The recurrent and parallel forms compare scalar decay, channel-wise decay, and corrective transitions within a common mathematical framework. Normalization and β_t are omitted for brevity, so the entries provide simplified algebraic relationships rather than complete implementation specifications.

6.2 Relation to DPLR

KDA’s transition can be rewritten as Diag(α_t) − β_t k_t k_tᵀ Diag(α_t), a constrained DPLR matrix. The low-rank factors are tied to the key and decay instead of being independently parameterized, allowing channel-wise forgetting followed by a delta-rule correction to share intermediate calculations.

• The side-by-side chunk_dplr and chunk_kda pseudocode reduces four secondary-chunk matrices to two.
• Additional simplifications remove roughly three matrix multiplications in inter-chunk and output computation. Figure 2 provides the measured kernel comparison.

6.3 Complexity Analysis

For one attention head and fixed chunk size C = 64, the paper gives FLOPs_KDA(T; C, d_h) = 6T d_h² + 3TC d_h + TC², compared with the dominant full-attention term FLOPs_Attn(T; d_h) = 2T² d_h. Here T is sequence length and d_h is head dimension; projection costs are treated as comparable, and the analysis assumes non-variable sequence lengths.

• Prompt processing uses the chunkwise kernel, while autoregressive generation, producing one token conditioned on previous tokens, uses recurrent updates.
• Each KDA head stores a fixed 128 × 128 state regardless of sequence length. The complete hybrid still contains full-attention layers with growing KV caches.
• The text cites 2.3× batch-size-1 acceleration at 1M, while Figure 7 labels it 2.2×. The larger 6.3× comparison additionally depends on reallocating memory to larger batches.

7 Related Works

Related work is organized around efficient subquadratic attention, whose sequence-processing cost grows more slowly than quadratic, and hybrids that combine different token-mixing mechanisms. Kimi Linear uses an inter-layer hybrid, alternating entire layer types, with a corrective recurrent operator and an optimized inference implementation.

7.1 Efficient Subquadratic Attention

Linear attention compresses history into recurrent memory, whereas sparse attention computes interactions with a selected subset of historical tokens. Learned gates improve retention and forgetting in recurrent models, and delta-rule updates correct individual associations; sparse methods improve token selection while retaining the historical KV cache.

• Table 7 uses the test-time training (TTT) framework, interpreting memory updates as optimization steps on incoming tokens, to compare learning objectives and state-update rules. KDA and GDN minimize reconstruction error on a decayed state.
• The paper reviews static sparse patterns, context-dependent selection, and hardware-aware chunk-level and token-level selection. It does not report a matched experimental comparison between Kimi Linear and those sparse architectures.
• Exact copying and fine-grained retrieval remain acknowledged limitations of purely linear memory. The authors discuss hybrid attention, state expansion, and combinations with sparse attention as ways to address them.

The learning-objective view distinguishes additive association updates from reconstruction-based correction. GDN and KDA share the latter interpretation but apply scalar and channel-wise decay respectively before correction, identifying the change introduced by KDA. The table compares conceptual relationships rather than benchmark performance.

7.2 Hybrid Model

Intra-layer hybrids combine different mechanisms within a layer or across portions of the sequence, which can require separate computational paths and complicate inference. Inter-layer hybrids alternate entire layer types in a predefined ratio; Kimi Linear adopts a regular 3:1 pattern to simplify cache management and integration with standard optimizations.

• The authors select KDA as the recurrent component based on its observed retrieval and copying performance.
• The discussion connects NoPE global attention to other hybrid designs seeking easier context extension. The reported NoPE ablation supports this choice within Kimi Linear’s tested setup.

Conclusion

The paper concludes that channel-wise delta-rule memory, periodic global attention, and specialized kernels improve the quality–efficiency trade-off. The matched 1.4T-token comparisons show gains across many benchmarks, while the timing experiments show larger inference benefits at long contexts. The reported maxima of 75% KV-cache reduction and 6.3× decoding acceleration depend on the hybrid architecture and the memory-enabled batching comparison.

Appendix

• “A Contributions” lists individual contributors and states that the project was developed at Moonshot AI. It identifies external collaborators and provides affiliations for Soochow University, China; The Hong Kong University of Science and Technology (Guangzhou); Massachusetts Institute of Technology; and Hangzhou Institute of Medicine, CAS.
• B Derivations for Chunkwise Parallelism of KDA gives inductive derivations of the compact expressions for the cumulative transition and accumulated token contribution. These derive the auxiliary-vector recurrences used to reorganize sequential rank-1 updates into parallelizable chunk computations.
• C Pseudo Code for chunkwise KDA provides the PyTorch-style function chunk_kda with inputs q, k, v, g, beta, an optional initial_state, and chunk_size: int = 64. PyTorch is the tensor-computation framework used for this pseudocode. The function accumulates log-decay terms, computes a triangular inverse through forward substitution, constructs auxiliary matrices, and combines recurrent chunk-state updates with parallel output calculation; secondary chunking is retained for numerical stability.
• D Kimi Linear@5.7T results compares the released model with Moonlight after both use 5.7T training tokens and 3B activated parameters, but Kimi Linear has 48B total parameters versus Moonlight’s 16B. Base-model MMLU-Pro is 54.8 versus 42.4, and instruction-tuned MMLU-Pro is 72.7 versus 43.8; Kimi Linear also scores 95.4 on RULER@128k and 94.8 on RULER@1M. Moonlight-Instruct is not evaluated on tasks exceeding its 8K context limit, so those missing entries provide no direct comparison.

Brief Thoughts

The matched experiments support the complete hybrid design: it improves many benchmark scores while reducing long-context inference costs. The close Kimi Linear and GDN-H timing curves show that finer forgetting control can improve quality without a substantial measured latency penalty in the reported setup.

The large-model gains reflect the combined architecture, including its 3:1 ratio, NoPE, output gate, and convolutions; the experiments do not assign all of those gains to channel-wise gating alone. For architectural comparison, the parameter-matched 1.4T runs provide stronger evidence than the 5.7T Moonlight comparison, which changes total capacity. Serving evaluations should separately measure batch-size-1 latency and the throughput gains available from larger batches.