Hybrid Architectures for Language Models: Systematic Analysis and Design Insights | Summary

This article explains the key points of Hybrid Architectures for Language Models: Systematic Analysis and Design Insights.

• 2025-10-06 (arXiv)
• Bae, Sangmin, Acun, Bilge, Lin, Chien-Yu, Habeeb, Haroun, Kim, Seungyeon, Luo, Liang, Wang, Junjie, Wu, Carole-Jean.
• FAIR at Meta, Meta, KAIST AI
• Paper

Read this article in Korean

Summary

• The paper investigates how to combine Transformer self-attention, which compares tokens across a context, with Mamba state space models, which compress prior context into a finite-dimensional state, to improve language prediction while reducing computation. In the approximately 1B-parameter comparison with an 8K-token training context and a matched budget of 4.5e20 floating-point operations (FLOPs), the intra-layer hybrid, which combines both components within selected layers, achieves 54.9% average few-shot accuracy, measured with task examples supplied in the evaluation context, versus Llama’s 52.0%. Its negative log-likelihood (NLL), the next-token prediction loss, decreases from Llama’s 2.750 to 2.709 on the DCLM-Baseline pretraining corpus.
• The study also evaluates inter-layer hybrids, which alternate complete Transformer and Mamba blocks. Ablations, experiments that vary individual design choices, favor approximately one attention-containing block per five Mamba blocks as a quality-speed compromise, attention at intermediate depths, and normalization of each component’s outputs with separate learned output transformations for intra-layer fusion.
• Hybrids generate faster than Llama and retain stronger retrieval beyond the 8K training context, although retrieval deteriorates in some extrapolated regions. The experiments cover models up to approximately 3B parameters and training contexts up to 32K; larger models, newer sequence-processing components, and extensions beyond language remain future validation targets.

1 Introduction

The research gap is the limited disclosure of controlled comparisons and design reasoning behind existing hybrid language models. Transformer attention has quadratic computation with sequence length, whereas Mamba offers linear sequence computation and a fixed-size recurrent state; combining them creates choices about block ratios, depth placement, and within-layer fusion.

• The baselines include Llama, homogeneous Mamba, and models mixing global attention with sliding window attention (SWA), which restricts attention to a local window and retained initial tokens called attention sinks.
• The evaluation considers language modeling loss, few-shot accuracy, task-specific fine-tuning, long-context retrieval, scaling, training time, decoding throughput, measured as generated tokens per second, and cache memory, the stored state reused during decoding.

The architecture panels distinguish alternating complete Transformer/Mamba blocks from blocks that split heads between both primitives. The quality-throughput panel shows favorable intra-layer trade-offs in the tested approximately 1B setting; its matched 4.5e20 FLOPs budget permits different training-token counts across architectures. Each point represents a block-ratio configuration, so the plot supports selecting a quality-speed compromise rather than a single universally best architecture.

2 Background

A Transformer combines multi-head attention, which processes dependencies through several attention heads, with a feed-forward network (FFN), which transforms each token’s representation. Its attention equation is Attention(Q, K, V) = softmax(QKᵀ / √d_k)V, where Q, K, and V are learned query, key, and value projections, or linear transformations of the input, and d_k is the key dimension. The softmax operation converts query-key similarity scores into attention weights.

• Grouped-query attention (GQA) reduces the key-value cache by sharing key and value heads across groups of query heads.
• Mamba updates a latent state through \(h_t=\bar{A}h_{t-1}+\bar{B}x_t\) and produces \(y_t=Ch_t+Dx_t\); h_t is the state, x_t is the input, and y_t is the output at time t. Input-dependent parameters control retention and incorporation of information.
• The experiments primarily use Mamba 2, whose state space duality reformulates state space computation as hardware-efficient matrix multiplications. Each sequence-processing block is followed by an FFN, optionally replaced by a Mixture-of-Experts (MoE) layer, which routes tokens through selected FFN experts.

The analytical comparison separates parameter counts, sequence-processing computation, and decoding cache costs, excluding the common linear-weight contribution from the displayed FLOPs expressions. Mamba’s fixed-size cache explains its growing inference advantage at longer contexts, while its larger sequence-processing parameter count means equal parameter budgets can require different layer counts.

3 Hybrid Architecture

The paper organizes hybrid architectures by the location of integration: between layers or within a layer. Both strategies vary the proportion and placement of computational primitives, the sequence-processing components used to construct blocks, allowing comparisons from attention-heavy configurations to homogeneous Mamba.

3.1 Inter-layer Hybrid Model

Inter-layer hybrids alternate complete softmax-attention and Mamba blocks at selected depths. The two research questions concern the Transformer-to-Mamba block ratio and the placement of Transformer blocks, evaluated across different ratios and model sizes.

• Prior hybrid models such as Jamba and Samba motivate Mamba-heavy configurations, but the paper tests a wider set of ratios rather than adopting a single existing recipe.
• Placement experiments compare early, middle, and late attention blocks to examine whether the same attention capacity behaves differently at different depths.

3.2 Intra-layer Hybrid Model

Intra-layer hybrids divide heads into Transformer and Mamba groups and fuse their outputs within one block; these blocks can themselves be interleaved with standard Mamba blocks. The Transformer branch reduces query and key dimensions while retaining an expanded value representation, and the Mamba branch reduces its hidden dimension according to the allocated capacity.

• Architectural ablations examine normalization, learned scaling or gates that weight component outputs, addition versus subtraction versus concatenation, output projections, and value-state sharing at 350M and 1B scales. Concatenation joins the outputs along their feature dimension.
• Dimension-ratio experiments vary the capacity assigned to the two branches. The expected benefit of concurrent branch execution depends on a parallel implementation, with the slower Transformer branch expected to bound throughput.
• Separate experiments vary the intra-hybrid-to-Mamba block ratio and the depth distribution of intra-hybrid blocks.

4 Experiments

Models are trained from scratch using Llama 3.2 and Mamba 2 configurations in TorchTitan, a PyTorch training framework, on H200 graphics processing units (GPUs). The default setup samples 60B tokens from DCLM-Baseline, uses an 8K context, and compares architectures under either equal token budgets or equal training FLOPs.

• Most reported model sizes refer to non-embedding parameters, excluding the parameters that map vocabulary tokens to representations; the approximately 1B models also contain embedding parameters.
• Language modeling evaluation uses 8,000 DCLM validation samples and 150 PG19 test samples, with PG19 serving as a long-document corpus.
• Few-shot evaluation uses LAMBADA, HellaSwag, PIQA, ARC, and OpenBookQA, the paper’s language prediction and question-answering benchmarks; ARC includes Easy and Challenge. The paper reports their aggregate accuracy.

4.1 Main Results on Quality

At 60B training tokens, the approximately 1:5 intra-layer hybrid achieves DCLM/PG19 NLL of 2.728/2.853 and 54.7% average few-shot accuracy, compared with Llama’s 2.750/2.875 and 52.0%, while using 3.7e20 rather than 4.5e20 FLOPs. At the matched 4.5e20 budget, its training expands to 72B tokens and yields 2.709/2.831 NLL and 54.9% accuracy; the inter-layer hybrid reaches 54.0%, SWA 53.8%, and Mamba 53.5%. The intra-layer improvement over Llama in this comparison is 2.9 percentage points.

• Training curves show hybrids following Llama early and reaching lower loss from the middle of training, with similar visible fluctuation patterns. Llama uses data parallelism, replication of training across data partitions, with degree 16 rather than the other models’ degree 8, which limits a strict comparison of loss fluctuations.
• Fine-tuning results show hybrids avoiding large Mamba deficits on retrieval-intensive summarization: Mamba scores 13.08 on GovReport, a summarization dataset, versus Llama’s 20.55 and has a −19.49 relative score on anonymized Task B.
• The downstream table does not show universal superiority over Llama. On TriviaQA, a question-answering dataset, inter-layer and intra-layer hybrids score 51.66 and 51.00 versus 51.89 for Llama; the intra-layer hybrid also has negative relative scores on Tasks A and B.

The two comparison groups separate a 60B-token constraint from a 4.5e20 FLOPs constraint. The intra-layer hybrid improves average accuracy from Llama’s 52.0% to 54.7% with less training compute, and reaches 54.9% when its compute budget is increased to match Llama through 72B training tokens. The latter difference is 2.9 percentage points and includes the benefit of additional training data.

The left panel shows similar visible training fluctuation patterns and lower hybrid loss later in training, with different data-parallel degrees for Llama and the other models. The right panel shows that hybrids avoid some large Mamba and SWA deficits, but records hybrid scores below Llama on several tasks. Its caption states 4.5 × 10^20 pretraining FLOPs, while Appendix I states 4e20.

4.2 Main Results on Efficiency

For approximately 1B models at 8K context, measured training steps take 5.61 s for Llama, 4.40 s for the inter-layer hybrid, and 4.66 s for the intra-layer hybrid. These measurements use 8 H200 GPUs, fully sharded data parallelism (FSDP), which distributes model training state across devices, torch.compile, which compiles PyTorch computation to accelerate execution, a local batch size of 4, and no activation checkpointing, a technique that saves memory by recomputing intermediate activations.

• The intra-layer hybrid’s 4.50 s result with branch parallelism is theoretical, whereas 4.66 s is measured. Peak training memory is 60.50 GiB for Llama, 62.93 GiB for the inter-layer hybrid, and 71.49 GiB for the intra-layer hybrid, so both hybrids use more peak training memory in this comparison.
• With a batch size of 4 and a 512-token prompt, 8K decoding throughput is 115 token/s for Llama, 241 token/s for SWA, 595 token/s for the approximately 1:5 inter-layer hybrid, and 683 token/s for the corresponding intra-layer hybrid.
• At a total length of 32K, throughput is 30, 110, 213, and 280 token/s respectively. Homogeneous Mamba remains fastest at 2917 token/s; choosing among these architectures also requires the quality and retrieval measurements.

Measured hybrid training steps are faster than Llama’s, but their peak memory is higher: 62.93 GiB for inter-layer and 71.49 GiB for intra-layer versus 60.50 GiB for Llama. The marked 4.50 s intra-layer result assumes theoretical branch parallelism and is separate from the measured 4.66 s result.

4.3 Long-context Capability Evaluation

Position-wise PG19 loss remains comparatively stable beyond the 8K training length for Mamba and the hybrids, whereas Llama’s loss rises sharply. This evaluation uses 30 unpacked long documents, evaluated individually without joining documents into training-style sequences, averages loss every 1K positions, and extends the Transformer components’ maximum positional-encoding length to 24K. Positional encoding supplies information about token positions.

• Needle-In-A-Haystack, a task that asks a model to recover inserted information from a long context, tests a random 7-digit number associated with a random city at different depths. Hybrids retain strong retrieval to about 1.5× the pretraining length, while Llama fails beyond its trained context and SWA or Mamba exhibit substantial location-dependent failures.
• Hybrid retrieval also degrades in portions of extrapolated contexts, particularly in the middle. Stable language modeling loss therefore does not establish uniformly reliable recall.
• The paper discusses No Positional Encoding (NoPE), removal of explicit positional information, as a way to address length-generalization limitations. Appendix K specifies extended positional-encoding lengths for the reported evaluations and does not identify the main comparisons as NoPE experiments.

Hybrid decoding throughput declines less severely than Llama’s as context grows, and its decoding caches are smaller. Loss for Mamba and the hybrid models remains comparatively stable beyond the 8K training length, while Llama’s rises sharply. Together with the retrieval heatmaps, this shows why extrapolated language modeling loss and recall require separate evaluation.

The heatmaps show distinct failure patterns: Llama deteriorates beyond its trained context, SWA loses information outside favorable local regions, and Mamba has depth-dependent long-range failures. Both hybrids retain broader successful retrieval regions, although accuracy also falls in parts of longer contexts. These results concern the paper’s city-number pattern-completion task.

4.4 Scaling Analysis

Replacing FFNs with MoE layers improves all tested architectures while preserving approximately the same activated parameter count, the parameters used for each token, and compute as their dense counterparts. With 60B tokens, the intra-layer hybrid improves from 2.728 to 2.648 DCLM NLL and from 54.7% to 56.9% average few-shot accuracy, while total non-embedding parameters increase from 0.98B to 3.27B.

• Each token uses one shared expert and the top-1 selected expert from eight candidates. MoE therefore improves measured quality while increasing the total stored model size.
• Compute-optimal scaling, selection of model size for the lowest loss at a fixed training budget, is studied at 100M, 350M, 1B, and 3B across five budgets from 2e19 to 4e20 FLOPs.
• The fitted optima favor larger models with fewer tokens per parameter for Mamba and intermediate allocations for hybrids. These estimates apply to the tested sizes and budgets.

The MoE results show quality gains at approximately fixed activated capacity while increasing total stored parameters. For the intra-layer hybrid, average accuracy rises from 54.7% to 56.9%, a 2.2-percentage-point gain. The adjacent scaling plot estimates architecture-specific optimal model sizes and places the hybrid allocation trend between Transformer and Mamba within the tested range.

4.5 Ablation Studies for Inter-layer Hybridization

Inter-layer ratio ablations favor 1:1 for language modeling loss and approximately 1:5 as a compromise among loss, cache size, and inference speed. At 1B and 60B tokens, DCLM NLL changes from 2.725 at 1:1 to 2.735 at 1:5, while the 8K cache decreases from 119 to 43 MiB and average few-shot accuracy changes from 54.0% to 53.3%.

• For a single Transformer block in the 1B model, placing it at layer 1 yields DCLM NLL 2.771, compared with 2.741 at layer 7 and 2.740 at layer 9. These results support avoiding front placement in the tested configurations.
• The best loss and best few-shot accuracy do not always coincide: at 350M, the 1:3 ratio achieves 50.2% accuracy versus 49.3% at 1:1, although 1:1 has lower loss.
• The authors hypothesize that early attention’s nearly uniform global aggregation conflicts with Mamba’s preference for local information. Attention visualizations show the descriptive difference but do not establish that conflict as the causal mechanism.

The ratio comparison favors 1:1 on language modeling loss, but accuracy rankings differ at 350M. The placement plots show a loss penalty for early Transformer blocks and better results at intermediate depths. They support a recommendation for the tested models rather than a universal prohibition on front placement.

4.6 Ablation Studies for Intra-layer Hybridization

Intra-layer ablations select module-wise Group normalization, which normalizes grouped features, no additional learned scaling, separate values and output projections, and subtraction fusion. With all layers converted to intra-hybrid blocks at 1B and 60B tokens, this configuration achieves DCLM/PG19 NLL of 2.712/2.831 and 54.9% average accuracy.

• Comparisons with Hymba and Falcon-H1, prior intra-layer hybrid designs, use the authors’ reimplementations with value-dimension expansion for GQA. Concatenation is also competitive, and the appendix shows that fusion interacts with projection and value-sharing choices.
• Increasing attention capacity generally improves loss relative to strongly Mamba-heavy branches, but the trend is not monotonic across every scale. A 1:1 dimension allocation is chosen as a quality-compute compromise.
• Scatter, which distributes intra-hybrid blocks across depth, improves average accuracy over Cluster, consecutive placement, and Sandwich, placement that includes both ends. At approximately 1:5, Scatter achieves 54.7% accuracy and 2.728 DCLM NLL versus Sandwich’s 52.4% and 2.746.
• A separate 350M model with learned interpolation coefficients, weights that combine the two component outputs, shows depth-dependent weighting: more Mamba weight early and more Transformer weight at selected middle layers. These coefficients are learned for next-token prediction.

The selected normalized subtraction variant with separate projections reaches 2.712 DCLM NLL and 54.9% average accuracy, with normalized concatenation close behind. The companion plots show the effects of branch capacity and block placement. Hymba and Falcon-H1 comparisons use modified reimplementations, and some reimplementation scores differ from those in the expanded appendix table.

5 Distinctions from Prior Works

The paper’s distinguishing contribution is its controlled comparison of inter-layer and intra-layer designs across quality, efficiency, and architectural ablations. It tests block ratios, depth placement, branch dimensions, normalization, scaling, fusion, value sharing, and output projections rather than reporting only a final architecture.

• The claimed novelty concerns systematic comparative analysis and disclosed design recipes. The study builds on established Transformer, Mamba, and hybrid components.

6 Conclusion and Future Work

The experiments support hybrid architectures as quality-efficiency compromises, with intra-layer hybrids particularly strong in the main approximately 1B comparison. The authors identify larger models, longer training, broader evaluation, newer attention or recurrent components, and additional modalities as future validation targets.

• The architecture study reaches approximately 3B model scale and uses a default 60B-token training regime, although matched-compute and scaling runs use other token counts.
• The paper does not experimentally establish that its recipes transfer to the newer sequence-processing components discussed in the conclusion, or to video and audio. The expectation of transfer remains a hypothesis.

Appendix

• A The Use of Large Language Models: The authors report using Llama 3.2 and Llama 4 to polish their own manuscript drafts and assist with bug fixes and initial figure drafts. B Broader Impact Statement: They discuss potential accessibility and environmental benefits of efficiency improvements alongside possible misuse for disinformation; these are proposed implications rather than measured outcomes.
• C Details for Computational and Memory Costs Comparison: The appendix estimates linear-weight training computation by multiplying parameter count by 6L_ctx, then adds sequence-processing costs, and derives bfloat16, a 2-byte numerical format, cache requirements. Transformer cache size is 4N_L d_head N_kv L_ctx, while Mamba cache size is 2N_L(d_ssm d_state + N_conv(2d_state + d_ssm)); N_L counts layers, N_kv counts key-value heads, N_conv denotes convolution width, d denotes a dimension, and L_ctx is context length. Mamba’s cache is independent of sequence length, with the 1B, 8K example reporting 13.4 MiB versus 256 MiB for Transformer.
• D Details for Intra-layer Hybrid Architectures: The generalized fusion equation varies module-wise normalization, scaling factors, addition/subtraction/concatenation, shared versus independent values, and unified versus separate output projections. Value sharing lets Mamba reuse the attention value projection. The proposed noise-reduction or complementary fusion effect is a hypothesis examined through variants.
• E Detailed Experimental Setup: Default training uses 8 H200 GPUs, a Llama 3.2 tokenizer that maps text to a 128K-token vocabulary, packed 8K sequences, a 2M-token batch, AdamW, the training optimizer, gradient clipping to bound gradient magnitude, FSDP, activation checkpointing, and torch.compile. The trapezoid learning-rate schedule has warmup, stable, and cooldown phases, with approximately 25% warmup and 20% cooldown; learning rates for 100M, 350M, 1B, and 3B are 6e-3, 3e-3, 6e-4, and 3e-4. Mamba’s state-transition parameter A is sampled from [0.01, 2.0] instead of [1, 16], while dt_bias retains its original random initialization.
• E Detailed Experimental Setup also specifies Rotary Positional Embeddings (RoPE), positional information encoded through rotations, with scaling factor 32, low/high frequency factors 1.0/4.0, original context 8K, and base frequency 500,000. SWA uses base frequency 10,000, a window of 512, and 64 sink tokens. Evaluation runs on one H200 GPU, and few-shot scoring uses target-byte-length normalization except for LAMBADA.
• F Expanded Main Results on Quality: Tables 9 and 10 extend equal-data and equal-FLOPs comparisons across ratios at 350M and 1B. Accuracy rankings depend on scale and ratio: at 350M and matched 1.9e20 FLOPs, a 1:1 inter-layer hybrid reaches 50.7% average accuracy, exceeding the reported intra-layer configurations. At 1B and matched compute, Table 9 also reports homogeneous SWA at 54.9%, matching the approximately 1:5 intra-layer hybrid’s accuracy despite worse language modeling loss.
• G Expanded Results of Sliding Window Attention Ablation: The SWA baseline is optimized by testing global-attention placement at 1:1, 1:3, and 1:5 ratios with a 512-token window and 64 sink tokens. Middle placement favors validation loss at the tested 1:3 and 1:5 ratios but does not maximize every accuracy score; at 1:5, Middle has 52.7% average accuracy versus End’s 52.9%.
• H Expanded Results on Longer Context Pretraining: At 16K and 4.5e20 FLOPs, intra-layer and inter-layer hybrids achieve 55.7% and 55.6% average accuracy versus Llama’s 54.0%. At 32K, inter-layer reaches 54.0%, intra-layer 53.5%, Mamba 53.4%, and Llama 50.7%; Mamba has the lowest DCLM NLL, 2.716 versus 2.722 for both hybrids. Hybrid accuracy benefits persist in these comparisons, while the best architecture depends on the metric and training context.
• I Experimental Setup for Downstream Fine-tuning Tasks: Fine-tuning runs for 1,000 steps with global batch size 16, peak learning rate 1e-4, 100 warmup steps, and 200 cooldown steps. Maximum sequence length is 8K except for SQuAD, a reading-comprehension question-answering dataset, at 2K; evaluation uses 1,000 instances per dataset, with SQuAD evaluated on its validation set. This appendix states a 4e20 pretraining budget, whereas Table 3’s caption states 4.5 × 10^20 FLOPs, leaving the downstream comparison’s exact pretraining budget inconsistent within the PDF.
• J Expanded Main Results on Efficiency: Table 13 reports decoding throughput and latency, the elapsed generation time, across total lengths of 2K, 4K, 8K, 16K, and 32K, with batch size 4 and a 512-token prompt. The approximately 1:5 intra-layer hybrid reaches 1089 token/s at 2K and 280 token/s at 32K, compared with inter-layer’s 969 and 213 token/s and Llama’s 373 and 30 token/s. Increasing the proportion of Mamba blocks generally increases throughput, with quality changes documented in the ratio ablations.
• K Experimental Setup for Long-context Evaluation Tasks: PG19 extrapolation uses 30 independently evaluated documents and 1B checkpoints trained at 8K with 4.5e20 FLOPs. Needle-In-A-Haystack uses concatenated Paul Graham essays, lengths L ∈ {960, 1984, …, 16320}, depths d ∈ {0%, 10%, …, 100%}, and the insertion format “The special magic {city} number is: {number}”. Because the models have no instruction tuning, training to follow explicit requests, evaluation uses a matching pattern-completion prompt; the model generates 20 subsequent tokens, and a result is correct when the generated text contains the target random 7-digit number.
• L Experimental Setup for Integration of Mixture-of-Experts: Every FFN is replaced by a token-choice MoE layer, where a router selects experts for each token, with one shared expert and top-1 selection from eight routed experts. Expert intermediate dimensions are halved, router scores use a sigmoid to map them to bounded values, and auxiliary-loss-free balancing adjusts routing biases without adding a balancing loss, using a bias update coefficient of 1e-3. The implementation uses torch._grouped_mm for grouped matrix multiplication; Table 4 shows increased total stored parameters despite approximately matched activated capacity.
• M Expanded Results of Scaling Laws Experiment: Models at 100M, 350M, 1B, and 3B are studied across budgets of 2e19, 4e19, 8e19, 2e20, and 4e20 FLOPs using 0.5M tokens/step and 8K context. Intermediate checkpoints are reused from the largest-budget runs, and quadratic fits estimate the best parameter count at each budget. Tables 14 and 15 provide the trained configurations and losses; their smallest-model rows do not report every budget, and fitted optimal allocations should be distinguished from directly trained configurations.
• N Expanded Results of Inter-layer Hybridization Ablation, N.1 Insights for Block Ratios: The detailed results show lower loss with more attention and smaller caches with more Mamba, motivating approximately 1:5 as a compromise. N.2 Insights for Positioning of Computational Primitives: Front, Middle, End, and single-block placement experiments at 350M and 1B generally support intermediate or later attention placement for loss. Accuracy has exceptions, including 350M 1:1 Front at 49.6% versus End at 49.3%.
• O Visualization of Attention Scores: Qualitative analysis uses 256-token PG19 contexts and 1B models trained on 60B tokens. Early Transformer attention is comparatively uniform across preceding positions, whereas Mamba’s implicit attention, a representation of how past inputs influence its state-space output, is concentrated near the diagonal; Mamba scores are converted to absolute values and normalized by each row’s maximum. These observations motivate the placement hypothesis, but the qualitative comparison and different score processing do not establish a causal explanation.
• P Expanded Results of Intra-layer Hybridization Ablation, P.1 Insights for Architectural Variants: Tables 18 and 19 examine primitive types, normalization, scaling, fusion, value sharing, and output projections at 1B and 350M. The authors prioritize NLL and layer count because few-shot accuracy is variable, selecting subtraction for later experiments; at 350M, normalized concatenation nevertheless achieves lower DCLM/PG19 loss, 2.846/2.978, than normalized subtraction’s 2.847/2.983. Table 6 and Table 18 also report different reimplementation results for Hymba and Falcon-H1, so their scores should be read within each table’s reported comparison.
• P.2 Insights for Dimension Ratios: Experiments test nominal 1:0, 2:1, 1:1, 1:2, 1:10, and 0:1 Transformer-to-Mamba capacity allocations using normalized concatenation, separate values, and a joint output projection. The chosen 1:1 allocation balances quality and cost, and at 350M it has lower loss than the larger nominal 2:1 attention allocation. P.3 Insights for Block Ratios and Positioning of Primitives: Scatter yields the highest average accuracy at each tested ratio, although individual loss minima occasionally favor Cluster or Sandwich.
• Q Analysis of Component Contributions in Hybrid Architectures: A 12-layer 350M intra-layer model trained on approximately 4B tokens learns weighted fusion, with α assigned to Mamba and \(1-\alpha\) to attention. Coefficients evaluated on about 100 DCLM samples vary substantially with depth, and three token-level examples associate greater attention weight with some syntactic boundaries, numerical information, and contextual keywords. This provides exploratory evidence about next-token prediction weighting; it does not directly demonstrate global reasoning or establish a causal account of specialization.

Brief Thoughts

The joint measurement of quality, throughput, and architectural choices under both token and compute constraints makes the design comparisons useful. Table 2 and the placement ablations support testing sparse intermediate attention and intra-layer fusion, while Figure 2 shows that decoding-cache savings and peak training-memory requirements need separate measurements.

The recipes provide starting configurations within the tested regime. Detailed tables qualify several broad claims: intra-layer hybrids do not lead at every scale or context, front placement does not always reduce accuracy, and Scatter does not minimize every loss. Limited model scale, anonymized downstream tasks, inconsistent downstream compute budgets, and acknowledged few-shot variance constrain how confidently these recipes can be transferred to a production training setup.