SmallThinker: A Family of Efficient Large Language Models Natively Trained for Local Deployment | Summary

This article explains the key points of SmallThinker: A Family of Efficient Large Language Models Natively Trained for Local Deployment.

• 2025-07-28 (arXiv)
• Song, Yixin, Xue, Zhenliang, Wei, Dongliang, Chen, Feiyang, Gao, Jianxiang, Liu, Junchen, Liang, Hangyu, Qin, Guangshuo, Tian, Chengrong, Wen, Bo, et al.
• Institute of Parallel and Distributed Systems, Shanghai Jiao Tong University, School of Artificial Intelligence, Shanghai Jiao Tong University, Zenergize AI
• Paper

Read this article in Korean

Summary

• SmallThinker addresses running large language models on local devices with limited computation, memory, and storage bandwidth. With Q4_0 quantization, the reduced-precision weight format used in the experiments, SmallThinker-21B-A3B decodes at 20.30 tokens/s on an i9 14900K PC under an 8 GiB memory limit, compared with 10.11 tokens/s for Qwen3-30B-A3B, while scoring 84.4 versus 85.1 on MMLU, a general-knowledge evaluation.
• The design combines Mixture-of-Experts (MoE), which selects a subset of expert networks for each token, with neuron-level sparsity, meaning inactive neurons inside selected experts. Routing before attention allows expert weights to be fetched from storage during attention computation, while a mixture of global and local attention reduces the key-value (KV) cache, the stored attention states used during generation.
• The 4B and 21B models are pretrained on 2.5 trillion and 7.2 trillion tokens, respectively, followed by supervised fine-tuning, training on instruction-response examples, and checkpoint merging, combining weights from saved training states. The strongest throughput gains occur under constrained memory; results vary by device, and the paper provides neither component ablations, experiments isolating individual changes, nor a complete decoding-workload specification.

1. Introduction

SmallThinker: A Family of Efficient Large Language Models Natively Trained for Local Deployment treats weak processors, limited memory, and slow storage as constraints to address during model training and inference-system design. Its contributions connect selective computation, early expert selection, expert caching, and reduced attention-state storage; Figure 1 summarizes the decoding-speed and general-knowledge tradeoff under limited memory.

SmallThinker-21B-A3B combines an MMLU score of 84.4 with approximately 20 tokens/s in the plotted limited-memory setting. Qwen3-30B-A3B has a slightly higher MMLU score but decodes at approximately half that speed, illustrating a speed-quality tradeoff.

2. Model Architecture

Both models use a decoder-only Transformer, an architecture that generates tokens from preceding context. SmallThinker-4B-A0.6B has 32 layers, hidden dimension 1536, 12 query heads, 2 key-value heads, and 32 experts with 4 selected per token; SmallThinker-21B-A3B has 52 layers, hidden dimension 2560, 28 query heads, 4 key-value heads, and 64 experts with 6 selected per token. Attention heads perform parallel attention computations; both models use head dimension 128 and feed-forward network (FFN) dimension 768, where the FFN is the per-token transformation implemented by the experts, and their names distinguish total parameters from approximately active parameters per token.

The larger model increases depth from 32 to 52 layers and expert count from 32 to 64, while selecting 6 rather than 4 experts per token. Both variants retain FFN dimension 768 and head dimension 128, expanding capacity while keeping computation selective.

The router branches from the input before attention, making expert selection available early enough to overlap weight transfers with attention computation. Within each selected expert, the gate is dense while up and down projections are sparse; a separate predictor selects rows of the final language model head.

2.1. Fine-Grained Mixture of Experts

Fine-grained expert selection provides the first level of sparsity. Each expert uses ReLU-activated Gated Linear Units (ReGLU), a gated activation whose rectified linear unit (ReLU) sets nonpositive gate outputs to zero, providing a second level of sparsity within the FFN.

• The pre-attention router selects experts before the attention block, creating time to fetch their weights while attention is computed.
• DP-Groups Global Load Balance Loss applies balancing within partitioned data-parallel groups, groups processing different training data. The authors intend this relaxed balancing to support expert specialization, where different experts handle different inputs, and make frequently used experts easier to retain in memory.
• The paper describes the balancing mechanism conceptually but provides neither an explicit loss formula nor an ablation isolating its effect.

2.2. NoPE-RoPE Hybrid Sparse Attention

NoPE-RoPE Hybrid Sparse Attention follows a repeating 1:3 pattern: one global-attention layer with No Positional Embedding (NoPE), which omits positional embeddings, followed by three Sliding Window Attention (SWA) layers, which attend within a 4096-token window, using Rotary Position Embedding (RoPE), the positional embedding mechanism used in those local layers. Adopted from prior work, this pattern limits full-context KV storage to part of the network. The paper states that it largely preserves long-context capability but reports no dedicated long-context evaluation or measured cache-reduction comparison.

3. Pre-Training

SmallThinker-4B-A0.6B-Base and SmallThinker-21B-A3B-Base are pretrained on predominantly English data combining curated open-source corpora and synthetic examples. The training process changes the data composition over time and extends context length near its end.

3.1. Data Construction

The collected pool includes 9 trillion web tokens from FineWeb-Edu, Nemotron-CC, mga-fineweb-edu, and the Knowledge Pile, plus 1 trillion mathematics tokens from sources including OpenWebMath, MegaMath, and FineMath. Code sources include StackV2 and OpenCoder; scoring models filter low-quality material, but rigorous deduplication is omitted.

• Genre- and audience-conditioned reformulation, together with persona-driven augmentation that varies the personas used to construct examples, produces an additional 269 billion mathematics and code tokens. Genre and audience prompts are generated from sampled data, manually curated, and randomly paired with examples.
• The final pretraining stage also incorporates instruction-response data from open-source collections and millions of question-answer pairs extracted from highly scored web text.
• The released synthetic datasets are PowerCoding at https://huggingface.co/datasets/PowerInfer/PowerCoding and PowerMath at https://huggingface.co/datasets/PowerInfer/PowerMath.

3.2. Data Mixture

The corpus contains six categories: General Knowledge, Mathematics, Code, Web-STEM, web material in science, technology, engineering, and mathematics, Chinese, and SFT-style data, instruction-response examples resembling supervised fine-tuning (SFT) data. A three-stage curriculum begins with broad general data, increases the proportion of high-quality and specialized material, and finishes with the highest-quality corpora and more instruction-response examples. Exact mixture proportions and stage token allocations are not reported.

3.3. Pretraining Hyperparameter Setting

The 4B model is trained on 2.5 trillion tokens with sequence length 4096, global batch size 1536, or 6,291,456 tokens per batch, and peak learning rate 3e-4. The 21B model uses 7.2 trillion tokens, sequence length 4096, global batch size 4352, and peak learning rate 4.2e-4.

• Both use cosine learning-rate decay to 10% of the peak, weight decay 0.1, and Adam, the training optimizer, with moving-average coefficients β1 = 0.9 and β2 = 0.95.
• The reported 21B pretraining duration is 20 days; the paper does not specify the training hardware configuration needed to interpret this duration as a compute-cost result.

3.4. Long Context Pre-Training

Training begins with a 4096-token context and extends to 32768 tokens for the 4B model and 16384 tokens for the 21B model in the final stage. The RoPE base frequency changes concurrently from 1e5 to 1.5e6. These are training context lengths; the reported evaluations do not establish accuracy across those full lengths.

4. Post-Training

Post-training consists of supervised fine-tuning on knowledge-intensive, mathematics, and code examples, followed by interpolation of checkpoint weights. The models do not receive a subsequent reinforcement-learning stage using human-preference feedback.

4.1. Supervised Fine-Tuning

Knowledge-intensive fine-tuning data comes from question-answer pairs extracted from the highest-scoring general pretraining texts, with refined answers yielding over 10 million samples. Mathematics and code problems are sampled from open-source datasets, expanded through persona-based augmentation, and answered with detailed solutions generated by Qwen3-32B (thinking). Generated answers are filtered to reduce degeneration and maintain linguistic consistency; the paper does not provide a complete fine-tuning recipe or a quantified correctness audit.

4.2. Model Merging

After supervised fine-tuning, the authors linearly interpolate weights from checkpoints captured at different training stages to balance general knowledge and instruction following. The paper does not report the interpolation coefficients, checkpoint-selection procedure, or a separate comparison measuring the contribution of merging.

5. Model Evaluation

Evaluation covers task accuracy, expert activation patterns, and neuron-level sparsity. These analyses examine model capability, patterns of expert reuse relevant to caching, and opportunities to skip computation inside selected experts.

5.1. Model Performance

An internal modified OpenCompass, the authors’ evaluation framework, scores base models through option-based perplexity, comparing candidate-answer likelihoods, and fine-tuned models through zero-shot answer generation, without demonstration examples, followed by answer matching. Table 2 compares general tasks using MMLU and LiveBench, mathematics or reasoning tasks using GPQA-Diamond and MATH-500, instruction following using IFEval, and coding using HumanEval. SmallThinker-21B-A3B and Qwen3-30B-A3B score, respectively, 84.4 versus 85.1 on MMLU, 55.1 versus 44.4 on GPQA-Diamond, 82.4 versus 84.4 on MATH-500, 85.8 versus 84.3 on IFEval, 60.3 versus 58.8 on LiveBench, and 89.6 versus 90.2 on HumanEval.

• SmallThinker-4B-A0.6B scores 66.1, 31.3, 60.6, 69.7, 42.2, and 82.3 on the same six evaluations, respectively. It leads Table 3 on GPQA-Diamond, LiveBench, and HumanEval, while Gemma3n-E4B leads on MMLU, MATH-500, and IFEval.
• Figure 3 separately tracks five-shot MMLU accuracy, using five examples in the evaluation context, for SmallThinker-21B-A3B-Base. Accuracy rises rapidly early in training, improves more gradually afterward, and accelerates again near the final learning-rate decay stage.

The base model’s five-shot MMLU accuracy improves rapidly at first, then more slowly through most of pretraining, and rises more sharply near the annotated decay stage. Because data composition also changes during training, this curve alone does not isolate the cause of the final improvement.

SmallThinker-21B-A3B leads the listed models on IFEval at 85.8 and LiveBench at 60.3. Against Qwen3-30B-A3B, it scores higher on GPQA-Diamond but lower on MMLU, MATH-500, and HumanEval, supporting competitive performance with task-dependent differences.

SmallThinker-4B-A0.6B leads on GPQA-Diamond, LiveBench, and HumanEval, including HumanEval 82.3 versus 74.4 for Gemma3n-E4B. Gemma3n-E4B remains stronger on MMLU, MATH-500, and IFEval, so the comparison does not establish a uniform accuracy advantage.

5.2. Expert Specialization

Expert activation is analyzed on GSM8K, HumanEval, Wiki, and Wiki-ZH, four datasets used to inspect routing differences across tasks and languages. The authors report that 70%–80% of experts have activation frequencies below 0.14, while the remaining 20%–30% have substantially higher frequencies of 0.4–0.6. Figures 4 and 5 show nonuniform, dataset-dependent activation, supporting the rationale for caching frequently reused experts without independently establishing the effect of the balancing loss.

The 21B model’s activation maps contain many low-frequency experts and a smaller number of hotspots that vary across datasets and layers. These uneven patterns support retaining frequently used experts in memory, while the figure itself does not measure cache effectiveness.

The 4B model also exhibits nonuniform expert usage across tasks and languages. Its maps support the rationale for selective caching in the smaller model, although the authors describe specialization as more pronounced in the 21B variant.

5.3. Neuron-Level Sparsity

Figure 6 shows neuron-level sparsity within experts across all layers of both models. Reported median sparsity remains above 0.6, with early layers reaching 0.9–1.0, so substantial computation can potentially be skipped even after an expert is selected. These distributions describe inactive neurons; realized speed depends on the inference kernels that exploit them.

Both models show high median neuron inactivity throughout the network, with the highest sparsity in early layers and wider variation among experts in several middle layers. This provides the empirical basis for skipping parts of the up and down projections after dense gate evaluation.

6. Inference Framework for Local Devices

The local inference framework combines expert offloading, storing some expert weights outside working memory, overlapping weight transfers with computation, and sparse execution. It is implemented in PowerInfer, the inference system used for the reported deployment measurements, at https://github.com/SJTU-IPADS/PowerInfer/tree/main/smallthinker.

6.1. Memory-Efficient Inference

Some expert weights reside on a solid-state drive (SSD), while frequently accessed experts remain in dynamic random-access memory (DRAM), the faster working memory. A shared cache across MoE layers uses least recently used (LRU) replacement, which evicts the expert whose most recent access is oldest. Pre-attention routing initiates expert reads during attention, and a further pipeline interleaves expert loading with FFN computation to reduce storage stalls.

6.2. Sparse Inference

Sparse ReGLU execution evaluates the gate matrix densely and computes only the needed parts of the up and down projections, the FFN transformations surrounding the activation. Fused kernels use Single Instruction, Multiple Data (SIMD), processor instructions that apply one operation to multiple values, to turn inactive neurons into reduced computation.

• The language model head, the final transformation from hidden states to vocabulary scores, has complexity O(HV), where H is hidden size and V is vocabulary size. A dedicated predictor selects vocabulary rows to compute, and the remaining logits, vocabulary scores before probability normalization, are set to zero.
• The paper does not specify the predictor architecture, selection threshold, or isolated accuracy impact of this output approximation.

6.3. Inference Performance Evaluation

All reported throughput comparisons use PowerInfer with Q4_0 quantization. With weights resident in memory, SmallThinker-21B-A3B reaches 30.19 tokens/s on the i9 14900K PC versus 33.52 for Qwen3-30B-A3B, but is faster on OnePlus 13 at 23.03 versus 20.18 and RK3588 at 10.84 versus 9.07; SmallThinker-4B-A0.6B reaches 108.17 tokens/s on the PC versus 62.24 for Qwen3-1.7B and 148.56 for Qwen3-0.6B.

• Under the 8 GiB memory limit, the 21B model achieves 20.30, 15.50, and 8.56 tokens/s on PC, OnePlus 13, and RK3588, versus 10.11, 0.18, and 6.32 for Qwen3-30B-A3B. The largest tabulated ratio is approximately 86.1× on OnePlus 13; the surrounding text inconsistently describes the maximum as 85× or 86×.
• Under the 1 GiB memory limit, the 4B model achieves 29.99, 20.91, and 15.04 tokens/s on the same devices, versus 2.66, 1.09, and 1.00 for Qwen3-1.7B. The largest ratio is approximately 19.2× on OnePlus 13.
• The largest gains occur in the memory-constrained comparisons, where the authors attribute baseline slowdowns to storage traffic. Prompt length, generation length, batch size, thread settings, storage characteristics, and measurement variability are not specified, limiting reproducibility.

In-memory SmallThinker-21B-A3B is slower than Qwen3-30B-A3B on the PC, 30.19 versus 33.52 tokens/s, but faster on OnePlus 13 and RK3588. On Raspberry Pi 5 it reaches 6.61 tokens/s while Qwen3-30B-A3B runs out of memory, showing that the ranking depends on platform.

SmallThinker-4B-A0.6B exceeds Qwen3-1.7B and Gemma3n-E2B throughput on every listed device. Qwen3-0.6B is faster on six of the seven devices, while SmallThinker leads it on Raspberry Pi 5.

Under the 8 GiB limit, the largest gain over Qwen3-30B-A3B occurs on OnePlus 13: 15.50 versus 0.18 tokens/s, approximately 86.1×. The gain is about 2.01× on PC and 1.35× on RK3588; the repeated Gemma3n-E4B values provide the paper’s fully resident reference.

With the stated 1 GiB limit, SmallThinker-4B-A0.6B exceeds Qwen3-1.7B by approximately 11.3× on PC, 19.2× on OnePlus 13, and 15.0× on RK3588. The Gemma3n-E2B values repeat its in-memory measurements, and SmallThinker exceeds that reference only on RK3588.

7. Limitations and Future Work

The authors identify limited pretraining-data scale as a possible constraint on knowledge breadth and performance across a wider range of tasks. They also note the absence of Reinforcement Learning from Human Feedback (RLHF), training with human-preference feedback, and propose it as future work to improve response quality and alignment with user preferences. The paper provides no direct helpfulness or safety evaluation, so its benchmark results do not establish those properties.

8. Authors

The listed authors are Yixin Song, Zhenliang Xue, Dongliang Wei, Feiyang Chen, Jianxiang Gao, Junchen Liu, Hangyu Liang, Guangshuo Qin, Chengrong Tian, Bo Wen, Longyu Zhao, Xinrui Zheng, Zeyu Mi, and Haibo Chen. The title page identifies the Institute of Parallel and Distributed Systems and the School of Artificial Intelligence at Shanghai Jiao Tong University, together with Zenergize AI.

Appendix

• The supplied 15-page paper contains no appendix; pages 14–15 contain References. Model releases are listed at http://hf.co/PowerInfer/SmallThinker-4BA0.6B-Instruct and http://hf.co/PowerInfer/SmallThinker-21BA3B-Instruct.

Brief Thoughts

The main contribution connects training-time sparsity with an inference pipeline that exploits it under tight memory limits. Tables 4 and 6 show the comparison condition clearly: the 21B model trails Qwen3-30B-A3B on the fully resident PC test but leads under the 8 GiB limit. Missing component ablations, incomplete workload specifications, and unreported accuracy effects of the vocabulary predictor leave the contribution of each optimization and the portability of the measured gains unresolved.