Grove MoE: Towards Efficient and Superior MoE LLMs with Adjugate Experts | Summary

This article explains the key points of Grove MoE: Towards Efficient and Superior MoE LLMs with Adjugate Experts.

• 2025-08-11 (arXiv)
• Wu, Haoyuan, Chen, Haoxing, Chen, Xiaodong, Zhou, Zhanchao, Chen, Tieyuan, Zhuang, Yihong, Lu, Guoshan, Huang, Zenan, Zhao, Junbo, Liu, Lin, et al.
• Inclusion AI, The Chinese University of Hong Kong, Renmin University of China, Zhejiang University, Shanghai Jiao Tong University, Westlake University
• Paper
• Github

Read this article in Korean

Summary

• Conventional mixture-of-experts (MoE) large language models (LLMs) select a subset of expert networks but activate the same parameter count for every token, limiting their ability to vary computation with the input. GroveMoE-Inst has 33B total parameters, where B denotes billion, activates 3.14–3.28B parameters per token, and leads the listed open-source non-reasoning baselines—models evaluated without a dedicated reasoning mode—on all general, alignment, mathematics, and scientific-knowledge benchmarks in Table 4.
• The architecture adds an adjugate expert, a parallel network shared by a group of existing experts, and computes it once whenever that group is selected. Upcycling, expansion through reuse of pretrained weights, converts Qwen3-30B-A3B-Base into GroveMoE-Base through 400B tokens of mid-training, continued training targeting specific capabilities, followed by supervised fine-tuning (SFT), training on prompt-response examples, to obtain GroveMoE-Inst.
• In the 50B-token architecture study, the selected group configuration scores 70.54 on the eight-benchmark average versus 69.59 for continued training without expansion; applying the same post-training strategy to the two base models later yields a reported average score improvement of 2.27 for the GroveMoE derivative. The reported implementation in SGLang, an inference framework, runs approximately 30% slower than Qwen3-30B-A3B, so the activated-parameter results do not establish faster serving.

The eight benchmark panels show GroveMoE-Inst leading the plotted instruction-model baselines, with particularly large gaps on AIME25 and Omni-MATH. The figure summarizes selected benchmark scores and contains no runtime measurements.

1 Introduction

Conventional MoE models route tokens to a fixed number of equally sized experts, limiting their ability to vary computation with token complexity. Grove MoE organizes experts into disjoint groups with shared parallel adjugate experts, preserving existing routing mechanisms while allowing the number of activated shared networks to vary. The paper introduces GroveMoE-Base and GroveMoE-Inst and provides the project URL https://github.com/inclusionAI/GroveMoE.

2 Related Works

The design draws on big.LITTLE, a processor architecture combining high-performance and energy-efficient cores. Previous dynamic MoE approaches use blank or constant experts, top-any gating that selects a variable number of experts, or ReMoE, a method that activates experts with positive routing scores; the paper identifies activation bounds and compatibility with conventional routing as practical concerns. Grove MoE instead varies shared-group activation while reusing a pretrained MoE checkpoint, extending earlier upcycling work.

3 Architecture

The architecture retains the original expert pool and router, the component that scores experts for each token, adds one adjugate expert per group, and controls expert load through a separate routing bias. Its capacity expansion combines grouped shared computation, load balancing, and initialization from pretrained weights.

3.1 Traditional MoE Layer

For a token representation x, the router R produces scores ρ = R(x), and expert E_i produces e_i = E_i(x). Top-k routing, selection of the k highest-scoring experts, forms the output y = ∑_{i∈arg topk(ρ)} ρ_i e_i; with uniformly sized experts and fixed k, the activated expert parameter count stays constant.

3.2 Grove MoE with Adjugate Experts

The n original experts are divided into g disjoint groups, where g divides n and each group contains n/g experts, with group index j = ⌊(i−1)/(n/g)⌋ + 1. The modified expert output is ē_i = E_i(x) + λA_j(x), where A_j is the group’s adjugate expert and λ scales its contribution; the layer sums these outputs using the original routing weights. If two selected experts share a group, their shared contribution becomes (ρ_r + ρ_s)λA_j(x), so A_j is evaluated only once.

• The paper restricts λ ≤ g/n; for n = 128 and g = 64, this gives λ ≤ 0.5.
• The paper states that the activated-adjugate count ranges from ⌈k/(n/g)⌉ to k. Dynamic computation follows the number of distinct selected groups; the section does not demonstrate that this count directly measures token difficulty.
• Figure 2 illustrates n/g = 2 and k = 4, showing how selected experts reuse group-level computation.

With two original experts per group and four selected experts, the diagram shows how two selected experts in the same group reuse one adjugate network. The router selects original experts, while their group membership determines which additional networks execute.

3.3 Experts Loading Balance

Auxiliary-loss-free load balancing, routing adjustment without adding a separate balancing objective to the training loss, introduces an expert-specific bias b for selection. With current assignment distribution F and uniform target Q = [1/n, …, 1/n], the update is b ← b − α(F−Q)/√[(1/n)∑_{i=1}^{n}(F_i−Q_i)²], using root-mean-square normalization of the imbalance. Selection uses Sigmoid scores, scores transformed independently into the interval between 0 and 1, plus b, while aggregation uses Softmax weights, scores normalized to sum to 1, allowing the authors to fix α = 0.001.

• Each selected expert receives assignment probability 1/k, and unselected experts receive 0.
• Equation (9) writes the aggregated expert output as e_i, whereas Equation (6) uses ē_i. The notation leaves the inclusion of the adjugate contribution in Equation (9) unclear.

3.4 Reuse of Pre-Trained Weights

Original experts, attention layers, and normalization layers are copied from the pretrained model. Newly inserted adjugate networks have zero-initialized down-projection blocks, the output projections of these networks, while their remaining weights follow a normal distribution with standard deviation 0.006. Zero down-projections initially suppress the added branch’s output while retaining the copied model components.

4 Mid-Training

Mid-training targets reasoning and code generation while expanding the pretrained architecture. The study first explores group sizes and output scaling on 50B tokens, then trains the selected GroveMoE-Base configuration on 400B tokens.

4.1 Mid-Training Data

The approximately 400-billion-token corpus includes web content, books, academic papers, social media, encyclopedias, mathematics, and programming code. The paper describes textual and non-textual sources but does not provide their mixture proportions or a detailed corpus construction procedure.

4.2 Evaluation Benchmarks

Base-model evaluation covers 13 benchmarks for general knowledge, scientific knowledge, reasoning, mathematics, and coding. Shot counts specify the number of examples supplied in the prompt, while chain-of-thought (CoT) settings prompt intermediate reasoning.

• General tasks: MMLU, a multitask language-understanding benchmark, uses 5-shot evaluation; MMLU-Pro, its more challenging counterpart, uses 5-shot CoT; CMMLU, Chinese multitask language understanding, SuperGPQA, graduate-level knowledge evaluation, and C-Eval, Chinese multidisciplinary evaluation, use 5-shot prompts; BBH, challenging Big-Bench tasks, uses 3-shot CoT.
• Mathematics and scientific tasks: GSM8K, mathematical word problems, and MATH, mathematical problem solving, use 4-shot CoT; GPQA-Diamond, graduate-level scientific questions, uses 5-shot prompts.
• Coding tasks: HumanEval+ and MBPP+, code-generation evaluations, and MultiPL-E, multilingual code generation, use 0-shot prompts; CRUX-O, code-output reasoning, uses 1-shot CoT. MultiPL-E covers Python, C++, Java, PHP, TypeScript, C#, Bash, and JavaScript.

4.3 Architecture Exploration

Architecture exploration uses 50B tokens and compares expanded 33B-parameter models with Qwen3-30B-A3B-Base continued on the same token count without upcycling. Table 1 compares g = 64, h = 128; g = 32, h = 256; and g = 16, h = 512, where h is the adjugate network’s intermediate dimension: their average scores are 70.54, 70.06, and 70.30 versus 69.59 for the continued-training baseline. Table 2 compares λ = 0.20, 0.10, and 0.05, obtaining averages of 70.19, 70.54, and 70.75 respectively.

• The g = 64 configuration achieves the highest eight-benchmark average with 3.26B average activated parameters; g = 16 reaches 44.95 on GPQA-Diamond but activates 3.93B parameters on average.
• λ = 0.05 gives the highest overall average, while λ = 0.20 gives the highest MATH and MBPP+ scores among the tested scaling factors.
• Figure 3 analyzes 1 million sampled tokens with n = 128. Fewer groups reduce the number of activated adjugate networks, but wider adjugate networks increase total average activated parameters: 3.26B, 3.51B, and 3.93B. The paper reports approximately 5% and 20% computational savings for g = 64 and g = 16 without explicitly defining the reference computation or providing corresponding runtime measurements.

All three expanded configurations improve the eight-task average over the 50B-token continued-training baseline. The g = 64, h = 128 configuration gives the highest average, 70.54 versus 69.59, and the lowest average activated parameter count among the expanded models, 3.26B.

The smallest tested scaling factor, λ = 0.05, gives the highest average score of 70.75, but task-specific optima differ. λ = 0.20 gives the highest MATH and MBPP+ scores, showing that stronger shared-branch contributions do not uniformly improve performance.

For 1 million sampled tokens and 128 original experts, reducing group count shifts the distribution toward fewer activated groups. This demonstrates increased reuse of adjugate computation, while Table 1 shows that wider adjugate networks can still raise the overall activated parameter count.

4.4 Hyper-Parameters

GroveMoE-Base uses g = 64, h = 128, and λ = 0.05. Training uses AdamW, an optimizer with decoupled weight decay, with β₁ = 0.9, β₂ = 0.95, weight decay 0.1, gradient clipping 1.0, maximum sequence length 8192 tokens, and batch size 16 million tokens over 400 billion training tokens. A cosine schedule decreases the learning rate from 1 × 10⁻⁴ to 5 × 10⁻⁵.

4.5 Mid-Training Evaluation

Table 3 compares GroveMoE-Base with five open-source base models under the same evaluation pipeline. Against its original Qwen3-30B-A3B-Base checkpoint, GroveMoE-Base improves MATH from 59.75 to 64.82, HumanEval+ from 83.54 to 85.98, and MBPP+ from 71.96 to 76.19, while MMLU-Pro decreases from 59.58 to 59.06 and MultiPL-E from 61.76 to 60.38. GroveMoE-Base leads all three listed mathematics and scientific benchmarks and three of four coding benchmarks, but does not lead every general-knowledge task.

• The model activates 3.14–3.28B parameters versus 24B–32B for the listed dense baselines, models that activate all their parameters per token, and 17B for Llama4-Scout-Base.
• Comparison with the original checkpoint includes both architecture expansion and 400B tokens of additional training. The 50B-token study provides a more controlled comparison against continued training without expansion.

GroveMoE-Base leads all three mathematics and scientific tasks and three of four coding tasks while activating 3.14–3.28B parameters. Its performance is mixed on general tasks, and Qwen3-30B-A3B-Base retains higher scores on MMLU-Pro and MultiPL-E.

5 Post-Training

Post-training converts GroveMoE-Base into GroveMoE-Inst using curated instruction-response data. Evaluation considers general capabilities, alignment with instructions and preferences, mathematics and scientific knowledge, coding, and function calling, generation of calls to specified functions.

5.1 Supervised Fine-Tuning

SFT starts from 1–2 million examples drawn from human annotations and open-source materials, then expands them through synthetic data generation. Magpie-style approaches, generation of new instruction prompts, and OSS-Instruct, instruction synthesis inspired by source materials, guide prompt creation; rejection sampling, generation and selection of candidate responses, uses several LLMs. Rule-based filters check reasoning-intensive examples, followed by an LLM evaluator using a detailed quality and relevance rubric across all data types.

• The paper does not specify the final expanded dataset size, full filtering rubric, or contribution of each data source.

5.2 Evaluation Benchmarks

Post-training evaluation includes eight general tasks, two alignment tasks, seven mathematics and scientific tasks, and six coding and agent tasks. IFEval, instruction-following evaluation, reports the average of prompt-level and instruction-level strict accuracy; Arena-Hard assesses alignment with human preferences; DROP evaluates discrete reasoning over paragraphs; and AGIEval evaluates foundation models on human-oriented examination tasks.

• MATH-500 is a mathematical problem-solving evaluation, Omni-MATH covers Olympiad-level mathematics, and AIME24 and AIME25 use mathematics competition problems. AIME scores average 16 samples per question, and GPQA-Diamond scores average 8 samples per query; OlympiadBench evaluates Olympiad-level scientific problems.
• LiveCodeBench, code-generation evaluation, uses v5 covering 2024.10–2025.02 and v6 covering 2025.02–2025.05. BFCL v3 (Live), the Berkeley Function Calling Leaderboard evaluation, uses the prompt format for all models.
• Maximum output length is 16K for evaluation of the non-reasoning models.

5.3 Hyper-Parameters

Post-training uses AdamW with β₁ = 0.9, β₂ = 0.95, weight decay 0.1, and gradient clipping 1.0. Its cosine learning-rate schedule decreases from 5 × 10⁻⁶ to a minimum of 1 × 10⁻⁶.

5.4 Post-Training Evaluation

Table 4 compares GroveMoE-Inst with Mistral-Small-3.2-Instruct-2506, Gemma3-27B-IT, Qwen3-32B Non-Thinking, Qwen3-30B-A3B Non-Thinking, and Llama4-Scout. GroveMoE-Inst leads all listed general, alignment, mathematics, and scientific tasks, including MMLU-Pro at 72.78, AIME25 at 44.38, and GPQA-Diamond at 61.30. It also leads five of six coding and agent tasks, but its BFCL v3 (Live) score of 76.11 trails Mistral’s 78.21.

• The strongest competing scores are 68.25 on MMLU-Pro, 28.12 on AIME25, and 55.56 on GPQA-Diamond.
• These results apply to the listed non-reasoning baselines under the paper’s evaluation conditions. Reasoning-mode models are outside this comparison.

GroveMoE-Inst leads 22 of the 23 listed benchmarks against the specified non-reasoning baselines. BFCL v3 (Live) is the exception: GroveMoE-Inst scores 76.11 versus 78.21 for Mistral-Small-3.2-Instruct-2506.

5.5 Effectiveness of GroveMoE-Base

To compare the base models’ suitability for instruction tuning, the authors apply the same post-training strategy to GroveMoE-Base and Qwen3-30B-A3B-Base. Figure 4 reports an average score improvement of 2.27 for the GroveMoE-based derivative, including gains of 4.65 on HumanEval+, 4.54 on GPQA-Diamond, and 3.46 on LiveCodeBench v6. The exceptions are OlympiadBench at −0.04 and MBPP+ at −0.53.

• Matching the post-training strategy strengthens the comparison of the two starting checkpoints, although their architecture and prior mid-training histories still differ.

Under the same post-training strategy, the GroveMoE-based derivative improves on 21 of 23 tasks and has a reported average gain of 2.27. The negative bars for OlympiadBench and MBPP+ identify the two exceptions.

5.6 Deployment of GroveMoE-Inst

The SGLang implementation runs approximately 30% slower than Qwen3-30B-A3B, exceeding the paper’s theoretical maximum 10% increase in activated parameters. The authors attribute the overhead to two separate fused-moe kernel calls, invocations of the generic expert-computation routine, per Grove MoE layer. A unified kernel is proposed for future work; no resulting latency improvement is measured, and this section does not provide hardware or workload details for the speed comparison.

6 Conclusion

Grove MoE expands a pretrained MoE model through group-shared adjugate experts whose computation is reused when multiple selected experts belong to the same group. The resulting GroveMoE-Base and GroveMoE-Inst models contain 33B parameters and activate 3.14–3.28B per token, with competitive benchmark results against the listed models. These results support capacity expansion with modest activated-parameter overhead, while the deployment measurements show unresolved runtime costs.

Limitations

The authors identify insufficient long-CoT data, examples containing extended intermediate reasoning, in the mid-training corpus as a limitation on advanced reasoning relative to models such as Qwen3-30B-A3B-2507. Model refinement also relies on rejection sampling without reinforcement learning (RL), optimization through reward feedback. The proposed benefits of adding RL remain untested in this paper.

Appendix

• The supplied 13-page paper contains no appendix. References begin on page 10 and continue through page 13.

Brief Thoughts

The strongest architectural evidence is the 50B-token comparison against continued training without expansion: grouped adjugate experts improve the reported average while keeping activated parameters close to the original model’s budget. The approximately 30% inference slowdown limits the practical efficiency claim until kernel integration is evaluated. Routing histograms demonstrate variable group activation, but the paper does not directly test whether token difficulty predicts that activation.