The Router Within: Eliciting Native Skill Routing from a Frozen LLM | Summary

Contents

Summary
1. Introduction
2. Related Work
3. Gavel: Routing on the Agent’s Own Forward Passes
4. Experiments
5. Conclusion
Appendix
Brief Thoughts

This article explains the key points of The Router Within: Eliciting Native Skill Routing from a Frozen LLM.

2026-09-14 (arXiv), arXiv preprint
Chen, Ruishuo, Wang, Xun, Chen, Yu, Li, Zhuoran, Huang, Longbo.
Institute for Interdisciplinary Information Sciences, Tsinghua University
Paper

Summary

Selecting a skill, a document containing instructions, scripts, and references, becomes difficult when a large language model (LLM) agent must search thousands of candidates without filling its working context with their descriptions. With Qwen3-32B’s weights held fixed, Gavel improves skill-selection accuracy by up to 13.4 percentage points on written tasks and 21.9 points during simulated agent execution over the strongest tested alternatives: progressive disclosure, which selects from skill metadata in the prompt, and external pipelines that retrieve candidates and then reorder them.
Gavel, short for Glance And Verdict from a frozen LLM, first ranks the library using two trained linear projections of intermediate token representations, then examines shortlisted skills using the same model’s task likelihood and yes/no relevance judgment. Its 7.9M projection parameters are trained on SkillRet, a corpus of task–skill pairs; new skills require one indexing forward pass, and evaluation on other libraries keeps the projections and calibration fixed.
The evidence includes three public written-task benchmarks, SkillTraj’s 372 simulated multi-turn trajectories, and deployment on Skill-Use, where Qwen3-32B with Gavel loads the correct skill on 90.9% of tasks versus 1.1% with progressive disclosure in the same agent harness, the software that manages model calls and tool interactions. These results measure skill selection and loading, rather than final task completion; deployment also requires access to model internals and an additional trained gate that decides when to route. For Skill-Use, see How Well Do Agentic Skills Work in the Wild: Benchmarking LLM Skill Usage in Realistic Settings | Summary.

1. Introduction

The Router Within: Eliciting Native Skill Routing from a Frozen LLM asks whether skill selection can retain the agent’s capabilities while keeping the library outside its working context. Skills are packaged in SKILL.md documents; loading an unsuitable skill can harm performance, while selecting through names and descriptions incurs context costs and omits details needed to distinguish candidates.

Progressive disclosure preloads installed skills’ metadata and lets the agent decide which document to read. The paper reports that metadata grows with library size and that Codex caps it at 2% of the context window.
External retrieval avoids this context burden but delegates selection to an embedding model, which represents text as vectors, and a reranker, which reassesses retrieved candidates. Gavel instead extracts selection signals from the frozen agent’s own forward passes. For embedding model, see Llama-Embed-Nemotron-8B: A Universal Text Embedding Model for Multilingual and Cross-Lingual Tasks | Summary.

Progressive disclosure, external retrieve-and-rerank, and Gavel as three skill-routing architectures.

The related work connects skill selection with tool retrieval and methods that extract information from LLM representations or predictions. Gavel combines these directions while keeping the backbone, the underlying language model, frozen.

ToolkenGPT represents tools as output tokens but trains an embedding for each tool, so adding a tool requires training. Gavel indexes a new skill through a forward pass without updating its projections.
Prior embedding methods adapt decoder models through training or prompting. GRIT, a method that jointly supports embedding and generation, fine-tunes the whole backbone; prior rerankers use query likelihood or yes/no relevance predictions, both of which Gavel combines with its learned representation read-out.

3. Gavel: Routing on the Agent’s Own Forward Passes

Gavel separates library-wide screening from detailed candidate assessment. Installation builds reusable skill representations, the glance ranks the library, the verdict evaluates a shortlist, and the ruling combines three scores to select the document to load.

Installation, token-level glance, candidate verdict, and combined ruling in Gavel.

3.1. Problem setup and design principles

The library consists of documents with a metadata header and an instruction body. At a routing point, the router observes the task context, which may be a user request or an execution state reached during a rollout, the agent’s sequence of messages and tool interactions.

The design requires task-only working context until selection, no standalone selection model, and installation-time indexing without training for each new skill.
The router processes skill documents in separate forward passes, keeping candidate text outside the agent’s ongoing working context until a skill is selected.

3.2. The glance: token-level read-out of the frozen forward pass

The glance applies a query projection Wq to task-token hidden states, the model’s internal token representations, and a key projection Ws to skill-token hidden states at one intermediate layer. The resulting vectors have unit norm; each task token scores a skill by its maximum inner product with any key in that skill’s bank, a late-interaction operation that compares token representations before aggregating them.

The read-out layer is selected at the compression valley, where matrix entropy, a measure of how many directions the hidden states occupy, reaches its minimum after removing the highest-norm token. On Qwen3-32B this is layer 44, the output of block 45 of 64.
Both projections are trained with a multi-positive contrastive loss, which raises the relative scores of annotated suitable skills against sampled unsuitable candidates. Gradients stop at the backbone’s hidden states.
Training uses mean token-match scores, but inference lets each token contribute only to its top-k skills so that a few informative tokens are not overwhelmed by unrelated context. The rule is k = min(10, max(3, round(0.1N))), where N is the library size; token weights can favor recent context.

3.3. Compressing the skill bank to an 𝜀-cover

An ε-cover compresses each skill’s bank by retaining a subset of keys such that every discarded key lies within Euclidean distance ε of a retained key. A farthest-first traversal repeatedly adds the key farthest from the retained set until this condition holds.

For unit queries and keys, replacing the full bank with its cover can lower the maximum inner product by at most ε and cannot increase it. The same one-sided bound applies to the aggregate vote when the skill remains more than ε away from every token’s top-k boundary.
The retained bank is also ε-separated, meaning retained keys are pairwise at least ε apart. Its size therefore lies between the covering number, the smallest possible cover size, and the packing number, the largest possible separated-subset size.
After compression, Wq receives a short additional fine-tune against the compressed training banks while Ws and the stored keys remain frozen.

3.4. The verdict: native signals under the model’s full attention

The verdict gives the frozen model full attention over each shortlisted skill and the task together. It appends the task to the installation-time skill render, the fixed prompt containing the skill, preserving a reusable prefix and extracting two scores from one continuation forward pass.

L is the task’s mean conditional log-likelihood given the skill: it measures how well the skill primes the model to predict the task’s wording.
V is the log-odds of yes over no, the logarithm of their probability ratio, after a fixed question asks whether the skill supplies what the task needs. The score includes capitalization and leading-space variants of yes and no.
The relevance question follows the task, so causal attention, which limits each position to preceding tokens, prevents it from changing the task-token likelihoods. Both scores therefore come from the same pass.

3.5. The ruling: a product of experts

The ruling selects the shortlisted skill with the highest combined score, S(s|x) = g(s|x) + αL(s|x) + γV(s|x), where g is the glance score. This is a product of experts, a combination that multiplies the component evidence after exponentiation.

Under a uniform skill prior, which assigns equal initial probability to every skill, the paper interprets the contrastive glance and generative task likelihood as estimates related to the skill posterior, its probability given the task. It treats the yes/no judgment as a discriminative estimate of relevance; that interpretation is stated rather than derived.
Validation-fitted α and γ reconcile score scales and temper confident components. The likelihood is averaged over task length to prevent long tasks from dominating the other evidence.

4. Experiments

The experiments test selection from written requests, selection at marked points during simulated execution, autonomous triggering in a live harness, and changes with backbone capability. Comparisons include external retrieval pipelines, metadata-based selection, and replacements of individual Gavel components.

4.1. Setup

The main model is frozen Qwen3-32B with 7.9M trainable projection parameters. Training uses 51,104 SkillRet queries generated by Qwen3.5 over 9,084 skills, with contrastive temperature τ = 40, the factor scaling match scores in the training loss.

The cover radius ε = 0.83 reduces skill banks by about 8.5×, with a reported loss of at most 1.6 percentage points on the evaluated benchmarks.
SkillRet validation sets (α, γ) = (1.0, 0.025) and pruning margin Δ = 0.133. Only candidates within Δ of the best glance score receive a verdict, approximately nine per query on average.
The projections and four scalar settings remain fixed for other libraries. This zero-shot transfer evaluates the trained router without target-library training or calibration.

4.2. Routing from written tasks

Written-task evaluation uses SkillRet v1’s 4,997 queries and 6,660 held-out skills, a stratified sample of 861 SRA-Bench tasks over 26,262 skills, and Eval-Core’s 75 task.md queries over two document pools. Adjudicated Hit@1 measures whether the single selected skill serves the task: because annotations omit valid alternatives, GPT-5.6 Sol (high) compares non-gold selections with every annotated gold skill in both presentation orders and grants credit when wins are at least as frequent as losses.

4.2.1 Comparison with current practice: progressive disclosure receives twenty candidates from Qwen3-Embedding-8B. Other pipelines combine SkillRouter’s 0.6B embedder and 0.6B reranker or Qwen3-Embedding-8B and Qwen3-Reranker-8B; BM25, a lexical retrieval baseline, and SKILLRET-Emb-0.6B provide additional references.
Gavel achieves adjudicated Hit@1 of 0.9460 on SkillRet, 0.8734 on SRA-Bench, 0.9467 on Eval-Core easy, and 0.9333 on Eval-Core hard. Its gains over the strongest baseline are 3.8, 13.4, and 1.3–2.7 percentage points, respectively.
The glance alone is strongest among retrieval stages on SRA-Bench and Eval-Core, but SKILLRET-Emb-0.6B slightly exceeds it on SkillRet. The detailed results therefore qualify the paper’s broader statement that Gavel leads at both stages.
4.2.2 Ablations, experiments that replace one design choice at a time: replacing the glance with Jina-ColBERT-v2, selecting solely by V, predicting the skill body instead of the task, or replacing the verdict with progressive disclosure reduces adjudicated accuracy on every benchmark. These comparisons support contributions from the agent-derived representations, score combination, and direction of likelihood scoring.

Retrieval-stage and full-pipeline Hit@1 on SkillRet, SRA-Bench, and the two Eval-Core pools.

Changes in Hit@1 when individual Gavel design choices are replaced.

4.3. Routing mid-rollout

SkillTraj contains 372 simulated trajectories with marked points where a skill becomes necessary: user request (116), tool evidence (106), agent plan (105), and wrong-skill recovery (45). GPT-5.6 Sol generates the dialogues, and a Claude judge checks that the need is absent before the marked point, present afterward, and addressed by the gold skill.

The glance reads the live rendered context and uses recency weights wi = 2^(−ai/64), where ai is the number of tokens between a position and the decision point. The verdict uses the most recent message as its task continuation.
Gavel’s adjudicated Hit@1 is 0.9655, 0.7264, 0.9048, and 0.8889 across the four scenarios, exceeding the strongest alternative by 8.6–21.9 percentage points.
Both external retrieval pipelines improve when given only the last message instead of the full trajectory. Progressive disclosure over the glance shortlist falls to 0.1524 on agent plan, where selecting a skill requires details beyond its metadata.
Every system receives the routing moment in this evaluation. The experiment measures skill choice at that moment, with autonomous timing evaluated separately in deployment.

SkillTraj routing accuracy by scenario, comparing full-context and last-message retrieval with Gavel.

4.4. End-to-end deployment

For deployment in mini-swe-agent, a minimal bash-agent harness, the authors train an additional linear gate to decide when to invoke routing. It reads accumulated glance scores and the current final-layer hidden state, fires after two consecutive positive token decisions, and loads nothing if the winning skill’s verdict has V < 0.

Skill-Use contains 177 executable tasks and 79 skills. Gavel with Qwen3-32B reaches a correct-skill trigger rate of 0.909, versus 0.011 for progressive disclosure in the same harness, 0.897 for the 8B-plus-8B retrieval pipeline, and 0.800 for SkillRouter.
The highest cited frontier-model trigger rate in Codex is MiniMax-M3’s 0.864. Those frontier results come from the Skill-Use paper and use a different harness, so the comparison includes both model and harness differences.
The paper reports that Gavel loads nothing on another approximately 5% of tasks. Deployment requires gate training beyond the two projections used for skill ranking.

Correct-skill trigger rates on Skill-Use for progressive disclosure, retrieval pipelines, and Gavel.

4.5. Scaling with backbone capability

On SRA-Bench, both the glance and full Gavel improve across Qwen3 sizes from 0.6B to 32B and across model generations. Gavel on the 0.6B backbone already exceeds progressive disclosure on the 32B backbone under the tested twenty-candidate retrieval setup.

On Qwen3.8-27B, Gavel remains 12.2 percentage points ahead of metadata-based selection using the same backbone.
The scaling study supports extracting better routing signals from more capable backbones. Transfer of trained projections between different backbones remains untested.

SRA-Bench routing accuracy across backbone sizes and generations.

5. Conclusion

The paper concludes that a frozen agent contains useful skill-selection information that trained projections and native predictive scores can make operational without adding library metadata to the working context. Its experiments evaluate skill documents; extending the read-out to tools, memories, or Model Context Protocol (MCP) servers, which expose external capabilities to agents, remains an open question.

Appendix

A. Native read-outs that fail: direct use of native attention queries and keys performs poorly across independently encoded documents. The best of 24 untrained configurations reaches SkillRet validation Hit@1 of 0.001 versus 0.918 for the trained glance; native-head initialization and a six-head structure also underperform the adopted randomly initialized single wide head. Appending 64 trained retrieval tokens learns recency-focused aggregation rather than selective attention to older evidence.
B. Layer selection: after removing the highest-norm token, median matrix entropy over 500 skills falls from 4.01 to 1.83 across one block and bottoms out at layer 44. Training read-outs at six candidate depths confirms that layer 44 has the best recall at 20, the fraction of queries whose twenty candidates contain an annotated gold. Raw cosine similarity between token states averaged across each text span instead performs poorly at this depth.
C. Training details: Wq and Ws are bias-free 768 × 5120 matrices trained in fp32, 32-bit floating-point precision, with the AdamW optimizer for 24,000 steps, learning rate 10^−3, and weight decay 0.01. Each step contains up to 256 tasks within a 200K skill-token budget; training allows at most 8,192 keys per skill, while evaluation uses a 30,720-token serving budget. Compression is followed by 3,000 Wq-only steps at learning rate 10^−4; written-task voting uses uniform token weights.
D. Compression guarantees: the appendix proves the maximum-inner-product distortion bound using the Cauchy–Schwarz inequality and derives vote stability with an extra term for tokens near top-k boundaries. It also proves that farthest-first traversal produces an ε-cover that is ε-separated. These bounds apply to representation scores; skill rankings can still change when candidate scores are close.
E. A non-Qwen backbone: the pipeline is rebuilt on gemma-4-31b-it with newly trained projections, read-out at block 34 of 60, ε = 0.77, and recalibrated (α, γ) = (0.15, 0.003). Gavel again achieves the highest adjudicated scores on the three written-task benchmarks. This supports applicability to a second model family with model-specific training and calibration.
F. Additional ablations: F.1. Progressive disclosure over full skill bodies expands the twenty-entry menu to complete documents, using a 131K-token window with a longest prompt of 124K tokens; it improves SRA-Bench but remains about 16 adjudicated points behind Gavel and raises median prompts from 1.3K to 28K–52K tokens. F.2. Dropping one verdict signal shows that the full combination exceeds either single-signal variant in adjudicated accuracy, including by 4.6 points on pooled SkillTraj. F.3. Replacing the glance with an off-the-shelf embedder shows that Gavel’s verdict substantially improves Qwen3-Embedding-8B candidates, yet the resulting pipeline still trails full Gavel by 9.1 points on SkillTraj.
G. Prompt templates: the appendix specifies installation, task, verdict, progressive-disclosure, reversed-order ablation, and adjudication prompts in non-thinking mode, which uses the shown empty thinking block. The verdict continuation extends the installation render exactly, supporting reusable prefixes; its yes/no score includes capitalization variants with and without leading spaces. The adjudicator sees the task and two skill documents with their roles swapped between comparisons.
H. Benchmark details and disclosures: H.1. Under-annotated golds: examples from all three benchmarks documents duplicate or adequate alternative skills that raw Hit@1, which credits only annotated golds, counts as errors; Eval-Core’s easy pool has 78,361 documents and its hard pool adds 780 adversarial distractors. H.2. Adjudication: raw scores and human agreement reports that Gavel’s main comparisons also hold under raw Hit@1, while author annotations agree with judge credit decisions on 171 of 200 pairs, reported as 86%; the judge grants a net nine additional credits relative to those annotations. H.3. Candidate-pool recall of the retrieval stages reports glance R@20, recall at 20, of 0.9706 on SkillRet and 0.9675 on SRA-Bench. H.4. SkillTraj: construction and examples describes two to eight user turns, up to five tool calls, blind before/after checks by Claude Sonnet 4.6, and screens against gold-name and verbatim-body leakage.
I. Deployment walkthroughs: Gavel makes 181 loads across 177 tasks, including seventeen loads in later turns after command output appears. Examples illustrate form editing, migration to a new software development kit, and a task requiring two different skills. Forcing Qwen3-32B’s first action to read its chosen metadata-menu skill raises the baseline trigger score to 0.834, indicating that its 0.011 autonomous result largely reflects failure to initiate skill consultation.
J. The cost of routing: under the paper’s August 2026 pricing assumptions, nine verdict passes over 100 metadata tokens, a 1,895-token body, and a 239-token query cost approximately $0.080 per gate firing. A cached progressive-disclosure menu costs approximately $4 × 10^−5 nT for n skills and T calls, yielding a stated crossover near n > 2,000m/T for m firings. The roughly half-second verdict latency is an estimate for a modern serving stack; the calculation excludes offline indexing and the selected-body costs shared by both approaches.
K. Discussion: the linear gate is a preliminary timing mechanism; sequence-aware gating and dedicated skill-call tokens remain future work. Experiments run the backbone locally. The proposed deployment behind a cloud application programming interface (API) requires the provider to run the projections, indexing, verdict passes, and gate internally because the router needs hidden states unavailable through a text-only interface.

Brief Thoughts

The fixed-calibration evaluations, component replacements, and raw-score checks provide complementary evidence: the gains persist across changes in query style and document genre and also appear without adjudication. The second model-family experiment adds support for the architecture, although it rebuilds the router with new parameters.

The evidence concerns skill selection and loading under specific evaluation conditions. SkillTraj is simulated, Eval-Core has only 75 queries, the human check uses author annotations, and Skill-Use measures correct-skill loading rather than final execution success. Access to hidden states and token probabilities, per-model projection training, bank storage, and verdict computation remain material deployment requirements despite the small number of ranking parameters.