LIMI: Less is More for Agency | Summary

Contents

Summary
1 Introduction
2 Preliminary
- 2.1 Long-Horizon Tasks and Agentic Complexity
- 2.2 Domain Specification: Vibe Coding and Research Workflows
3 Dataset Construction
4 Experiments
5 Related work
- 5.1 Agentic Language Model
- 5.2 Data Efficiency in Language Models
6 Conclusion
Appendix
Brief Thoughts

This article explains the key points of LIMI: Less is More for Agency.

2025-09-22 (arXiv)
Xiao, Yang, Jiang, Mohan, Sun, Jie, Li, Keyu, Lin, Jifan, Zhuang, Yumin, Zeng, Ji, Xia, Shijie, Hua, Qishuo, Li, Xuefeng, et al.
SJTU, SII, PolyU, USTC, GAIR
Paper
Github
Project Page

Read this article in Korean

Summary

LIMI: Less is More for Agency investigates whether language models can improve agency—the capacity to discover problems, formulate hypotheses, and execute solutions through interaction with environments and tools—using a small set of curated demonstrations. After fine-tuning GLM-4.5 on 78 demonstrations, LIMI achieves a reported 73.5% average on AgencyBench, a benchmark of collaborative coding and research tasks evaluated with an execution environment, compared with 45.1% for the same base model and 47.8% for GLM-4.5 fine-tuned on 10,000 AFM-CodeAgent-SFT-Dataset samples.

AgencyBench performance and training-sample comparisons for LIMI and baseline models.

The method combines 60 real-world queries with 18 queries synthesized from GitHub pull requests, proposed code changes submitted to a repository. GPT-5 and four human collaborators collect successful trajectories, complete interaction sequences containing reasoning, tool calls, and observations, in SII CLI, a command-line interface for agent execution; trajectories average 42.4k tokens and range from 13k to 152k tokens.
The clearest result is the improvement over the same pretrained model on AgencyBench; gains on the additional benchmarks vary by task. Demonstration count does not measure training-token or collection cost, and differences in aggregate composition prevent the reported averages from isolating the effect of SII CLI access.

1 Introduction

The introduction asks whether agentic capabilities require large training datasets or can improve through selective demonstration curation. LIMI, expanded in the paper as Less Is More for Intelligent Agency, combines authentic task collection, pull-request-based query synthesis, and complete successful interaction traces.

The motivation extends the small-data approach of LIMA, a study of adapting model responses to desired behavior using 1,000 examples, and LIMO, a mathematical reasoning study using 817 examples, to collaborative agent execution.
The authors call their claim the Agency Efficiency Principle: curated demonstrations can support effective autonomy with few examples. The experiments compare selected datasets and two pretrained models rather than establishing a general scaling law, a quantitative relationship between resources and performance.

2 Preliminary

The preliminary framework defines agency through autonomous problem discovery, hypothesis formulation, and solution execution with environments and tools. Its operational emphasis is the integration of task execution, multi-step reasoning, and human collaboration.

2.1 Long-Horizon Tasks and Agentic Complexity

Long-horizon tasks, tasks requiring sustained work across multiple interactions, demand state tracking, objective decomposition, coordinated tool use, and plan revision after feedback. Figure 2 illustrates this with Gomoku, a board game centered on forming five stones in a row, whose five connected subtasks progress from board rendering to stronger game-playing algorithms.

The example combines implementation, validation, state management, and performance constraints within one query. A single complete demonstration can therefore contain supervision for several distinct behaviors.

A Gomoku development query with five interconnected subtasks and explicit implementation requirements.

2.2 Domain Specification: Vibe Coding and Research Workflows

Vibe coding denotes collaborative software development in which language models work alongside human developers, including code understanding, environment navigation, debugging, and changing requirements. Research workflows include literature search, data analysis, experimental design, hypothesis generation, and communication of findings.

The authors select these domains to collect demonstrations that connect reasoning to executable work. Their assertion that the domains cover most knowledge work is not accompanied by a quantitative coverage analysis.

3 Dataset Construction

Dataset construction separates query selection and quality review from trajectory collection. Figure 3 shows real-world and synthesized queries entering a shared pool before human–GPT-5 collaboration produces recorded execution sequences.

Query construction, quality review, and collaborative trajectory collection in SII CLI.

3.1 Framework and Notation

Each training interaction is represented as \((q_i,\tau_i)\): q_i specifies the user’s objective and success criteria, while τ_i = {a_i,1, …, a_i,n_i} records the subsequent interaction sequence. Actions comprise model reasoning m_i,j, structured tool calling t_i,j, and environment observation o_i,j, including tool outputs and human feedback.

The index j preserves interaction order, and n_i is the number of actions for query i. The representation records the task-solving process as well as its outcome.

3.2 Query Pool Construction

The query pool contains 60 tasks collected from professional development and research scenarios and 18 selected synthetic tasks generated with GPT-5 from GitHub pull requests. Four computer science PhD students review the semantic alignment between synthesized queries and their source changes.

The pipeline selects 100 repositories with more than 10,000 GitHub stars, collects 1,000 pull requests per repository, and randomly samples 100 per repository for synthesis.
Filtering retains unified diff patches, text representations of code changes, below 1,200 tokens and excludes changes confined to Markdown files. Several thousand synthetic queries are generated before selecting the 18 used in LIMI.
The paper does not report an ablation, an experiment removing or varying one component, that separates the contributions of real-world queries, synthetic queries, and expert review.

3.3 Trajectory Collection for Training Dataset

Four PhD student collaborators work with GPT-5 in SII CLI and repeat collection until each of the 78 queries is successfully resolved. The recorded trajectories include reasoning, tool invocations, observations, clarification, refinement, and recovery from intermediate failures.

Figure 4 reports a minimum of 13k tokens, a maximum of 152k tokens, and a mean of 42.4k tokens per trajectory. The small example count therefore represents substantial interaction content.
SII CLI is selected for integrated tools, detailed logging, and human collaboration. The paper does not quantify GPT-5 generation cost, annotation hours, or unsuccessful collection attempts.

Training trajectory lengths and task-domain coverage across coding and research workflows.

3.4 Evaluation Benchmarks

AgencyBench contains 10 tasks: four collaborative coding tasks and six research or information-processing tasks, as summarized in Table 1. Evaluation also covers tau2-bench-airline and tau2-bench-retail for tool use, EvalPlus-HumanEval and EvalPlus-MBPP for code generation, DS-1000 for data science programming, and SciCode for scientific computing.

First-Turn Functional Completeness (FTFC) measures the percentage of requirements implemented correctly in the initial response. Success Rate (SR@R) measures completion within R allocated rounds, with R = 3.
Remaining Chances (RC@R) is described as the average number of unused rounds on successful queries. Table 2 reports RC@3 on a percentage-like scale without explaining the normalization.
For tau2-bench, the paper names Pass^4, a measure of success over four independent runs, and describes it as the fraction of those runs that succeed. It does not provide an explicit aggregation formula; the other benchmarks use accuracy.

4 Experiments

The experiments compare pretrained baselines, variants fine-tuned on LIMI data, and GLM-4.5 variants fine-tuned on three alternative agent datasets. They also compare evaluation with and without SII CLI access to examine the role of the execution environment.

4.1 Experiment Setup

Supervised fine-tuning, training on recorded demonstrations, uses the slime framework to adapt GLM-4.5 and GLM-4.5-Air into LIMI and LIMI-Air. The paper states that fine-tuning configurations are identical across dataset comparisons, but provides no numerical learning rate, batch size, epoch count, sequence handling policy, or compute budget.

Table 2 lists LIMI at 355B parameters and LIMI-Air at 106B parameters, where B denotes billion. Baselines include GLM-4.5, GLM-4.5-Air, Qwen3-235B-A22B-Instruct, DeepSeek-V3.1, and Kimi-K2-Instruct.
Alternative training sets are CC-Bench-trajectories with 260 samples, AFM-CodeAgent-SFT-Dataset with 10,000 samples, and AFM-WebAgent-SFT-Dataset with 7,610 samples.
AgencyBench is evaluated with the execution environment because its tasks require tool interactions. Evaluation without SII CLI is restricted to the additional benchmarks.

4.2 Main Results

On AgencyBench, LIMI improves FTFC from 37.8% to 71.7% and SR@3 from 47.4% to 74.6% relative to GLM-4.5; the reported average rises from 45.1% to 73.5%. Other baseline averages are 24.1% for Kimi-K2-Instruct, 11.9% for DeepSeek-V3.1, and 27.5% for Qwen3-235B-A22B-Instruct.

Table 3 reports LIMI scores of 34.0% on TAU2-bench-airline, 45.6% on TAU2-bench-retail, 36.6% on DS-1000, 92.1% on EvalPlus-HumanEval, and 82.3% on EvalPlus-MBPP.
SciCode Main Problem and Sub Problem scores, measuring performance on full scientific problems and their constituent subproblems, are 3.1% and 25.3%, respectively. LIMI ties Kimi-K2-Instruct on HumanEval and SciCode Main Problem, trails it on airline tasks, and trails DeepSeek-V3.1 on DS-1000.
Table 3 reports an average of 57.2% versus 43.0% for GLM-4.5, and its caption states that AgencyBench is included. Averaging its seven displayed metrics with the three AgencyBench metrics gives approximately these reported aggregates; the paper does not explicitly specify that formula.

AgencyBench results for pretrained models, LIMI variants, and alternative fine-tuning datasets.

Results on tool-use, data science, code-generation, and scientific computing benchmarks with SII CLI access.

4.3 Data Efficiency Analysis

With the same GLM-4.5 base, LIMI’s 78 demonstrations yield a 73.5% AgencyBench average, compared with 29.2% for 260 CC-Bench samples, 47.8% for 10,000 code-agent samples, and 36.7% for 7,610 web-agent samples. This comparison supports the effectiveness of the curated dataset under the reported training setup.

The gap between 73.5% and 47.8% is 25.7 percentage points, approximately 53.7% relative improvement. Section 4.3 incorrectly labels 53.7 as a percentage-point improvement.
The sample-count ratio of 10,000 to 78 is approximately 128. The paper also reports relative gains of 39.9%, 69.7%, and 45.9% over the alternative datasets using Table 3 aggregates, which include AgencyBench.
No comparison matches total training tokens or annotation effort. Dataset size, trajectory length, task relevance, GPT-5 supervision, and selection strategy remain intertwined.

4.4 Generalization Analysis

LIMI-Air improves the AgencyBench average of GLM-4.5-Air from 17.0% to 34.3%, a gain of 17.3 percentage points. LIMI improves GLM-4.5 from 45.1% to 73.5%, a gain of 28.4 percentage points.

These results demonstrate gains at two model sizes within the GLM-4.5 family. They do not test adaptation across unrelated model architectures.
LIMI-Air’s reported Table 3 aggregate rises from 33.2% to 39.0%, while individual scores decline from 34.7% to 34.4% on DS-1000 and from 90.2% to 89.0% on HumanEval.

4.5 The Impact of CLI Environment

Without SII CLI access, Table 4 reports averages of 50.0% for LIMI and 48.7% for GLM-4.5. LIMI improves airline, DS-1000, HumanEval, MBPP, and SciCode Sub Problem scores, ties SciCode Main Problem, and declines on retail from 52.6% to 49.1%.

The paper interprets Table 3’s 57.2% versus Table 4’s 50.0% as a 7.2-percentage-point benefit from SII CLI. Table 3 includes AgencyBench, whereas Table 4 omits it, so this difference does not isolate the environment effect.
For LIMI, the displayed matched scores are lower with SII CLI on airline, retail, DS-1000, HumanEval, and both SciCode metrics; MBPP improves from 80.4% to 82.3%. The results do not show a uniform benefit from SII CLI access.
The reported Table 4 averages imply a 1.3-percentage-point gain over the base model. However, LIMI’s seven displayed scores have an arithmetic mean of approximately 49.7%, not 50.0%; the paper does not explain this discrepancy.

Generalization benchmark results without SII CLI environment access.

Related work places LIMI at the intersection of language-model agents and data-efficient adaptation. The discussion connects earlier tool-learning and agent-training approaches with selective curation of complete collaborative demonstrations.

5.1 Agentic Language Model

Toolformer studies learning external tool use, while ReAct interleaves reasoning traces and task-specific actions. The paper also discusses autonomous task decomposition, GLM-4.5 and Kimi-K2 agent capabilities, and reinforcement learning, training based on rewards, as context for contemporary agent development.

LIMI uses supervised demonstrations without introducing a new reinforcement-learning objective or agent architecture. Its experimental contribution centers on selecting and collecting successful collaborative execution traces.

5.2 Data Efficiency in Language Models

The data-efficiency discussion cites LIMA’s 1,000 curated prompt–response examples and LIMO’s 817 mathematical reasoning examples as precedents. LIMI applies this motivation to demonstrations involving tools, environmental feedback, and human collaboration.

These studies motivate testing small curated datasets, but their different tasks and evaluation settings do not establish a shared quantitative efficiency law.

6 Conclusion

The conclusion presents the Agency Efficiency Principle and reiterates the 73.5% AgencyBench result from 78 demonstrations. The experiments support a substantial adaptation gain for the evaluated collaborative tasks, while the claim that agency follows fundamentally different scaling principles extends beyond the reported dataset comparisons.

Appendix

A Agency Bench Case supplies all 10 task examples, organized into A.1 Vibe Coding and A.2 Research Workflow. The coding tasks build a C++ console chat system, a Java task manager, a Gomoku game, and a local pipeline progressing through persistent storage, orchestration, planning, and autonomous repair; tasks 6 and 8 are presented as queries without their accompanying raw code and data.
A.2 Research Workflow covers evaluation on DynToM, a dataset concerning changes in human states, statistical comparison of standard and reasoning-enabled model responses, dataset discovery and metadata extraction, scientific function fitting, and multi-condition searches about basketball players and companies. Function-fitting requirements progressively demand loss below 1 × 10^−3, 1 × 10^−5, 1 × 10^−6, and 1 × 10^−7; these are task acceptance thresholds, not reported model achievements.
B PR Query Synthesis Prompt provides the template for turning pull-request descriptions, code changes, commits, and discussion context into actionable developer queries. Its output includes test_query, repo_id, pr_id, task_category, modified_files, related_files, and reasoning, with one primary task category and a testable implementation request.
C Case Study compares GLM-4.5 and LIMI on selected AgencyBench tasks. C.1 Real-world Vibe Coding Scenarios reports successful completion of the chat-system subtasks by LIMI and fewer problems in Gomoku development, while acknowledging that LIMI’s final game-playing difficulty implementation was imperfect.
C.2 Research Workflow Applications reports expert preference for LIMI’s retrieved datasets and fewer hints in several basketball identification subtasks. On function fitting, the reported losses are 1.14e-6 for GLM-4.5 after multiple interactions and 5.95e-7 for LIMI on its first attempt; the latter is approximately 1.9 times smaller, although the source describes it as an order of magnitude smaller.

Brief Thoughts

The useful demonstration design captures execution, feedback, and recovery across complete tasks. The same-base AgencyBench comparison provides the clearest evidence of its effectiveness: FTFC increases by 33.9 percentage points and SR@3 by 27.2 percentage points. A controlled follow-up could match training tokens and collection effort while varying query selection, the generating model, trajectory length, and human intervention. This would help identify which parts of the collection process produce the gains. The paper reports no repeated-run variability, confidence intervals, or explicit training–evaluation overlap audit. AgencyBench’s 10 top-level tasks and the selected qualitative cases leave broad generalization uncertain. The numerical and aggregation discrepancies warrant clarification before using the reported averages to infer a general efficiency principle or an execution-environment benefit.