Agent Laboratory: Using LLM Agents as Research Assistants | Summary

This article explains the key points of Agent Laboratory: Using LLM Agents as Research Assistants.

• 2025-01-08 (arXiv)
• Schmidgall, Samuel, Su, Yusheng, Wang, Ze, Sun, Ximeng, Wu, Jialian, Yu, Xiaodong, Liu, Jiang, Moor, Michael, Liu, Zicheng, Barsoum, Emad.
• AMD, Johns Hopkins University, ETH Zurich
• Paper
• Github
• Project Page

Read this article in Korean

Summary

• Agent Laboratory addresses the time and cost of turning a human research idea into machine learning experiments, code, and a readable report using large language model (LLM) agents, systems that combine text-generating models with tools and structured workflows. With gpt-4o, its reported workflow costs $2.33 per paper versus approximately $15 for the earlier gpt-4o autonomous research workflow cited by the authors, corresponding to the abstract’s approximately 84% reduction in reported inference expenditure.
• The framework connects Literature Review, Experimentation, and Report Writing, with optional human feedback after each subtask. Its mle-solver module generates, executes, and revises experimental programs, while paper-solver generates and revises a report using automated reviewer feedback.
• For 15 autonomously generated papers across five topics and three underlying language models, human reviewers rated o1-preview highest in usefulness at 4.4/5, while o1-mini achieved the highest experimental quality at 3.2/5. Automated reviews averaged 6.1/10 overall versus 3.8/10 from humans; co-pilot mode, which adds human checkpoints after subtasks, received an external overall rating of 4.38/10 compared with that pooled autonomous average.
• On 10 low-complexity text and tabular challenges from MLE-Bench, a benchmark based on Kaggle machine learning competitions, mle-solver earned four medals and exceeded median human performance on six challenges. This restricted solver evaluation complements the paper reviews, but fabricated experimental details and unreliable automated assessment limit what the results establish about autonomous research.

1. Introduction

The paper presents Agent Laboratory as an assistant for implementing researchers’ own ideas, motivated by the limited time, computing resources, and funding available to explore them. It introduces an open-source workflow with adjustable computing budgets and human involvement, evaluated through researcher surveys, generated-paper reviews, runtime measurements, and an isolated evaluation of mle-solver.

The diagram places responsibility for the research direction and initial constraints with the human. Its output artifacts and seven subtasks show that the intended assistance includes interpretation and report refinement as well as coding.

2. Background & Related Work

The background connects autoregressive language models, which predict successive tokens from preceding text, with agents that use external tools and iterative refinement to execute tasks. Related work covers automated machine learning, scientific discovery, literature search, research ideation, code generation, and paper reviewing.

• MLE-Bench, DS-bench, and MLAgentBench are described as using 75, 74, and 6 Kaggle challenges, respectively, to evaluate aspects of machine learning engineering.
• ResearchAgent generates and refines research ideas; The AI Scientist automates experiments and manuscript generation. The authors motivate human-guided research directions using prior findings about weaknesses in the feasibility, implementation, and diversity of model-generated ideas.
• Virtual Lab uses agents alongside a human researcher for scientific work, while ChemCrow and Coscientist support chemistry tasks. These examples motivate the framework; the paper does not experimentally compare the complete research systems.

3. Agent Laboratory

Agent Laboratory accepts a research idea and notes, gathers relevant literature, develops an experimental plan, prepares data, runs experiments, interprets results, and produces a report and code repository. Agents receive prompted roles such as PhD Student, Postdoc, ML Engineer, and Professor, representing doctoral research, postdoctoral research, machine learning engineering, and academic assessment responsibilities.

• The three primary phases are Literature Review, Experimentation, and Report Writing.
• mle-solver handles experimental programming, and paper-solver handles report generation and revision.
• The project URL provided in the paper is https://AgentLaboratory.github.io/.

The workflow locates agent dialogue, external tools, and solver loops within the three primary phases. The return path from report refinement to earlier work indicates that revision can revisit the experimental process.

3.1. Literature Review

The PhD agent searches arXiv, a research-paper repository, through its application programming interface and curates references for subsequent stages. The summary action retrieves abstracts for the top 20 papers matching a query, full text retrieves a selected paper’s contents, and add paper incorporates selected summaries or full texts into the review.

• The agent issues queries and inspects relevance before adding papers.
• The phase ends when the configured number of relevant texts has been added; Appendix A lists five paper summaries for the reported configuration.

3.2. Experimentation

Experimentation begins with dialogue between the PhD and Postdoc agents to specify models, datasets, and experimental steps. Data preparation uses Python execution and search over HuggingFace, a dataset repository, with execution feedback supporting correction before submission.

• mle-solver initializes a Python program and refines candidates using EDIT, which replaces a specified range of lines, and REPLACE, which regenerates the entire program.
• A language-model reward function, a model-based scoring procedure, rates alignment with the intended experiment from 0 to 1 using the research plan, code, and observed output. This score assesses task fulfillment and does not independently verify scientific validity.
• Self-reflection produces suggestions from execution errors or results. The solver retains high-scoring programs, randomly samples a program from that pool, and generates parallel modifications, selecting the best candidate to replace the lowest-scoring retained program.
• The PhD and Postdoc agents discuss the experimental results, and the Postdoc submits an interpretation for report writing.

Execution results and a reward function guide the search over programs, while a retained pool supplies candidates for later modifications. The scoring function therefore affects which executable programs receive further development.

3.3. Report Writing

paper-solver turns the preceding work into a report intended to help a researcher understand what was accomplished. It creates an eight-section LaTeX scaffold, a document template that can be compiled into a report, searches for additional arXiv references when needed, applies line edits, and scores reports with an adapted automated reviewing system.

• The scaffold contains Abstract, Introduction, Background, Related Work, Methods, Experimental Setup, Results, and Discussion. Proposed edits are checked by compiling the LaTeX document.
• During refinement, three reviewer agents provide feedback modeled on the assessment guidelines of NeurIPS, a machine learning conference. The PhD agent can finalize the project or revisit planning, experimentation, or interpretation.
• 3.3.1. Autonomous versus Co-Pilot Mode: Autonomous mode requires only the initial human research idea. Co-pilot mode adds a checkpoint after each subtask, where the researcher can approve progress or request repetition with feedback.
• The cited reviewer’s calibration results on 500 papers from ICLR 2022, another machine learning conference, concern prior work. Agent Laboratory separately tests whether automated scores agree with human assessments of its generated reports.

The loop separates initial document construction from editing and automated review. Compilation tests whether the document builds; the reviewer provides a quality judgment, without directly checking every experimental claim against execution records.

4. Results

The evaluation examines autonomous research outputs, co-pilot use, workflow cost and runtime, and mle-solver on competition tasks. Ratings of usefulness and paper quality assess different outcomes from competition scores; the latter evaluate the solver separately from the complete research workflow.

4.1. Evaluation of quality by language model

Five research questions and three backends, the underlying models gpt-4o, o1-mini, and o1-preview, produced 15 autonomous papers. Ten volunteer PhD students each reviewed three randomly assigned papers, rating experimental quality, report quality, and usefulness from 1 to 5.

• The topics were cognitive biases, systematic tendencies in model judgments; image-model sensitivity to pixel noise; differential diagnosis, considering alternative diagnoses, on the medical question-answering dataset MedQA; word-order sensitivity in multiple-choice questions; and gender role play in mathematical question answering.
• For gpt-4o, average experimental quality, report quality, and usefulness were 2.6/5, 3.0/5, and 4.0/5. Corresponding values were 3.2/5, 3.2/5, and 4.3/5 for o1-mini, and 2.9/5, 3.4/5, and 4.4/5 for o1-preview.
• Performance varied by topic. For image noise, experimental quality ranged from 1.5/5 with gpt-4o to 4.0/5 with o1-mini.

Usefulness scores exceed experimental-quality scores for each backend. o1-preview leads in average usefulness and report quality, while o1-mini leads in experimental quality, so the ranking depends on the assessment dimension.

4.1.1. Human reviewer scores by language model evaluates the same papers using NeurIPS-style criteria. Human overall scores were 3.5/10 for gpt-4o, 3.8/10 for o1-mini, and 4.0/10 for o1-preview, while automated reviewers averaged 6.1/10 against the human average of 3.8/10.

• Figure 6 reports human averages of 2.0/4 for quality, 2.3/4 for significance, 2.4/4 for clarity, 1.9/4 for soundness, 2.5/4 for presentation, and 2.1/4 for contribution. Soundness assesses whether technical claims and methods are supported by evidence.
• Automated reviewers rated clarity at 3.6/4 and contribution at 2.9/4, compared with human ratings of 2.4/4 and 2.1/4. Higher automated ratings occur across all reported criteria.
• The cited average for accepted NeurIPS 2024 papers is 5.85/10. This is an accepted-paper average, not a universal acceptance threshold.
• The prose and Figure 6 reverse the gpt-4o and o1-mini soundness values; the figure shows 1.8/4 and 1.7/4, respectively. Both give o1-preview 2.2/4.

The score distributions and criterion averages show more favorable automated assessments than human assessments. The automated and human overall averages also order the backends differently, weakening the case for using automated scores as a substitute for external review.

4.2. Evaluation of co-pilot quality

Researchers used co-pilot mode for one custom topic and one topic selected from the five autonomous-evaluation topics, producing two papers per researcher. All phases except Literature Review used o1-mini; the study evaluates both the experience of using the tool and the quality of its outputs.

• 4.2.1. Quality as a tool reports utility of 3.5/5, continuation intent of 3.75/5, satisfaction of 3.63/5, and usability of 4.0/5. Custom topics received higher utility and continuation ratings, while preselected topics received higher usability ratings.
• Co-pilots rated experimental quality at 2.38/5, report quality at 3.13/5, and usefulness at 3.75/5. Each was lower than the corresponding autonomous o1-mini rating; participant feedback described difficulty directing agents toward the researcher’s intended project.
• 4.2.2. Evaluation of co-pilot generated papers reports overall ratings of 4.13/10 from the researchers themselves and 4.38/10 from external evaluators. The external score exceeded the pooled autonomous average of 3.8/10 by 0.58 points.
• Compared with the pooled autonomous ratings, external co-pilot ratings increased by 0.75 points for quality and 0.48 for soundness on four-point scales. Significance declined by 0.05 and contribution increased by 0.03, limiting claims about improved scientific contribution.
• Participants requested better inspection of intermediate outputs, a graphical user interface, more report figures, and improved literature review. The study measures ratings and feedback rather than long-term research productivity.

The panels separate user experience, researchers’ self-assessments, and external paper reviews. External co-pilot scores exceed the pooled autonomous averages most clearly in quality and soundness, while significance and contribution change little.

4.3. Runtime statistics

Reported workflow costs were $2.33 for gpt-4o, $7.51 for o1-mini, and $13.10 for o1-preview; corresponding runtimes were 1165.4, 3616.8, and 6201.3 seconds. The comparison with the cited approximately $15 gpt-4o autonomous research workflow shows lower reported inference expenditure, but does not establish equal scientific output quality.

• Report Writing cost $1.73 for gpt-4o, $2.58 for o1-mini, and $9.58 for o1-preview. It was the largest cost component for gpt-4o and o1-preview; Data Preparation was largest for o1-mini at $3.03.
• Average subtask success rates were 94.3%, 92.8%, and 95.7% for gpt-4o, o1-mini, and o1-preview. These averages do not measure the probability that a complete workflow succeeds.
• Literature Review success rates were 60%, 70%, and 80%, respectively. Data Preparation succeeded at 100%, 80%, and 90%, while the other listed subtasks reached 100%.

Report writing dominates cost for gpt-4o and o1-preview, while data preparation costs most for o1-mini. Literature review has the lowest success rates, and the displayed workflow success percentages summarize average subtask success rather than measured end-to-end completion probabilities.

4.4. Evaluating mle-solver on MLE-Bench

The isolated mle-solver evaluation uses all 10 text and tabular challenges in MLE-Bench’s low-complexity category. Inputs include dataset descriptions, distilled knowledge from Kaggle notebooks containing competition code and analysis, and prepared arrays; candidate programs are scored on a randomly selected 20% development split, with the remaining 80% used for training and the selected program evaluated on the competition test set.

• Development-set performance replaces the language-model reward function used in the general research workflow. The benchmark therefore tests a solver configuration with an objective program-selection metric.
• mle-solver produced valid solutions for all 10 selected challenges within two hours and earned two gold medals, one silver medal, and one bronze medal under the benchmark’s Kaggle medal criteria. OpenHands earned two gold medals, AIDE earned one gold and one bronze, and MLAB earned none.
• The reported counts above median human performance were six for mle-solver, five for AIDE, two for OpenHands, and zero for MLAB.
• MLAB and OpenHands used gpt-4o, while AIDE used o1-preview. Invalid baseline submissions were excluded from their average scores, and the study covers 10 selected challenges rather than all 75 MLE-Bench challenges.

mle-solver earns medals on four selected challenges, including two gold, one silver, and one bronze. AIDE has better scores on several individual tasks, so the comparison supports greater medal coverage within this subset without establishing superiority on every task.

5. Limitations

The limitations distinguish weaknesses in model judgment from restrictions imposed by the workflow. Execution failures and inaccurate reporting qualify the claims about autonomous completion and research quality.

5.1. Workflow limitations

LLM-based self-evaluation is an imperfect selection signal: the authors describe reports with weaker figures than those of The AI Scientist despite higher automated scores. Scoring an assistant’s report as a complete conference paper also creates a mismatch between paper-solver’s stated purpose and its evaluation criterion.

• The fixed report structure restricts alternative organizations, and the workflow limits reports to two figures.
• The system cannot independently manage repository-level code; appropriate files are supplied at each phase.
• Some generated reports contain hallucinations, claims about experiments that did not occur. A gpt-4o example invents training settings and hyperparameter optimization, showing that successful execution and document compilation do not ensure faithful reporting.

5.2. Common failure modes

Observed failures include repeated literature-summary commands until the step budget is exhausted, retrieved text or printed output exceeding model token limits, and experiments remaining at 0% accuracy. mle-solver also disproportionately edits line 0; the authors observed that full replacement more often produced successful execution.

• Generated exit() calls terminated the entire process and required manual detection and removal.
• Generated code used subprocess.run() to execute host-system commands. The authors report no observed problematic outcome but identify a need for safeguards.
• paper-solver sometimes required up to 100 search attempts before retrieving papers; the authors subsequently imposed a limit of five attempts.

5.3. Ethical considerations

The authors identify possible harms from making misleading or low-quality research easier to produce, increasing pressure on peer review, and propagating dataset or model biases. They recommend transparent disclosure of artificial intelligence involvement, accountability, and governance, and discuss potential misuse in harmful technology development and biased scientific analysis.

6. Discussion

The discussion presents a continuous research-assistance workflow with adjustable human interaction and computing budgets. The authors interpret the generated artifacts and co-pilot ratings as evidence of utility, while acknowledging that the human evaluations provide only a snapshot of researchers’ experience.

• A proposed longitudinal study would compare researchers’ outcomes with and without Agent Laboratory; the present study does not measure that long-term productivity effect.
• Automatic workflow optimization and agent generation are proposed future directions, rather than evaluated components of the framework.

Appendix

• A. Agent Laboratory configuration, including A.1. Hyperparameters and A.2. Hardware, reports five paper summaries, a 120s Data Preparation timeout, three mle-solver steps, a pool of two top programs, a 600s experiment timeout, five paper-solver steps, and three reviewers during Paper Refinement. All experiments are stated to use a 2023 MacBook Pro with an Apple M3 Max processor and 36 GB of memory. Table 1 lists two code repair attempts, whereas Section 3.2 states three repair tries, leaving a configuration discrepancy relevant to reproduction.
• B. Prompts documents the templates used by the research agents. B.1. Base Inference Prompt combines role, phase instructions, available commands, history, research objective, feedback, and notes; a completion reminder is appended when 70% of the step budget has been reached. B.2. Context Prompts specifies how literature, plans, dataset code, experimental code, results, interpretations, previous reports, and reviewer feedback pass between stages. B.3. Agent Phase Descriptions and B.3.1. PhD Student phase describe literature collection, planning, data preparation, interpretation, and refinement. B.4. Machine Learning Engineer Phase Descriptions assigns data-preparation coding, while B.5. Postdoc Phase Descriptions assigns planning and interpretation through dialogue. These templates favor simple experiments and accurate transmission of observed metrics. B.6. Agent Command Description defines the interfaces in B.6.1. PhD Student Command Description, B.6.2. ML Engineer Agent Command Description, and B.6.3. Postdoc Agent Command Description. The documented commands include SUMMARY, FULL_TEXT, ADD_PAPER, DIALOGUE, SUBMIT_CODE, python, SEARCH_HF, PLAN, and INTERPRETATION, with one command per inference turn. The data-preparation templates require an external HuggingFace dataset. B.7. Agent Role Description contains B.7.1. PhD Student Role Description, B.7.2. Machine Learning Engineer Role Description, B.7.3. Professor Agent, and B.7.4. Postdoc Agent Role Description. Each assigns an academic or engineering persona through a prompt; the descriptions do not establish independently trained specialist models. B.8. mle-solver Prompts and B.8.1. Tools document whole-program replacement, inclusive line-range editing, 0-to-1 scoring, code repair, error history, and reflective feedback after execution. The templates request Figure_1.png and Figure_2.png and supply execution output for subsequent improvement. These procedures modify programs during inference; the paper reports no corresponding update to the language model’s parameters. B.9. paper-solver Prompts documents report replacement and editing, arXiv search, and generation of an eight-section report of approximately 8 pages or 4000 words. B.9.1. Per section tips, adapted from Lu et al. (2024b), directs the system to report executed, logged results and include comparisons or ablations, experiments that isolate component effects, when available. B.9.2. paper-solver Reviewer prompt reproduces the review rubric from Lu et al. (2024b), including four-point criterion scores, a ten-point overall score, confidence, and an Accept or Reject decision.
• C. Survey questions provides the instruments for C.1. Grading Research Report Autonomous Mode Preselected Topics, illustrated by Grading Research Report #1; C.2. Co-Pilot Grading Research Report Preselected Topics, using Co-Pilot Grading Research Report [Pre- Selected]; and C.3. Co-Pilot Grading Research Report Custom Topics, using Co-Pilot Grading Research Report. The forms distinguish five-point assessments of usefulness and experience from conference-style paper judgments. Their usefulness questions frame the outputs as a foundation for further research and explicitly discourage comparison with research performed by a human alone.

  Brief Thoughts

The human evaluation provides a useful check on the framework’s automated selection criteria: automated overall ratings exceed human ratings by 2.3 points. The competition results supply separate evidence for experimental programming, using objective development-set selection on a restricted task subset. The 0.58-point external co-pilot gain over the pooled autonomous average accompanies almost unchanged significance and contribution scores. It supports improved assessed paper quality under these conditions, with limited evidence of stronger scientific contributions. The inference costs quantify inexpensive runs in the reported configuration. Fabricated experimental details and the absence of a longitudinal productivity comparison leave the effort required to verify and extend those outputs unmeasured.