Gorio Tech Blog search

Paper2Code: Automating Code Generation from Scientific Papers in Machine Learning | Summary

한국어로 읽기

|
Contents

Version note. The paper is classified in 2025-W17 by its original release date of 24 April 2025. This retrospective article examines arXiv:2504.17192v5, dated 27 February 2026 and published at ICLR 2026. Its results and appendices should not be assumed to have appeared in the original version.

Minju Seo, Jinheon Baek, Seongyun Lee, and Sung Ju Hwang study the generation of multi-file machine-learning repositories from research papers. Their framework is called PaperCoder. Paper, code.

Source and license. This article summarizes Paper2Code: Automating Code Generation from Scientific Papers in Machine Learning, arXiv:2504.17192v5 by Minju Seo, Jinheon Baek, Seongyun Lee, and Sung Ju Hwang. The arXiv record lists the paper under CC BY 4.0. The images below are crops from the cited version's PDF: Figure 2 (PDF p. 4), Table 1 (PDF p. 7), Table 2 (PDF p. 7), and Figure 5 (PDF p. 9).

Read this article in Korean

Summary

  • PaperCoder generates a repository in three stages: planning the architecture and execution order, analyzing each planned file, and coding files sequentially with their dependencies available. [3–5]
  • Paper2CodeBench contains 90 papers from ICLR, ICML, and NeurIPS 2024, with 30 from each venue. The study uses reference-based and reference-free model grading, as well as a smaller human evaluation. [5–7]
  • On the ICML subset, the full pipeline’s reference-based score is 3.72 out of 5, compared with 3.28 for one-shot generation from the full paper. These are the study’s model-graded scores, not a guarantee that a repository reproduces its paper. [7–8]
  • The authors report partial or better reproduction in four of five repository case studies. Figure 5 plots method coverage at 86%, while the adjacent prose states 80%; the source contains a discrepancy. [9–10]

1. Introduction

Research papers often omit an implementation or leave details needed to run it unstated. The authors report an average code-availability rate of 19.5% among accepted ICLR, ICML, and NeurIPS 2024 papers. Their task is to produce a coordinated repository from a paper without starting from the authors’ code or a partial implementation. [1–2]

This task demands more than generating an isolated function. A repository must assign responsibilities to files, connect those files, and express the method and experiments described in the paper. The work evaluates both generated structure and whether selected implementations can be executed or reproduce results. [2–3, 9–10]

The paper places its method alongside language models for code, multi-agent repository generation, and tools for scientific research. Its input is a full paper rather than an existing repository to edit. It distinguishes its proposed generation workflow from PaperBench, an evaluation setting for paper reproduction. [2–3]

3. Method

3.1 Repository-Level Code Generation from Machine Learning Papers

The authors formulate the input as a machine-learning paper and the output as a repository whose files collectively implement the described work. Because files have dependencies, an effective order matters: a later file may rely on components generated earlier. The paper targets this coordination through intermediate artifacts rather than a single full-paper coding request. [3]

3.2 PaperCoder: LLM-Powered Multi-Agent Framework for Paper-to-Code

PaperCoder divides generation into planning, analysis, and coding. Specialized language-model agents handle these stages and pass their outputs forward: planning agents derive the repository design, an analysis agent specifies each planned file, and a coding agent generates files in dependency order. The sequence is intended to make file roles and dependencies explicit before source files are written; this organization does not itself verify the resulting implementation. [3–5]

3.2.1 Planning

Planning produces an overall plan, an architecture design, an execution-logic design, and a configuration. Architecture includes a file list and class and sequence diagrams. The logic design describes how components interact and establishes a dependency-aware implementation order. [3–4]

3.2.2 Analysis

Analysis considers planned files individually. It translates the paper and the plan into more detailed specifications, including functions, inputs and outputs, dependencies, and algorithmic constraints. These specifications are intermediate guidance, so an omission in the paper or an incorrect interpretation can still propagate to code. [5]

3.2.3 Coding

The coding stage generates files one at a time in the planned order. It uses the planning and analysis artifacts together with relevant files generated earlier. The procedure aims to keep later files consistent with their dependencies, but the structure alone does not establish that the resulting program runs correctly. [5]

PaperCoder decomposes repository generation into planning, file-level analysis, and sequential coding, in contrast to the one-shot example.

PaperCoder planning, analysis, and coding stages compared with one-shot generation.
PaperCoder workflow and one-shot comparison.

4. Experiment

4.1 Experimental Setup

Paper2CodeBench contains 90 papers, equally distributed across ICLR, ICML, and NeurIPS 2024. The authors also use 21 papers for human evaluation and 20 ICML 2024 papers in PaperBench Code-Dev. Comparisons include ChatDev, MetaGPT, abstract-only generation, full-paper one-shot generation, and PaperCoder ablations. Unless otherwise stated, the generation and evaluation backbone is o3-mini-high. [5–7]

The evaluation includes reference-based grading where author code is available and reference-free grading without such code. Model-based repository scores use a five-point scale. The authors also compare evaluation with human judgments and paper-author judgments. These procedures assess generated repositories through their chosen criteria and do not by themselves prove full scientific reproduction. [5–8]

4.2 Experimental Results and Analysis

Table 1 reports reference-based scores of 3.68, 3.72, and 3.83 for PaperCoder on the ICLR, ICML, and NeurIPS subsets. Full-paper one-shot generation receives 3.08, 3.28, and 3.22. The reference-free scores are 4.73, 4.73, and 4.77 for PaperCoder, versus 4.15, 4.30, and 4.08 for one-shot generation. The authors report no statistically significant difference in a reference-free comparison with the author repositories, but explicitly describe that oracle setting as non-comparable; reference-based oracle scores are unavailable. [7]

PaperCoder scores higher than the listed generation baselines across the three conference subsets; the oracle represents author repositories.

Reference-based and reference-free repository scores and repository statistics across three conferences.
Paper2CodeBench results.

The reported Pearson correlation between reference-based and reference-free scores is 0.79. Correlations with human judgments are 0.74 and 0.71 for GPT-4o, and 0.78 and 0.73 for o3-mini-high, respectively for reference-based and reference-free evaluation. The paper reports an inter-annotator Cohen’s kappa of 0.79. Correlation supports agreement on this evaluation; it does not remove the limits of model grading. [7–8]

In the reported human comparison, 92% of judges agreed that the top-ranked PaperCoder repository would make reproduction easier than starting from scratch. The paper reports that 22 of 25 annotator choices preferred PaperCoder’s repository, summarized in the introduction as 88% ranked best against the baselines. These figures concern the study’s particular sample and questions, rather than a measured success rate across all generated repositories. [2, 7–9]

Human and model evaluations are compared as scores and rankings in the reported samples.

Human and model-based evaluation scores and rankings for generated repositories.
Human and model evaluation.

For PaperBench Code-Dev, Table 3 reports replication scores of 45.14 ± 0.3 with o3-mini-high and 51.14 ± 1.4 with Claude 3.5 Sonnet. BasicAgent scores 5.1 ± 0.8 and 35.4 ± 0.8, while IterativeAgent scores 16.4 ± 1.4 and 27.5 ± 1.6 with those respective backbones. [8]

Table 6 compares pipeline components on the ICML subset. The complete sequence reaches 3.72 reference-based and 4.73 reference-free, compared with 3.28 and 4.30 for the baseline. Architecture design alone lowers the scores before logic design is added; the authors attribute the recovery to specifying file order and dependencies. This is an ablation within the reported setup, not proof that each component helps every paper. [8]

Figure 5 labels component coverage as 56% for data processing, 86% for method, and 79% for evaluation. The surrounding prose instead says 80% for method coverage. The source does not reconcile these values, so the discrepancy remains open. The authors identify underspecified data formats, preprocessing, and loading instructions as a source of data-processing errors. [9]

The chart shows 86% method coverage, whereas adjacent prose reports 80%; the paper does not resolve the discrepancy.

Bar chart showing helpfulness 92%, data processing 56%, method 86%, and evaluation 79%.
Fine-grained analysis; the method value differs from the prose.

4.3 Additional Analysis on Reproduction from Implemented Code Repository

For five manually executed repositories, the authors report modifying an average of 0.81% of code lines to obtain successful runs. The sample is small and involves manual debugging, data and hyperparameter adjustments; the percentage should not be generalized to every generated repository. [9–10, 18]

In a reproduction case study of the same five repositories, four reproduce results at least partially and one fails because of loss-function design. On a ten-paper PaperBench reproduction sample, Table 8 reports 28.46 for PaperCoder, 11.22 for IterativeAgent, and 2.60 for BasicAgent. The authors note that hardware constraints led to hyperparameter changes and, in rare cases, subsampling. [9–10, 18]

5. Conclusion

PaperCoder’s central proposal is to make repository generation a sequence of explicit design and implementation decisions. The reported comparisons suggest that the staged pipeline improves scores in the evaluated settings, and the case studies test a narrower form of reproduction. They leave open how often the generated code would run and reproduce results across other papers. [9–10]

Ethics Statement

The authors note that a paper may intentionally withhold code for security reasons, including work on jailbreaks or exploitation. A system that reconstructs code from text could nevertheless reproduce sensitive implementations. They propose safeguards such as harmful-content filters, protective prompting, and secure execution environments for production use; the paper does not establish that those measures eliminate the risk. [10]

Appendix

The appendices document experimental design and additional analyses, limitations, prompts, and examples of planning artifacts. Appendix C identifies visual inputs, environment setup, and complete executability or reproducibility as continuing challenges. A cross-family judge check uses 30 papers, and the authors note that indirect model exposure to benchmark papers cannot be fully ruled out. [18–21, 28–59]

Additional Experimental Results and Analysis

In a 30-paper subset using o3-mini-high as the backbone, removing the methodology section reduced the average reference-based score from 4.26 to 3.75. The authors interpret this as evidence that detailed method descriptions help generate faithful code; it is an ablation on that subset, not an estimate for every paper. [20, Table 12]

For the benchmark’s 90 papers, the authors use o4-mini-high to assign categories. Average reference-based scores range from 3.38 to 4.21: theory and interpretability/explainability score 4.21 and 3.97, while reinforcement learning/control and dataset-focused papers each score 3.38. Category assignment and model grading limit how broadly these differences can be interpreted. [21]

The recorded execution errors most often concern missing dependencies, imports, or missing modules. In another 30-paper experiment, prompts for inferring and repairing environment requirements were followed by no observed dependency-related failures; that result does not establish full executability. The authors also report that o4-mini-high and GPT-5 repaired the issues in their reported repair study, mostly within a small number of iterations, with errors categorized by GPT-5.1. These are observations from the evaluated cases, not a guarantee that generated code can always be repaired. [20, Table 13; 22, Table 14]

For 30 randomly selected papers, the authors report a Spearman correlation of 0.73 between Gemini-Flash 2.5 and o3-mini-high evaluations. They state that the primary models’ October 2023 knowledge cutoff predates the 2024 benchmark papers, while acknowledging that indirect exposure cannot be excluded. Cross-family agreement and the date argument reduce some concerns but do not prove that model-based grading is unbiased or that contamination was absent. [21]

Brief Thoughts

The separation between designing a file graph and implementing files makes the failure modes easier to inspect: a wrong plan, an incomplete per-file specification, and broken generated code can be examined separately. The main evidence still relies heavily on LLM grading, and the execution and reproduction studies use small samples.