AutoCodeBench: Large Language Models are Automatic Code Benchmark Generators | Summary

This article explains the key points of AutoCodeBench: Large Language Models are Automatic Code Benchmark Generators.

• 2025-08-12 (arXiv)
• Chou, Jason, Liu, Ao, Deng, Yuchi, Zeng, Zhiying, Zhang, Tao, Zhu, Haotian, Cai, Jianwei, Mao, Yue, Zhang, Chenchen, Tan, Lingyun, et al.
• Hunyuan Team, Tencent
• Paper

Read this article in Korean

Summary

• AutoCodeBench addresses the cost of constructing challenging code generation tests across programming languages without extensive manual annotation. Among more than 30 evaluated large language models (LLMs), systems that generate text and code, Claude Opus 4 achieves the highest average Pass@1, the percentage of problems solved by one generated solution, at 52.4% on AutoCodeBench versus 64.5% on its model-selected AutoCodeBench-Lite subset, using reasoning mode, which produces intermediate reasoning before the answer.
• AutoCodeGen constructs solutions and test inputs first, executes them in a multilingual sandbox, an isolated code execution environment, and then writes problem descriptions from the solutions and tests. Difficulty filtering, model-based consistency checks, and category sampling produce 3,920 problems across 20 programming languages, with 37,777 test cases.
• Models score lower on multi-logic problems, tasks requiring multiple distinct functions or classes, while execution feedback improves all three models tested for iterative refinement. Six professional annotators report 87.6% validity across six languages under an assessment that permits incomplete boundary-case coverage, leaving test completeness and validity in the other 14 languages unverified.

1 Introduction

AutoCodeBench: Large Language Models are Automatic Code Benchmark Generators identifies costly manual construction, narrow programming language coverage, and limited combinations of difficulty and practical task diversity as shortcomings in existing evaluations. AutoCodeGen uses solution-first synthesis and executable tests; AutoCodeBench and Lite evaluate solutions generated from problem instructions, while AutoCodeBench-Complete evaluates code completion from supplied examples. The paper also releases a multilingual execution service and examines benchmark quality and possible generation bias.

AutoCodeBench combines multilingual coverage, tasks with multiple core functionalities, automatic construction, and balanced language counts in this comparison. Its 3,920 records and 14 displayed categories exceed the listed comparators, although difficulty ratings and category assignments depend on models.

2 AutoCodeBench: A Challenging, Practical, and Diverse Multilingual Benchmark

AutoCodeBench combines approximately balanced coverage of 20 programming languages with repository-derived problems filtered for difficulty. This section presents dataset statistics and the automated construction process, including the selection of Lite using the evaluated models’ observed successes.

2.1 Data Overview

AutoCodeBench contains 3,920 problems and 37,777 test cases; AutoCodeBench-Lite contains 1,586 problems and 15,341 test cases. Difficulty is defined by ten DeepSeek-Coder-V2-Lite attempts: zero successes means hard, 1–5 means medium, and more than five means easy. The resulting easy/medium/hard counts are 646/846/2,428 for the full benchmark and 263/421/902 for Lite.

• The languages are Python, C++, Java, JavaScript (JS), Go, Shell, C#, Dart, Elixir, Julia, Kotlin, Perl, PHP, Racket, R, Ruby, Rust, Scala, Swift, and TypeScript (TS). Full-benchmark counts range from 184 to 200 per language; Lite counts range from 57 to 106.
• Claude Sonnet 4 proposes 20 language-independent task categories, and DeepSeek-V3-0324 assigns labels. Categories representing less than 2% are merged into “Other,” leaving 14 displayed categories.
• Table 2 reports average problem/solution lengths of 498.2/487.5 for the full benchmark and 517.2/469.3 for Lite. Section 2.1 calls problem length characters, whereas Table 1 specifies the Qwen2.5-32B-Instruct tokenizer, which divides text into model input tokens; the PDF does not reconcile these descriptions.

The full benchmark contains 2,428 hard problems out of 3,920, while Lite contains 902 out of 1,586, under the ten-attempt DeepSeek-Coder-V2-Lite definition. Lite reduces evaluation volume while retaining all 20 languages.

Language counts are approximately equal across 20 languages, while task categories have unequal proportions. Object-oriented programming, algorithms, and data structures occupy prominent category shares.

2.2 Automated Workflow

AutoCodeGen has four stages: Code Solution Generation, Test Function Generation, Programming Problem Generation, and Data Filtering. Executing generated reference solutions supplies expected test outputs, avoiding direct model prediction of those outputs. This establishes execution consistency for the supplied cases, while the later human assessment documents remaining specification errors.

• 2.2.1 Code Solution Generation: Stack-Edu, an educational code dataset filtered from The Stack v2, supplies function-, class-, and file-level snippets from GitHub repositories. DeepSeek-V3-0324 uses language-specific prompts with examples to produce self-contained solutions, remove nonessential logic, and add comments; the sandbox checks execution.
• 2.2.2 Test Function Generation: Public input functions contain no more than 3 basic cases, and private functions contain 7+ inputs, including edge cases, inputs at boundaries or unusual conditions. The system executes each solution with its inputs, asks DeepSeek-V3-0324 to combine the inputs and observed outputs into test functions, and executes the public and private tests again.
• 2.2.3 Programming Problem Generation: DeepSeek-V3-0324 writes descriptions under six specifications: explicit programming language, unambiguous task, exact function/class naming, input/output types and ranges, public examples, and no solution hints. Each record contains <programming problem, code solution, public test function, private test function>.
• 2.2.4 Data Filtering: Problems solved in all ten DeepSeek-Coder-V2-Lite attempts are discarded, removing 25.1% of the Python pool. DeepSeek-R1-0528 acts as LLM-as-Critic, a model checking problem–test consistency while assuming the problem is correct. Checks cover signatures, reproducibility, test objectives, precision, exception handling, unstated requirements, and coverage; category-based cyclic sampling controls diversity.
• 2.2.5 Approximate Language Translation: Python, C++, Shell, Java, JavaScript, and Go use direct construction. The other 14 languages use translated unused records followed by filtering; Table 3 lists the source–target pairs.
• 2.2.6 AutoCodeBench-Lite Construction: Problems are ordered by successful solution counts across evaluated models. Those with fewer than 2 passes are discarded, and approximately 1,500 problems are selected in ascending pass-count order, producing the reported 1,586-problem subset. Lite selection therefore depends on the evaluated model pool.

Solutions and inputs precede execution-derived outputs, complete tests, and question generation. The final sampling, criticism, and tagging stages separately address difficulty, consistency, and category diversity.

The pairs are Python to R, Elixir, Swift, Ruby, Julia, and Racket; Java to Scala, Kotlin, Dart, and C#; JavaScript to PHP and TypeScript; Shell to Perl; and C++ to Rust. They document the translated origins of 14 language subsets.

3 Experiments

The experiments examine overall and language-specific performance, multi-logic tasks, model size, additional solution sampling, and iterative correction using execution errors. A separate completion experiment evaluates base models, models before instruction-oriented post-training, with three demonstrations.

3.1 Experimental Setup

The evaluation includes more than 30 open-source and proprietary models, with open-source sizes spanning 1.5B to 1T parameters, where B and T denote billion and trillion, and uses Pass@1 as the default metric. A common output-format prompt requests one Markdown code block containing the solution without execution commands, tests, or usage examples.

• Proprietary models and specified large deployed models are called through application programming interfaces (APIs), programmatic service interfaces, without additional parameters. DeepSeek-V3-0324, DeepSeek-R1-0528, Seed-Coder-8B, and the Qwen3 series use officially recommended parameters; remaining models use greedy decoding, selecting the highest-probability next token, with temperature 0.
• Non-API models run through VLLM, the model-serving framework named in the paper. The results compare the reported deployment settings rather than a uniform decoding configuration.

3.2 Main Results

Claude Opus 4 (20250514) leads in reasoning mode with 52.4% on AutoCodeBench and 64.5% on Lite; its non-reasoning scores are 50.9% and 61.7%. Claude Sonnet 4 in reasoning mode and o3-high both score 51.1% on the full benchmark, while o3-high reaches 63.2% on Lite. No evaluated configuration exceeds 53% on the full set.

• Reasoning mode generally improves performance in the reported comparisons. Qwen3-32B rises from 31.0% to 41.7% on the full benchmark and from 32.3% to 47.6% on Lite.
• The Current Upper Bound is the union of problems solved by any evaluated model: 74.8% on the full set and 100.0% on Lite. It describes complementary observed successes rather than an implemented model-selection system or a limit on future capabilities.
• Lite retains difficult tasks with at least two observed passes. Its 100.0% union coverage follows from this selection, and its higher scores compare performance on a different test set.

The best reported full-benchmark average is 52.4%, and the union of observed model successes is 74.8%. Different models lead different language columns, so the overall leader does not dominate every language.

Claude Opus 4 reasoning reaches 64.5%, followed by o3-high at 63.2% and Grok-4 at 63.0%. The 100.0% union score reflects Lite’s selection of problems already solved by the evaluated model pool.

The ranked averages show a leader at 52.4% and substantially lower scores for many other models. Reasoning configurations often rank above their non-reasoning counterparts, although inference costs are not compared.

3.3 Performance Across Popular and Low-Resource Programming Languages

The paper compares Python, C++, Java, and C# with a designated low-resource set of Racket, Shell, Elixir, and TypeScript, languages treated as having more limited resources in this comparison. The sets are selected using the TIOBE Index, a programming language popularity index linked at https://tiobe.org.cn/tiobe-index/. Figure 3 reports average Pass@1 ranges of 50.4–53.8% for the popular set and 45.3–62.0% for the designated low-resource set across eight configurations.

• Figure 3 reports Claude Opus 4 at 62.0% with reasoning and 60.3% without it on the designated low-resource set. These plotted values do not match simple averages of the corresponding language columns in Tables 4 or 5; the PDF does not explain the discrepancy.
• The authors suggest that DeepSeek-Coder-V2-Lite removes easy popular-language tasks more effectively than easy low-resource tasks. Uneven difficulty filtering therefore complicates attributing the observed language differences to training-data coverage.

The plotted configurations span 50.4–53.8% on popular languages and 45.3–62.0% on the designated low-resource set. These values show wider separation in the latter comparison, but the PDF does not reconcile them with simple averages of the corresponding language columns in Tables 4 and 5.

3.4 Performance Across Multi-Logic Programming Problems

DeepSeek-V3-0324 identifies 1,622 multi-logic problems requiring multiple distinct functions or classes in one task. All eight displayed configurations score lower on this subset than on the full benchmark: Claude Opus 4 with reasoning falls from 52.4% to 49.0%, DeepSeek-R1-0528 from 50.2% to 45.1%, and GPT4.1 from 48.0% to 43.0%.

• The comparison documents lower scores on tasks with multiple requirements. Because the labels are model-generated and the subset is not matched to other tasks by length or difficulty, it does not isolate the effect of multiple requirements alone.

Every displayed configuration loses 3.1–5.1 percentage points on the 1,622 multi-logic problems relative to the full dataset. Claude Opus 4 reasoning retains the highest subset score at 49.0%, while DeepSeek-R1-0528 has the largest displayed drop, 5.1 points.

3.5 Performance Analysis of Scaling Laws

The parameter-scaling plots show increasing Pass@1 across the selected Qwen2.5-Coder, Qwen3, DeepSeek-Coder, and Qwen3 reasoning series. Qwen3 reasoning has the steepest plotted curve, which the authors associate with chain-of-thought reasoning, explicit intermediate reasoning steps, benefiting larger models.

• The second panel evaluates Pass@K, success when at least one of K sampled solutions passes, for DeepSeek-V3-0324, Qwen2.5-Coder-32B-Instruct, and Qwen3-8B at K = 1, 3, 5, 7, 9, and 10. All three improve with additional samples, with smaller gains at larger K.
• The plots show empirical trends for selected model families. The paper supplies neither a fitted general scaling equation nor an equal-cost comparison of inference strategies.

The left panel shows different performance trajectories as parameter counts increase across four model series. The right panel shows gains from increasing K for three models, with smaller increments at larger K; neither panel supplies a fitted general scaling equation.

3.6 Performance Analysis of Multi-Turn Refinement with Sandbox Feedback

Three refinement turns using sandbox execution errors increase DeepSeek-V3-0324 from 48.1% to 59.7%, Qwen2.5-Coder-32B-Instruct from 35.8% to 47.4%, and Qwen3-8B from 23.3% to 30.2%. The first turn gives the largest gain for each model under the tested procedure.

• DeepSeek-V3-0324 progresses through 56.8%, 58.4%, and 59.7%; Qwen2.5-Coder-32B-Instruct through 44.7%, 46.1%, and 47.4%; Qwen3-8B through 28.2%, 29.3%, and 30.2%.
• Refinement adds both execution feedback and generation attempts. The paper does not include a matched-cost comparison with independent sampling that would separate their contributions.

Execution-assisted refinement raises the three models from 48.1%, 35.8%, and 23.3% to 59.7%, 47.4%, and 30.2%, respectively. The first correction produces the largest gain for each model.

3.7 AutoCodeBench-Complete: Evaluating Base Model Capabilities

AutoCodeBench-Complete selects 1,000 problems from Lite, exactly 50 per language, and evaluates completion using 3-shot demonstrations, three examples supplied before the target problem. DeepSeek-Coder-V2-Base scores 39.0%, Qwen2.5-72B 35.9%, and Qwen2.5-Coder-32B 35.5%; Seed-Coder-8B-Base leads the listed models with 8B parameters or fewer at 31.6%.

• OpenCoder-8B-Base scores 26.1%, exceeding Qwen2.5-Coder-7B at 24.6%, although their instruction-tuned versions, models further trained to follow instructions, show the opposite ordering on AutoCodeBench and Lite.
• The authors suggest that differences in multilingual post-training data explain this reversal. The results establish the ordering, but the paper does not examine the training corpora to test that explanation.

The completion evaluation uses 50 problems per language and three demonstrations. DeepSeek-Coder-V2-Base leads the listed larger models at 39.0%, while Seed-Coder-8B-Base leads the approximately 8B group at 31.6%.

4 Further Discussion

Further Discussion examines benchmark validity and possible preferences introduced by the construction models. Human verification covers six directly generated languages, and a separate experiment tracks five models’ scores across filtering stages.

4.1 Manual Verification

Six professional annotators inspect problem statements, test functions, and DeepSeek-R1-0528 criticism for Python, C++, Java, JavaScript, Go, and Shell, assigning binary validity labels. The reported average validity is 87.6%, ranging from 83.5% for Python to 93.3% for Shell; the corresponding union of model successes is 66.9%, and Claude Opus 4 reasoning performance is 44.6%.

• Table 8 reports validity rates of 83.5%, 88.0%, 86.0%, 89.0%, 86.0%, and 93.3% in that language order. Incomplete descriptions are the most frequent issue, including omitted function/class names and unspecified edge-case outputs.
• Appendix B assumes problem statements are correct and permits tests that omit boundary conditions. The assessment checks test correctness and alignment rather than exhaustive coverage, and does not directly cover the 14 translated languages.
• The 43.0-percentage-point difference between validity and Claude Opus 4 performance compares different quantities. Table 8 does not report model performance recomputed after removing invalid records.

Reported validity averages 87.6%, compared with 66.9% for the union of model successes and 44.6% for Claude Opus 4 reasoning across six languages. The differences compare annotation validity with model performance; the table does not report scores recomputed on a cleaned benchmark.

4.2 Hypotheses on Model Bias in the Generation Process

The authors hypothesize that DeepSeek-based generation and criticism may favor the DeepSeek family, while difficulty filtering with DeepSeek-Coder-V2-Lite may counteract that preference. They track five models over 3,600 records from Python, C++, Java, JavaScript, Go, and Shell.

• Simple-problem filtering reduces scores by 18.3–22.8 percentage points. Critic filtering then raises them by 4.9–7.5 points: DeepSeek-V3-0324 gains 5.9, DeepSeek-R1-0528 7.5, o3 6.8, Gemini2.5 Pro 7.0, and Qwen2.5-Coder-32B-Instruct 4.9.
• Gemini2.5 Pro remains first, DeepSeek-R1-0528 second, and Qwen2.5-Coder-32B-Instruct fifth across all three stages; o3 overtakes DeepSeek-V3-0324 after difficulty filtering. Critic filtering therefore does not benefit the generator model most.
• The authors judge the remaining favorable bias minimal. Without an alternative-generator control, the stage comparisons do not establish its causal magnitude.

Difficulty filtering lowers all five scores, and critic filtering raises them without restoring their initial levels. Gemini2.5 Pro remains first throughout; the mixed changes document filtering effects without isolating generator-family bias.

5 Related Work

Related Work places AutoCodeBench alongside code generation benchmarks and automatic code data synthesis. The paper’s distinction is applying interaction between models and executable code to benchmark construction, whereas much earlier synthesis work produces training data.

5.1 Code Generation Benchmarks

HumanEval and MBPP are benchmarks of short Python programming tasks; competition-oriented benchmarks increase algorithmic difficulty, while McEval and FullStackBench broaden language and application coverage. The paper describes McEval as covering 40 languages and FullStackBench as covering 16, and positions AutoCodeBench around automatic construction and approximately balanced language counts.

• LiveCodeBench continually sources competition problems to reduce data contamination, overlap between evaluation problems and model training data. The discussion also includes OJBench with 232 contest problems and APPS with 10,000 problems.
• ArtifactsBench evaluates visual code generation through multimodal assessment, evaluation using visual as well as textual information, while CodeCriticBench evaluates code criticism. AutoCodeBench evaluates executable solutions using generated test functions.

5.2 Code Data Synthesis

Evol-Instruct increases task complexity through prompted evolution, OSS-Instruct derives tasks from open-source snippets, and KodCode generates Python questions, solutions, and tests with self-verification. Inverse-Instruct derives instructions from existing code, while SelfCodeAlign uses the same base model for generation and validation.

• AutoCodeGen adds execution-derived expected outputs and solution-first problem generation to this line of synthesis work. The authors propose generating training data with the workflow, but report no training experiment evaluating that use.

6 Conclusion

The paper introduces AutoCodeGen, AutoCodeBench with 3,920 problems, its Lite and completion variants, and a multilingual execution service. The evaluations show lower performance on multi-logic tasks and gains from execution-assisted refinement under the tested settings. Resources are linked at https://autocodebench.github.io/; the reviewed version is https://arxiv.org/abs/2508.09101v1.

Appendix

• A Data Category and Language Distribution Statistics groups 20 proposed labels into Core Programming Concepts, Data and Algorithms, Application Domains, and Advanced Topics and Tooling. The displayed comparison shows broad category coverage but uneven language counts for FullStackBench, and balanced language counts but concentrated categories for McEval. AutoCodeBench has 14 displayed categories and approximately balanced language counts; Lite has 13 displayed categories and less even language counts, ranging from 57 to 106.
• B Manual Verification describes model-assisted human assessment of names, types, reproducibility, logical alignment, numerical precision, and unstated requirements. Annotators see the critic’s reasoning and results, assume problem statements are correct, and permit incomplete boundary coverage. These conditions define the scope of the reported 87.6% validity estimate.
• C Multilingual Code Sandbox Service describes support for more than 30 programming languages, Docker containers for execution isolation, iptables rules for network filtering, automatic solution–test integration, and Gunicorn multiprocessing for concurrent requests. It also provides an HTTP-based programmatic interface and configurable execution environments. The main body states 20+ language support; Appendix C gives the more specific description without throughput measurements or a security evaluation.
• D Prompts for Automated Workflow identifies the solution-generation, test-generation, problem-generation, criticism, and translation prompts. The displayed solution prompt permits no more than 2 demonstration cases, whereas Section 2.2.2 permits no more than 3; the test prompt preserves supplied inputs and outputs, and the problem prompt requires exact names and public examples without solution hints. The solution and test prompt figures omit their latter portions, so the PDF alone does not provide their complete text.

Brief Thoughts

The construction pipeline provides a concrete way to obtain test outputs through execution before writing problem statements. Its human verification also documents a remaining weakness: executable tests can depend on requirements omitted from those statements. Language-dependent filtering and model-dependent Lite selection are therefore central to interpreting the reported scores. The results characterize the tested models under the reported evaluation conditions. They do not establish exhaustive test correctness, a measured absence of generation bias, repository-level engineering ability, or training benefits from AutoCodeGen data.