Dynamic Scaling of Unit Tests for Code Reward Modeling | Summary

This article explains the key points of Dynamic Scaling of Unit Tests for Code Reward Modeling.

• 2025-01-02 (arXiv)
• Ma, Zeyao, Zhang, Xiaokang, Zhang, Jing, Yu, Jifan, Luo, Sijia, Tang, Jie.
• School of Information, Renmin University of China, Tsinghua University, Key Laboratory of Data Engineering and Knowledge Engineering, Beijing, China
• Paper
• Github
• Project Page

Read this article in Korean

Summary

• Generated code can include correct solutions that automated selection misses because generated unit tests, checks pairing inputs with expected outputs, are unreliable. On HumanEval Plus, a Python programming benchmark with expanded correctness tests, selecting among 100 candidate solutions using CodeRM-8B, an 8B-parameter test generator, raises Llama3-8B performance from 53.58% with random selection to 72.01%, an 18.43 percentage-point improvement.
• CodeRM-8B is trained from Llama3.1-8B through supervised fine-tuning, learning from problem-and-solution inputs paired with validated tests. Its synthetic training data pipeline repairs tests using execution feedback and filters tests that accept incorrect solutions; combining many generated tests produces solution-selection performance broadly comparable to Llama3.1-70B.
• Increasing test counts generally improves solution selection, with larger benefits on harder problems in an analysis restricted to questions with at least one correct sampled candidate. Dynamic allocation, assigning different test budgets according to predicted problem difficulty, adds up to approximately 0.5 percentage points on MBPP Plus, a programming benchmark with expanded tests, compared with equal allocation at fixed budgets. The allocation rule is adapted from solution sampling and provides modest gains on HumanEval Plus.

1 Introduction

Dynamic Scaling of Unit Tests for Code Reward Modeling studies whether spending more inference computation on verification improves code selection. A large language model (LLM), a model used here to generate code or tests, can produce multiple candidate solutions, yet unreliable tests can assign misleading reward signals, execution outcomes used to rank those candidates.

• The paper separates the policy model, which generates solutions, from the reward model, which generates executable tests.
• Its contributions combine a test-count scaling study, a trained 8B-parameter test generator, and difficulty-dependent allocation of test generation.
• The project URL provided in the PDF is https://code-reward-model.github.io.

2 Pioneer Experiment

The Pioneer Experiment examines how test quantity affects the quality of execution-based solution ranking. It varies solution counts, test generators, and policy models before analyzing whether the observed benefits depend on problem difficulty.

2.1 Methodology

The method follows best-of-N selection, generating N candidate solutions and returning the candidate that passes the largest number of generated test suites, each a collection of input-output checks. In the paper’s majority-voting framework, each suite supplies one binary vote: a solution passes that suite only when every test case in it succeeds.

• Equation (1) selects C_opt = argmax over C_i of the sum of r_i,j for j = 1,…,M, where r_i,j is 1 when candidate C_i passes every case in suite T_j and 0 otherwise.
• For HumanEval Plus, the exploratory study generates N = 200 solutions and M = 100 test suites per question, then evaluates the selected solution against the benchmark’s ground truth tests.
• Smaller solution and test counts are estimated through bootstrap resampling, sampling repeatedly from the generated pools; 100 bootstrap samples provide means and confidence intervals.

2.2 Observations

More generated tests generally improve selection, especially for weaker policy models and harder problems. With GPT-4o generating tests, the paper reports gains of about 11 percentage points for Llama3-8B and 5 percentage points for Llama3-70B as test counts increase.

• Figure 1 also shows that increasing candidate counts can reduce performance for Llama3-70B with Llama3.1-70B verification, so additional sampling does not guarantee better selection.
• Gemma-2-27B-it begins below Llama3.1-70B with one test but becomes comparable at 100 tests; greater diversity or coverage is proposed as an explanation rather than directly measured.
• The difficulty analysis removes questions with no correct sampled solution and divides the remainder into five equal groups by observed solution pass rate. Figure 3 labels Quintile 5 as hardest, although its caption mistakenly calls Quintile 2 hardest.

The curves generally rise as more test suites vote on candidate correctness, but the Llama3-70B policy with Llama3.1-70B verification also shows that larger candidate pools can hurt selection. Shaded bootstrap intervals show uncertainty around the estimated performance.

With 200 candidates fixed, Gemma-2-27B-it approaches Llama3.1-70B as the test count reaches 100 despite a weaker starting point. GPT-4o remains higher, showing that scaling helps without eliminating differences in generator quality.

The easiest groups already approach perfect selection, while the hardest group has substantially more room to benefit from additional tests and candidates. The panels identify Quintile 5 as hardest; the caption's reference to Quintile 2 as hardest is inconsistent with those labels. Questions without any correct sampled solution were excluded.

3 Towards Efficient and High-Quality Unit Test Scaling

The proposed system addresses test generation cost through two components: a smaller trained generator and adaptive test budgets. CodeRM-8B supplies tests, while a learned difficulty estimator guides a greedy allocation procedure that assigns computation incrementally across questions.

3.1 Unit Test Generator

Training data comes from CodeFeedback-Filtered-Instruction, a collection of code instruction queries, and the training set of TACO, a dataset of algorithmic programming problems. The pipeline uses Llama3.1-70B to filter questions unsuitable for ordinary unit testing, prepares solutions in functional form, and uses Llama3.1-70B to generate test suites and repair failing suites using Python execution feedback.

• A suite is initially accepted when the reference solution passes it. Additional quality control removes tests that fail to reject incorrect solutions generated by a less capable model.
• Supervised fine-tuning uses the programming problem and code solution as the input and a retained test suite as the target output, producing CodeRM-8B from Llama3.1-8B.
• The source datasets are linked at https://huggingface.co/datasets/m-a-p/CodeFeedback-Filtered-Instruction and https://huggingface.co/datasets/BAAI/TACO.

3.2 Dynamic Unit Test Scaling

Difficulty is represented by the policy model’s estimated solution pass rate. A two-layer feedforward probe, a small predictor trained on intermediate model representations, predicts this rate using hidden states, internal representations already computed during code generation.

• Training targets are mean success rates of repeatedly sampled solutions checked against previously collected valid tests.
• Following Damani et al. (2024), the allocation rule uses q(x,b) = 1 − (1 − λ)^b, the probability of at least one correct answer in b attempts when the pass probability is λ, to guide greedy budget assignment.
• This probability describes solution sampling rather than the benefit of additional verification tests; the paper explicitly identifies that mismatch as a limitation.

Reference-solution execution checks whether a generated suite accepts valid code, and a separate filtering stage checks whether it rejects incorrect code. These two criteria feed generator training before difficulty prediction changes the number of tests generated for each problem.

4 Experiments

The experiments evaluate solution selection, the quality of generated verification signals, and the effects of synthetic data construction. They also compare difficulty-based allocation with equal allocation under fixed test-generation budgets.

4.1 Experimental Setup

Evaluation uses HumanEval Plus, MBPP Plus, and 168 problems with functional solutions from LiveCodeBench, a benchmark that continually collects new programming problems. The LiveCodeBench subset covers 2024-01 to 2024-09, and the reported metric is Pass@1, the proportion of questions solved by the single returned solution after selection.

• The main comparison generates 100 candidate solutions per question using Llama3-8B-Instruct, Llama3-70B-Instruct, GPT-3.5-turbo, or GPT-4o-mini.
• Baselines include random selection; scalar scoring with ArmoRM-Llama3-8B-v0.1; MBR-Exec, which ranks candidates through minimum Bayes risk decoding based on execution results; CodeT, which uses agreement among solutions and generated tests; and MPSC, multi-perspective self-consistency, which combines solution, specification, and test consistency.
• Execution-based baselines use Llama3.1-70B for test generation and receive the same stated budget of 100 inferences. This matches generation-call counts; the reported comparisons do not establish equal runtime or token cost across generators and output formats.

4.2 Main Results

CodeRM-8B improves all four policy models on all three benchmarks relative to random selection. On HumanEval Plus, scores increase from 53.58% to 72.01% for Llama3-8B, 73.74% to 78.69% for Llama3-70B, and 82.96% to 86.38% for GPT-4o-mini.

• On MBPP Plus, CodeRM-8B reaches 66.71%, 72.44%, 75.96%, and 75.20% for Llama3-8B, Llama3-70B, GPT-3.5-turbo, and GPT-4o-mini, respectively; on LiveCodeBench, it reaches 15.21%, 27.73%, 21.76%, and 39.20%.
• Comparison with Llama3.1-70B is mixed: CodeRM-8B wins all four MBPP Plus comparisons but trails for Llama3-8B and GPT-3.5 on HumanEval Plus and for Llama3-70B and GPT-3.5 on LiveCodeBench.
• Figure 6 shows small improvements from predicted-difficulty allocation over equal allocation at fixed budgets, with gains up to approximately 0.5 percentage points on MBPP Plus. Gold pass rates, observed rather than predicted success rates, provide a comparison using measured difficulty.

CodeRM-8B improves all 12 policy–benchmark combinations over random selection. It exceeds the Llama3.1-70B verifier in every MBPP Plus column, while results on HumanEval Plus and LiveCodeBench are mixed, supporting broad comparability rather than uniform superiority.

Fine-tuning raises the 8B generator substantially above its original version as test counts increase. CodeRM-8B approaches Llama3.1-70B on HumanEval Plus and surpasses it at larger test counts on MBPP Plus.

Predicted-pass-rate allocation improves slightly over equal allocation at the same budget, with a clearer separation on MBPP Plus. The gap to allocation using observed pass rates shows the effect of replacing predictions with measured difficulty; even that comparison yields modest gains on HumanEval Plus.

4.3 Quality of Generated Unit Tests

The paper treats tests as classifiers of solution correctness and reports accuracy and F1, the harmonic mean of precision, the fraction of accepted solutions that are correct, and recall, the fraction of correct solutions accepted. It also reports False Acceptance Rate (FAR) for accepting incorrect solutions and False Rejection Rate (FRR) for rejecting correct solutions, although the appendix’s formulas use denominators that differ from these verbal descriptions.

• On HumanEval Plus with Llama3.1-8B as policy, individual CodeRM-8B tests reach 69.64% accuracy and 63.63 F1, below Llama3.1-70B’s 73.65% and 70.15.
• Combining 100 tests reverses the comparison: CodeRM-8B reaches 80.46% accuracy and 81.27 F1, versus 78.30% and 78.76 for Llama3.1-70B.
• This reversal supports evaluating the combined verifier separately from individual tests. It is consistent with useful test diversity, but does not directly establish that diversity causes the improvement.

Individual CodeRM-8B tests have lower accuracy than Llama3.1-70B tests, 69.64% versus 73.65%, but the 100-test combination reverses the ordering to 80.46% versus 78.30%. This establishes an advantage for combined verification without directly measuring the proposed diversity mechanism.

4.4 Ablation Study of Synthetic Data

Filtering tests that accept incorrect solutions improves the trained generator beyond fine-tuning without this quality control. HumanEval Plus rises from 69.71% to 71.09%, and MBPP Plus rises from 64.96% to 66.31%.

• Relative to the zero-shot generator, the model evaluated without task-specific fine-tuning, at 66.67% and 63.27%, quality-controlled training adds 4.42 and 3.04 percentage points, compared with 3.04 and 1.69 without filtering.
• The prose’s approximately 45% and 80% relative gains correspond to increases in improvement over zero-shot, rather than relative increases in final benchmark accuracy.
• Figure 7 shows increasing performance across training sizes of 0, 5k, 10k, 30k, and 60k examples, reaching approximately 71.1% on HumanEval Plus and 66.3% on MBPP Plus.

Quality control adds 1.38 percentage points on HumanEval Plus and 1.35 on MBPP Plus beyond training without filtering. The approximately 45% and 80% figures discussed in the text correspond to increases in gains over zero-shot rather than final accuracy.

Both benchmark curves improve across the evaluated training sizes through 60k examples. These results support the value of more retained synthetic training data within the tested range.

5 Related Work

Related work distinguishes execution-based reranking, which checks candidate behavior using tests, from learned rerankers that score candidates without executing them. The unit test generation discussion covers search-based methods that explore candidate tests, constraint-based methods that derive tests from program conditions, probability-based methods, and LLM-based approaches. This paper contributes a scaling analysis and a generator trained on synthetic data within the execution-based setting.

6 Conclusion

The conclusion reports a positive relationship between generated test count and code reward quality, with larger benefits on challenging questions. CodeRM-8B achieves broadly comparable solution-selection performance to a larger test generator, while dynamic allocation provides an additional, smaller improvement under fixed budgets.

7 Limitations

The authors acknowledge that an allocation rule developed for generating solutions may be poorly matched to generating verifiers. Even allocation using actual pass rates provides relatively modest gains on HumanEval Plus.

• The proposed diversity and coverage explanation remains an open question: neither is directly quantified as test counts increase.
• The difficulty-stratified exploratory results exclude problems with no correct sampled candidate, so they characterize selection among potentially solvable pools.
• The preprocessing excludes several difficult-to-test behaviors, including randomness and external resource dependence; the evidence therefore concerns a narrower setting than general software testing.

Appendix

• A License states that CodeFeedback-Filtered-Instruction and TACO use Apache 2.0 and announces plans to release CodeRM-8B and its synthetic training data under Apache 2.0 upon acceptance. This PDF records a release intention rather than evidence that release has occurred.
• B More Results of Unit Test Scaling provides Figure 8, a finer difficulty analysis across policy and reward model combinations. The heatmaps generally support larger benefits from additional tests on more challenging questions, with variation across model pairings and difficulty bins.
• C Experiment Settings and Baselines specifies 100 candidate solutions per question, temperature 0.8 and Top P 0.95, which control sampling randomness and the cumulative probability of eligible tokens, and frequency penalty 0 and presence penalty 0, which leave those repetition penalties disabled. It reports use of 8 NVIDIA A800 graphics processing units (GPUs). MBR-Exec and CodeT each receive 100 test-generation inferences requesting 10 test cases per inference; MPSC receives 50 test cases and 50 specifications under the stated shared budget.
• D Metrics for Assessing Unit Test Quality defines accuracy, precision, recall, and F1 using true positives (TP), correct solutions accepted; true negatives (TN), incorrect solutions rejected; false positives (FP), incorrect solutions accepted; and false negatives (FN), correct solutions rejected. Its formulas are \(\mathrm{FAR}=\frac{FP}{FP+TP}\) and \(\mathrm{FRR}=\frac{FN}{FN+TN}\), which condition on accepted and rejected predictions, respectively, despite prose describing probabilities conditional on actual solution correctness. For multiple tests, candidates with the largest acceptance count are classified as correct and the remaining candidates as incorrect.
• E Prompt for Data Synthetic Pipeline supplies templates for question filtering, test repair, function interface extraction, TACO solution generation, code and test generation, and question and solution reformatting. Filtering covers randomness, time dependence, external resources, concurrency, hardware, user interfaces, side effects, cryptography, adaptive learning, and complex mathematical or simulation behavior. Figure 9 illustrates a problem-and-solution input paired with output using unittest, Python’s unit-testing library; its purported positive-return and zero-return cases evaluate to −45 and −36, respectively, despite their labels.

Brief Thoughts

The strongest evidence concerns combined verification: CodeRM-8B’s individual tests trail the 70B generator, yet its 100-test combination improves correctness classification and achieves broadly comparable solution selection. The dynamic allocator has weaker support because its objective models solution sampling rather than verification quality, and its measured gains are small. Assessing practical efficiency requires direct runtime and token-cost measurements beyond parameter counts and matched inference counts.