SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution | Summary

This article explains the key points of SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution.

• 2025-02-25 (arXiv), 39th Conference on Neural Information Processing Systems (NeurIPS 2025)
• Wei, Yuxiang, Duchenne, Olivier, Copet, Jade, Carbonneaux, Quentin, Zhang, Lingming, Fried, Daniel, Synnaeve, Gabriel, Singh, Rishabh, Wang, Sida I..
• Meta AI, University of Illinois Urbana-Champaign, Carnegie Mellon University
• Paper
• Github

Read this article in Korean

Summary

• SWE-RL addresses how to train large language models (LLMs) to fix repository bugs when running candidate repairs during training is costly or unsupported. Its Llama3-SWE-RL-70B model resolves 41.0% of SWE-bench Verified, a collection of 500 human-verified GitHub issues, compared with 36.2% for the paper’s supervised fine-tuning (SFT) baseline, trained on example responses, using the same Agentless Mini pipeline for finding files, generating repairs, testing, and selecting a final repair.
• The method applies reinforcement learning (RL), optimization using rewards for generated outputs, to 273k selected GitHub pull requests (PRs), proposed repository changes linked to issues. It rewards textual similarity between generated patches, representations of code changes, and merged developer patches, with a −1 penalty for malformed outputs. Training therefore avoids execution-based rewards.
• Training only on issue repair improves several tasks outside the training domain, particularly code execution reasoning and mathematics, while preserving the starting model’s library-use scores. The 41.0% result uses 500 candidate patches and up to 30 reproduction tests, generated tests that distinguish the reported bug from its repaired behavior, per issue. Textual similarity does not establish semantic correctness, and the paper does not provide error bars for all SWE-bench evaluations.

1 Introduction

The introduction motivates improving the underlying model’s software engineering capabilities alongside improvements to prompting and inference pipelines. Existing coding RL commonly uses executable, self-contained programming problems; repository repair introduces expensive execution and missing environments. SWE-RL instead learns from software evolution data, the record of code snapshots, changes, issues, and PRs, without requiring proprietary teacher-model outputs for its RL procedure.

The diagram separates repository-data preparation from policy optimization. Generated patches receive similarity-based rewards against merged developer patches, while malformed responses receive −1; execution is absent from this training loop.

2 SWE-RL

SWE-RL selects 273k high-quality PR seeds using heuristics that require linked issues describing bug fixes and changes involving programming files. Each seed supplies an issue description, complete contents of changed files, some relevant unchanged files, and an oracle patch, the merged developer change used as the reference. Including unchanged files addresses the observed tendency to edit every file supplied in the input.

2.1 Reward modeling

The policy, the model being optimized, generates reasoning followed by search/replace edits, blocks specifying existing code and its replacement. Equation (1) assigns R(o) = −1 for an incorrectly formatted output; otherwise, R(o) = compare(patch_pred, patch_gt), where Python’s difflib.SequenceMatcher returns a sequence-similarity score between 0 and 1. Group Relative Policy Optimization (GRPO), an RL method that normalizes rewards within groups of sampled outputs, optimizes the policy using a clipped probability-ratio objective that limits update size and a penalty for divergence from a reference policy.

• For each prompt, the method samples G outputs and computes each advantage, the relative reward used for the update, as its reward minus the group mean, divided by the group standard deviation.
• Training supplies full file contents, so the model must identify detailed fault locations before producing repairs. File-level localization, identifying files relevant to a bug, and test generation are required by the evaluation pipeline but are not explicitly trained as RL subtasks.

The template makes repair the sole explicitly trained RL task and requires separate reasoning and solution fields. The formatting penalty enforces this structured response; the starting model’s formatting difficulties are quantified later in Table 2.

2.2 Aha moments and generalized reasoning capabilities

The authors describe “aha moments,” generated reasoning passages that revisit assumptions, explore alternative approaches, or divide a problem into smaller parts. Figure 3 illustrates these behaviors in repository repair, function implementation, and mathematics. These selected examples illustrate the authors’ interpretation of model behavior; the quantitative evidence for transfer appears in the later benchmark evaluation.

Three selected outputs illustrate revisiting an initial bug hypothesis, considering alternative function implementations, and decomposing a mathematics problem. They illustrate the authors’ claimed reasoning behaviors without measuring their frequency or causal contribution to accuracy.

3 Evaluation

The evaluation examines complete issue resolution, repair with reference files supplied, inference-time sampling, transfer to other tasks, and reward design. These experiments distinguish performance of the full Agentless Mini pipeline from the model’s repair ability under simplified conditions.

3.1 Experimental setup

Llama3-SWE-RL-70B starts from Llama-3.3-70B-Instruct and trains for 1,600 steps with a 16k-token context window, a global batch size of 512, and 16 sampled outputs for each of 32 problems per batch. Each global step performs one update with Adam, the training optimizer; a run uses 512 NVIDIA H100 graphics processing units (GPUs) and takes approximately 32 wall-time hours. Agentless Mini supplies full file contents during repair and delegates detailed fault identification to that step.

• The main evaluation generates 500 patches per issue at temperature 1.0, the sampling setting controlling output randomness, and uses the top 30 reproduction tests for execution and reranking, ordering candidate patches using their test results.
• Only the highest-ranked patch is submitted for each issue. The reported pass@1, the proportion of issues solved by that final submission, therefore follows substantial candidate generation and selection.
• Llama3-SWE-SFT-70B uses the same starting model and mixes synthetic localization and editing examples with Llama 3 coding and general SFT data. Llama-3.3-70B-Instruct generates the synthetic examples using the PR seeds as guidance.

3.2 Main results

Llama3-SWE-RL-70B reaches 41.0% on SWE-bench Verified with Agentless Mini, exceeding Llama3-SWE-SFT-70B’s 36.2% by 4.8 percentage points. It has the highest score among the paper’s listed models with at most 100B parameters, where B denotes billion. Table 1 reports GPT-4o with Agentless at 38.8% and o1-preview with Agentless at 41.3%, but those comparisons use different model and pipeline combinations and do not isolate the training method.

The most directly comparable rows use Agentless Mini: RL scores 41.0 and SFT scores 36.2. Other rows establish the paper’s reported competitive position, but their different models and scaffolds prevent a controlled attribution of score differences to training alone.

3.3 Baseline comparison

With oracle localized files, the files identified by the reference repair, supplied as context, greedy decoding, selecting the most probable next token at each step, yields repair scores of 34.8% for RL and 29.6% for SFT. Their formatting accuracy is similar, at 95.6% and 96.2%, respectively, supporting an improvement beyond output-format compliance. The starting Llama-3.3-70B-Instruct model achieves 5.4% repair with 12.2% correct formatting under greedy decoding; filtering malformed outputs and using 20-sample majority voting, selecting the most common repair, at temperature 0.6 raises repair to 16.6% and formatting to 44.6%.

Supplying reference files removes file-localization errors from the repair comparison. RL improves repair from SFT’s 29.6 to 34.8 while correct formatting decreases from 96.2% to 95.6%, indicating that its repair advantage is not explained by higher formatting accuracy.

3.4 Scaling analysis with more samples

With 30 reproduction tests fixed, increasing repair candidates from 20 to 160 raises issue resolution from 33.6% to 40.0%; increasing to 320 and 500 candidates yields 40.6% and 41.0%. With 500 repair candidates fixed, using 1, 5, 10, 20, and 30 reproduction tests yields 38.8%, 39.8%, 40.6%, 41.0%, and 41.0%, respectively. The gains diminish within the tested ranges, showing how candidate generation and test-based selection contribute to the headline score.

With 30 tests fixed, the repair-candidate curve gains 6.4 percentage points between 20 and 160 candidates but only another 1.0 point by 500. With 500 candidates fixed, the reproduction-test curve reaches 41.0 with 20 tests and remains there with 30.

3.5 Generalizability of RL

Under zero-shot evaluation, supplying no task examples, with greedy decoding, RL improves HumanEval+ function generation from 76.2 to 79.9, CRUXEval-I/O code execution reasoning from 60.5/61.9 to 71.6/75.5, and MATH mathematical reasoning from 63.2 to 73.7 under strict answer extraction. BigCodeBench-Hard library-use scores remain unchanged at 28.4 and 29.1, while Massive Multitask Language Understanding (MMLU), the general language-understanding evaluation, rises from 86.49 to 86.82. MMLU reports accuracy averaged equally across categories; the other benchmarks report pass@1.

• MATH uses the simple-evals evaluation software’s “Answer: …” format. Also accepting answers written with “\boxed…” raises the starting model to 70.9 and SFT to 71.7, while RL remains at 73.7; the strict-score improvement includes better format compliance.
• The SFT baseline declines on HumanEval+, both BigCodeBench-Hard settings, strict MATH, and MMLU, but improves both CRUXEval settings and lenient MATH relative to the starting model. Transfer therefore includes gains and ties for RL, and mixed changes for SFT.
• The paper reports aggregate significance at the 0.05 level using Eval Arena, a tool for assessing evaluation noise, and cites individual significance thresholds of > 0.8 percentage points on MMLU, 3 points on CRUXEval, and > 3 points on full MATH. The observed 0.33-point MMLU increase does not exceed its stated individual threshold; the aggregate claim should not be read as significance for every reported gain.

The table supports transfer beyond issue repair while revealing uneven gains: library-use scores tie the starting model, CRUXEval improves substantially, and MMLU improves only slightly. Strict and lenient MATH scores show how answer formatting affects the measured mathematics improvement.

3.6 Reward ablation

The reward ablation compares continuous patch similarity with a discrete reward that assigns 1 only for an exact patch match and 0 otherwise, retaining the formatting penalty and the same training setup. Continuous rewards yield 34.8% oracle-file repair and 95.6% correct formatting, compared with 29.0% and 94.2% for discrete rewards. The discrete training reward remains approximately zero at completion, supporting the practical value of partial similarity feedback when exact developer-patch matches are sparse.

The figure combines an outcome comparison with training-reward curves. Continuous similarity produces better oracle-file repair, 34.8 versus 29.0, while the discrete exact-match reward remains near zero at training completion, consistent with sparse positive feedback.

4 Related work

The related-work discussion covers language models for software engineering and training methods for software agents. It distinguishes the surrounding inference pipeline from the process used to improve the underlying model.

4.1 Language models for software engineering

The paper distinguishes agentic scaffolds, inference frameworks where a model chooses actions through tool interactions, from pipeline-based scaffolds, where predefined stages organize issue resolution. It positions Agentless Mini as a simplified pipeline that emphasizes code editing and independently scalable stages. The conclusion also identifies separate inference stages as a limitation because they restrict learning through interaction feedback across the workflow.

4.2 Training software agents

Lingma-SWE-GPT, SWE-Gym, and SWE-Fixer are presented as prior approaches that train software engineering models through SFT and incorporate distilled samples, training examples generated by stronger models, from GPT-4o or Claude-3.5-Sonnet. SWE-RL applies RL to a Llama model using public repository artifacts as seed data and developer patches as reward references. Because the compared systems also differ in starting models, training data, and scaffolds, their leaderboard scores alone do not establish the causal benefit of RL.

5 Conclusion

The conclusion attributes improved issue repair and transfer to RL on software evolution data. Its stated limitations are that sequence similarity may discourage functionally equivalent alternative repairs, localization based on repository structure lacks comprehensive context, and separate pipeline stages restrict learning from interaction feedback. These limitations affect the reward’s relationship to correctness and the workflow’s ability to consider the entire repair process.

Appendix

• NeurIPS Paper Checklist: The authors describe the work as empirical, document compute and experimental settings, and state that reward and evaluation implementations are included in supplementary material. The checklist also says that training data and the training pipeline cannot be shared at that time because of privacy and proprietary considerations. It acknowledges that error bars are not included for all SWE-bench evaluations, citing execution expense and the benchmark’s single-attempt evaluation convention.
• A Raw pull request data curation: The collection covers GitHub events from Jan 1, 2015 to Aug 31, 2024 and processes 4.6M cloned repositories, excluding all SWE-bench repositories to prevent training contamination from those repositories. Aggregation produces 24M PR instances; filtering leaves around 11M unique instances, from which 273k training seeds are selected. Llama-3.1-70B-Instruct predicts relevant unchanged files to counter the observed tendency to edit every file presented in the context.
• B Agentless Mini: The pipeline samples relevant file paths, generates repairs from complete file contents, retrieves a relevant test file to guide reproduction-test generation, and selects regression tests, existing tests used to check that repairs preserve prior behavior. Reranking first retains patches with the fewest regression failures, then groups patches that pass the same reproduction tests. Each consensus group receives score \(|\mathcal{P}| \times |\mathcal{T}|^2\), where P is its set of patches and T its passed reproduction tests, followed by majority voting within the highest-scoring group.
• C Synthesizing supervised-finetuning data: The baseline’s synthetic-data procedure follows Magicoder’s approach of generating instruction examples from open-source seeds. Llama-3.3-70B-Instruct produces localization and search/replace editing examples with reference files and patches as guidance; filtering checks file coverage, formatting, and whether search blocks match the input context. The SFT model trains for 2B tokens with a 16k context window using these examples plus Llama 3 coding and general SFT datasets.
• D Complete prompt: The appendix supplies PROMPT_TEMPLATE with issue and file-content placeholders, separate <think> and <solution> fields, and the required search/replace syntax. Each edit identifies a file and uses <<<<<<< SEARCH, =======, and >>>>>>> REPLACE delimiters, with exact source indentation and a separate code block for each edit. These requirements explain the output structure enforced during training.

Brief Thoughts

The comparison with the paper’s SFT baseline provides the clearest evidence for SWE-RL: it improves both full-pipeline resolution and greedy repair with reference files supplied, despite slightly lower formatting accuracy in the latter setting. Transfer gains are largest relative to the starting model on CRUXEval and remain present under lenient MATH scoring, while library-use scores are unchanged and the MMLU gain is small. The method offers an execution-free training reward, but evaluating and selecting useful repairs still requires execution, and the best reported result uses extensive candidate sampling.