SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering? | Summary

This article explains the key points of SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering?.

• 2025-02-17 (arXiv), Proceedings of the 42nd International Conference on Machine Learning, PMLR 267, 2025
• Miserendino, Samuel, Wang, Michele, Patwardhan, Tejal, Heidecke, Johannes.
• OpenAI
• Paper
• Github

Read this article in Korean

Summary

• SWE-Lancer tests whether large language models (LLMs), text-generating models used here as coding agents, can complete paid freelance software engineering work and select suitable implementation proposals. Under the paper’s single-attempt evaluation, Claude 3.5 Sonnet earns $403,325 of the full dataset’s $1,000,000 in historical payouts, compared with $380,350 for o1 and $303,525 for GPT-4o.
• The benchmark contains 1,488 tasks from Expensify’s repository and Upwork postings: 764 Individual Contributor (IC) software engineering (SWE) tasks requiring code patches and 724 SWE Manager tasks requiring proposal selection. Patches are graded with hidden end-to-end tests, browser-based checks of complete application workflows, while managerial selections are compared with the original managers’ decisions.
• On the public SWE-Lancer Diamond split, Claude 3.5 Sonnet solves 26.2% of implementation tasks and selects the accepted proposal on 44.9% of managerial tasks. Additional attempts and greater reasoning effort improve o1’s results, but estimated cost savings depend on verification and human completion of unresolved tasks; the dataset covers one repository and one freelance platform.

1. Introduction

The paper motivates SWE-Lancer with the need for evaluations that remain challenging as coding benchmark scores increase. It combines implementation and technical management tasks from a commercial application with actual historical payouts, measuring both task completion and the monetary value assigned to completed tasks.

• The public Diamond split contains 502 tasks worth $500,800; the remaining $499,200 is retained as a private holdout, a separate evaluation set. The released evaluation environment and benchmark are available at https://github.com/openai/SWELancer-Benchmark.
• Diamond tasks took approximately 26 days to resolve and accumulated approximately 47 comments on average. These statistics include discussion and elapsed time, rather than directly measuring engineering labor.
• The authors commissioned 100 professional software engineers to develop and validate tests instead of selecting only issues that already had tests.

1.1. Related Work

The related-work discussion spans generating functions from specifications, competitive programming, scientific coding, repository completion, and issue resolution. SWE-bench and SWE-bench Verified evaluate real repository issues, while SWE-bench Multimodal extends evaluation to visual software domains; SWE-Lancer adds historical payouts, proposal selection, and professionally written tests of application workflows.

• The paper argues that selecting issues with existing unit tests, checks of isolated functions or components, can favor readily testable problems. Its alternative is to construct tests after selecting freelance tasks.
• These differences describe evaluation coverage. The cited scores do not constitute a controlled comparison of benchmark difficulty with identical models, tools, and execution budgets.

2. SWE-Lancer

SWE-Lancer contains 764 IC SWE tasks worth $414,775 and 724 SWE Management tasks worth $585,225. Implementation tasks provide the issue description and a repository snapshot preceding the fix; managerial tasks provide the issue, the earlier codebase, and competing proposals from the original discussion.

• An implementation earns its payout only when all associated hidden tests pass. The tests use Playwright, a browser automation library, and are unavailable to the model during evaluation.
• A managerial answer succeeds when its selected proposal matches the historical choice. An additional review by experienced engineers achieved 99% agreement with the original selections.
• Figure 3 distinguishes task topics from kinds of work: application logic, server-side logic, interface behavior, and system quality can involve bug fixes, features, or reliability improvements.

The workflow separates the model-visible issue and reverted repository from the human-created grader. Its all-tests-must-pass payout rule explains why a patch that fixes only one symptom can receive zero earnings.

The managerial track evaluates selection among supplied proposals while allowing local codebase inspection. Success reflects agreement with the historically implemented proposal, so the measured skill differs from writing and validating a patch.

The two groupings distinguish where engineering work occurs from what change is requested. Application logic or interface tasks can include features and reliability work as well as bug fixes.

2.1. Evaluation Metrics and Task Types

The evaluation reports the percentage of tasks resolved and the historical payouts attached to successful tasks. Earn rate is earned payout divided by the total available payout, so it weights results by task price.

• Figure 4 illustrates one postal-code validation issue whose bounty rose from $1,000 to $8,000 over four weeks after five early proposals were rejected. The example supports interpreting price as a market-derived signal of difficulty, while showing that it also reflects an evolving review process.

The postal-code example links bounty increases to unsuccessful proposals and emerging edge cases. It supports interpreting payout as a difficulty signal while showing that price changes during review.

2.2. Benchmark Construction

Construction proceeds through repository selection, task review, task generation, end-to-end test development, and provision of a user tool, a script that simulates the reported user actions against the modified local application. Engineers reviewed issue clarity, executable environments, and test coverage; Section 2.2 specifies three reviews for IC tasks, two for management tasks, and validation by ten experienced engineers for IC tasks exceeding $5,000.

• The user tool writes a text trajectory, a record of application actions and states, and screenshots to the working directory without explicitly reporting success or failure. In the evaluated text-only setup, agents inspect the textual records to assess their changes.
• Tests simulate complete workflows such as signing in and performing application actions. Automated checks additionally confirm that tests pass on the fixed version and fail on the broken version.

3. Experiments and Results

The experiments measure implementation and proposal-selection performance, then vary the number of attempts, reasoning effort, and access to the user tool. A separate price analysis combines model execution costs with the historical freelancer costs of tasks left unresolved.

3.1. Experiment Setup

Agents operate in Docker containers, packaged execution environments, with a preconfigured repository, no Internet access, no GitHub remote, and no future commits. The baseline evaluates gpt-4o-2024-08-06, o1 with high reasoning effort, and claude-3-5-sonnet-20240620 using a basic scaffold, the execution framework that exposes repository browsing, file editing, and terminal commands.

• The principal metric is pass@1, success from one sampled attempt. The user tool is enabled for baseline implementation tasks.
• Appendix A.5 specifies sampling temperature 1.0, at most 100 tool calls, and a maximum of 3 hours per rollout, one complete agent attempt. Agents receive no image or video inputs.
• The authors note that single-attempt results can vary substantially between runs; Claude 3.5 Sonnet was evaluated on the full set with pass@1 only because of rate limits.

3.2. Main Results

Claude 3.5 Sonnet leads the baseline comparison on both splits: Diamond pass@1 is 26.2% for IC SWE and 44.9% for SWE Manager, versus 16.5% and 41.5% for o1 and 8.0% and 37.0% for GPT-4o. Its Diamond payout is $208,050 of $500,800, and its full-set payout is $403,325 of $1,000,000.

• On the full set, IC SWE pass@1 is 8.6%, 20.3%, and 21.1% for GPT-4o, o1, and Claude 3.5 Sonnet respectively; SWE Manager pass@1 is 38.7%, 46.3%, and 47.0%.
• Table 1 separates task success from payout-weighted performance: Claude 3.5 Sonnet achieves 36.1% overall Diamond pass@1 and a 41.5% earn rate, compared with 33.7% and 40.3% on the full set.
• Managerial results are consistently higher than implementation results, but they measure agreement on proposal selection rather than successful execution of the selected proposal.

Total historical payouts are $303,525 for GPT-4o, $380,350 for o1, and $403,325 for Claude 3.5 Sonnet. Even the leading baseline earns less than half the available $1,000,000; these are benchmark payout totals rather than payments received through deployed freelance work.

All three models score higher on proposal selection than implementation on both splits. Claude 3.5 Sonnet’s implementation score changes from 26.2% on Diamond to 21.1% on the full set; the figure reports performance on distinct task sets rather than a controlled change in difficulty.

The table jointly reports task success, rounded earnings, and earn rate, revealing differences between task-count and monetary weighting. Its reasoning-effort and tool columns specify the conditions needed to compare baseline and ablation results.

3.3. Increasing Number of Attempts

The repeated-attempt experiment measures pass@k, the estimated probability that at least one of k attempts succeeds, on Diamond IC SWE tasks for k from 1 to 7. o1 rises from 16.5% at one attempt to 48.5% at seven attempts, while GPT-4o rises from 8.0% to 16.5%.

• The estimator is pass@k = E_Problems[1 − C(n − c, k) / C(n, k)], where n is the number of sampled attempts, c is the number of correct attempts, and C denotes a binomial coefficient.
• These results measure whether the attempts include a successful candidate. Using that candidate also requires recognizing success, which the five-attempt price analysis assumes can be done quickly.
• The prose states that GPT-4o reaches 16.5% at pass@6, whereas Figure 7 labels 15.2% at six attempts and 16.5% at seven; the figure’s endpoint is used here.

o1’s success estimate increases from 16.5% with one attempt to 48.5% with seven, compared with GPT-4o’s increase from 8.0% to 16.5%. The plotted endpoint places GPT-4o’s 16.5% result at seven attempts, differing from the accompanying prose’s reference to six.

3.4. Increasing Test Time Compute

With the user tool enabled on Diamond IC SWE tasks, increasing o1’s reasoning effort, its inference-time computation setting, from Low to High raises pass@1 from 9.3% to 16.5%. Rounded earnings rise from $16K to $29K, and earn rate rises from 6.8% to 12.1%; Medium reasoning effort reaches 15.6% pass@1.

• Figure 8 shows that the benefit varies by price range. For $1K–2K tasks, pass@1 rises from 8.6% at Low effort to 25.7% at High effort; for $2K+ tasks, it rises from 7.7% to 15.4%.
• The under-$500 group is not monotonic: Medium effort scores 19.0%, compared with 17.1% at High effort. Greater effort improves the aggregate result without improving every price group.

Higher reasoning effort produces the largest plotted gain in the $1K–2K group, from 8.6% to 25.7%. The under-$500 group peaks at Medium effort, showing that the overall improvement does not imply monotonic gains within every price range.

3.5. Removing User Tool

Removing the user tool leaves GPT-4o’s Diamond IC SWE pass@1 at 8.0% and reduces o1’s from 16.5% to 13.1%. This ablation, an experiment removing one component, shows a larger task-success benefit for o1 under the evaluated scaffold.

• Table 1 also shows that GPT-4o’s earn rate changes from 6.0% with the tool to 7.2% without it despite identical task success rates. The reported benefit therefore varies by model and metric.

Removing the tool reduces o1’s implementation success by 3.4 percentage points, from 16.5% to 13.1%, while GPT-4o remains at 8.0%. The result shows a model-dependent benefit from access to the simulated user trajectory.

3.6. Price Analysis

The price analysis averages five runs of GPT-4o and o1 on 237 Diamond IC SWE tasks worth $236,300. Cost savings equal one minus the sum of model costs and freelancer costs on model failures, divided by the freelancer cost of solving all tasks.

• With one attempt followed by freelancer fallback, in which humans complete unresolved tasks, estimated savings are 8.84% for GPT-4o and 13.26% for o1. Allowing five attempts before fallback raises the estimates to 18.6% and 33.5%, respectively.
• The model contribution ratio divides payouts on successful model tasks by combined model and fallback freelancer costs. GPT-4o contributes $0.10 and o1 $0.16 per $1 of combined spending.
• The reported mean execution costs and 95% confidence intervals are $194.31 ± $17.31 for GPT-4o and $1,623.76 ± $56.68 for o1. Estimates use customer application programming interface (API) prices, include retries, exclude batch pricing, and rely on historical bounties; the five-attempt scenario assumes quick verification even though verification can take over 24 hours.

3.7. Discussion

The discussion identifies incomplete diagnosis and ineffective validation as recurring obstacles: agents often locate relevant files but fail to account for interactions across components and submit partial fixes. Stronger agents more effectively wait for the user tool, parse its output, and use it during debugging; the tool commonly takes 90 to 120 seconds.

• Table 2 shows Claude 3.5 Sonnet’s advantage on Diamond IC interface tasks: 31.7% versus o1’s 17.1% and GPT-4o’s 2.4%. However, o1 leads the managerial server-side category at 61.5%, compared with Sonnet’s 38.5%, so the aggregate ranking does not hold in every category.
• Some categories contain very few tasks: system-wide quality and reliability includes only three Diamond IC tasks and two managerial tasks. Their extreme percentages provide limited evidence about broader capability.

The category breakdown qualifies the aggregate ranking: Claude 3.5 Sonnet leads IC interface tasks, while o1 leads managerial server-side tasks. The quality-and-reliability groups contain only three IC tasks and two managerial tasks, limiting interpretation of their 0% and 100% outcomes.

4. Limitations

All tasks come from Upwork and the Expensify repository, limiting generalization across organizations, languages, and engineering roles. Infrastructure work is underrepresented, and tasks extend an established application rather than measuring development from an empty starting point; freelance tasks also tend to be more self-contained than full-time engineering work.

• The text-only environment excludes screenshots and videos and prevents clarifying questions. These restrictions limit the context available to agents.
• Public Diamond issues date from 2023 to 2024, so contamination, exposure to evaluation material during training, remains possible. Disabling Internet access prevents direct solution lookup during evaluation but cannot remove prior training exposure.
• The before-and-after-cutoff analysis does not show a consistent advantage on earlier tasks. It is an indirect contamination check rather than proof that the dataset was absent from training.

5. Future Work

The authors propose using task-linked payouts to investigate productivity, labor-market effects, and the economic consequences of autonomous software engineering. They also identify multimodal evaluation, supplying screenshots and videos alongside text, as an extension because many original issues contain recordings of expected behavior.

6. Conclusion

The paper concludes that SWE-Lancer broadens software engineering evaluation through paid application-development tasks, technical proposal selection, and execution-based grading. Most tasks remain unresolved by the evaluated models under the specified environment and execution framework.

Impact Statement

The Impact Statement describes possible productivity gains and wider access to engineering, alongside potential shifts in demand for entry-level and freelance labor. It also identifies risks from insecure or disruptive generated code and autonomous systems capable of self-improvement or unauthorized information transfer; these are prospective considerations, not outcomes measured by the benchmark.

Appendix

• A. Appendix supplies category analyses, contamination checks, evaluation details, prompts, qualitative trajectories, dataset statistics, price calculations, and preliminary open-source model results. These materials document benchmark construction and qualify the interpretation of the reported scores.
• A.1. Pass@1 Rate Analysis by Category reports Diamond results by nature of work. For IC bug fixes, GPT-4o, o1, and Claude 3.5 Sonnet score 9.6%, 19.2%, and 28.4%; for new features or enhancements, they score 0.0%, 4.8%, and 14.3%. Categories were assigned using three o1 assessments of each complete issue thread, with manual resolution when the assessments disagreed.
• A.2. Contamination Analysis divides tasks around the stated training-data cutoff dates: 2023-10-01 for GPT-4o and o1 and 2024-04-01 for Claude 3.5 Sonnet. IC performance is higher after the cutoff for all three models, while managerial differences have mixed directions; the table supplies no consistent pattern of higher scores on earlier tasks, but does not directly test whether training included particular solutions.
• A.3. Further exploration of SWE Management Tasks examines image pasting on iOS. The accepted proposal handled multiple clipboard formats, reduced permission prompts, and supported native pasting behavior; Figure 11 shows why less comprehensive alternatives were rejected as the bounty increased from $250 to $2K.
• A.4. Coding and Software Engineering Evaluations Comparison Table contrasts benchmark languages, data sources, tested capabilities, monetary metrics, and grading methods. Its cross-benchmark score comparisons use different scaffolds and input conditions, and its statement that GPT-4o can ingest screenshots conflicts with the text-only configuration specified in A.5 and the main limitations.
• A.5. Additional Scaffold Details reports Microsoft Azure Standard_D2as_v4 virtual machines with 2 vCPUs, virtual processor cores, and 8GB RAM, working memory, alongside Docker settings of 64 GB shared memory and 192GB maximum available memory. The PDF does not reconcile these host and container memory figures. The stated rollout limits are 100 tool calls, 3 hours, temperature 1.0, and no multimodal image or video inputs; experiments other than pass@k use a single rollout.
• A.6. Task Curation Criteria requires an open-source issue with a concrete payout, a historical resolution, a clear specification, and a locally verified fix; managerial tasks additionally require at least two sufficiently detailed proposals. It states that each final task was screened by at least three engineers, whereas Section 2.2 specifies two reviews for management tasks. Issue clarity and test validity are rated from 0 to 3, with acceptance requiring a majority rating of 0, the strongest rating; Figure 12 connects proposal selection to managerial tasks and the implemented fix to IC tasks.
• A.7. SWE-Lancer Robustness to Cheating illustrates grader exploitation, passing an evaluation without fixing the intended behavior, with an o1 patch in SWE-bench Verified that changes the attribute lookup from filterable to _filterable, bypassing an error branch. The authors argue that complete user-workflow tests resist such shortcuts better, but this example does not quantify SWE-Lancer’s resistance to exploitation.
• A.8. Prompting provides IC SWE Task Instructions and SWE Management Task Instructions. The implementation template specifies the issue, repository, payout, and user-tool trace inspection; the management template requests a selected_proposal_id in /app/expensify/manager_decisions.json. The authors report no observed performance difference from including versus excluding prices in the prompt.
• A.9. Qualitative Analysis of Rollouts examines Issue #14958, a missing postal-code validation error. In o1 Interaction Summary, removing onValueChange addresses part of the symptom but misses broader validation requirements; in 4o Interaction Summary, changing regular expressions, text-pattern rules used for validation, fails to repair the error-display mechanism; in Claude 3.5 Sonnet Interaction Summary, a comma-specific check fixes the stated symptom but remains incomplete. All three fail the end-to-end test, which also checks international postal formats and other requirements developed in the historical discussion.
• A.10. Model Usage of User Tool and Example of o1 using the user tool describe parsing trace.trace logs and frame-snapshot records, saved representations of the application page, to investigate sign-up and Concierge navigation. The account shows iterative inspection, edits, and rerunning the tool, including a navigation.canGoBack() check for users without navigation history; it illustrates a debugging process without establishing the aggregate causal effect of tool use.
• A.11. Holdout Set describes the $499,200 private split, including $178,475 of IC tasks augmented by engineers. For those tasks, descriptions and tests remain constant while engineers reintroduce the issue through a distinct but equivalent code change, followed by three-way validation for clarity, faithfulness, and feasibility.
• A.12. Additional SWE-Lancer Dataset Details shows that Diamond IC tasks are 74.26% client-side application logic and 87.76% bug fixes, while Diamond management tasks are 75.85% client-side application logic and 93.58% bug fixes. Diamond issues average 26.15 days to resolution and 47.01 comments across both task types; IC changes average 69 lines across 2 files. Diamond also has a larger share of tasks priced at $2000 and up than the full set, 9.4% versus 3.8%, so the splits differ in their payout distributions.
• A.13. Example IC SWE and SWE Management Task Prompts includes Example IC SWE Task, a $500 offline attachment-download issue, and Example SWE Manager Task, the $8000 postal-code validation issue with six numbered proposals. These examples show the reproduction steps, expected and actual behavior, platform information, and proposal discussions supplied to models.
• A.14. Price Analysis - Additional Details reports that model execution costs, without conditioning on success, equal 0.08% of total freelancer costs for GPT-4o and 0.7% for o1, averaged over five runs. Payouts on correctly solved tasks divided by model API costs across all tasks are $109 per $1 for GPT-4o and $20 per $1 for o1; these ratios exclude the freelancer fallback costs included in the main model contribution ratio.
• A.15. Evaluation of Open-Source Models reports one-seed results, a single experimental run per configuration, for Deepseek-R1 and Meta Llama 3.3 70B Instruct under the default scaffold. Diamond IC pass@1 is 3.8% and 4.6%, respectively, while both achieve 40.4% on Diamond managerial tasks; full-set earn rates are 29.3% and 27.7%. The authors caution that the scaffold may understate their capabilities and describe repository-navigation errors, premature patching or termination, repeated failed strategies, and limited user-tool use.

Brief Thoughts

Combining complete application-workflow grading with actual task prices makes SWE-Lancer useful for examining both implementation reliability and the distribution of completed work by value. Table 1 demonstrates why task success and earn rate should be reported separately. The managerial track measures a different skill: matching a historical proposal choice does not establish that an agent can deliver the implementation.

The postal-code case warrants a specification audit because its tests incorporate requirements developed through discussion beyond the initial symptom. The paper reports clarity and test-validity reviews, but assessing this example’s fairness would require checking whether every tested requirement was inferable from the model-visible description and codebase. The cost-saving estimates likewise depend on recognizing correct solutions and should be read as conditional calculations, not observed production savings.