An Empirical Study of Harness Design for Coding Agents | Summary

This article explains the key points of An Empirical Study of Harness Design for Coding Agents.

• 2026-09-17 (arXiv)
• Fan, Run-Ze, Zhang, Zihao, Ma, Simin, Hu, Yebowen, Wang, Shouju, Song, Kaiqiang, Liu, Fei, Zamani, Hamed, Wang, Xiaoyang.
• UMass Amherst, Zoom, Emory University, UNC Charlotte
• Paper

Read this article in Korean

Summary

• Coding agents depend on a harness—the software layer controlling execution, tools, and available interaction history—but comparisons of complete systems obscure which components improve task completion. Across four models and 176 matched settings, context management, which compresses interaction history to fit the model’s input window, improves success over no management by an average of 35.7 percentage points at 32k tokens on SWE-Bench Verified, a repository issue-resolution benchmark, compared with 2.7 points at 128k.
• T4, the tested policy combining removal of older tool-output bodies, external recovery of those outputs, and selective large language model (LLM) summarization, has the lowest aggregate cost among managed strategies at every tested context budget while maintaining broadly similar success rates. Adding recall, the ability to retrieve removed observations from external storage, produces no consistent accuracy improvement: mean success changes by −0.36 percentage points across 32 matched comparisons.
• Persistent planning, an explicit task plan updated and supplied to the model throughout execution, raises Nemotron-3 30B’s SWE-Bench success from 13.60% to 25.20% under T4 with a 128k-token window and predefined tools, while mainly reducing cost for stronger models. With planning enabled at T4/128k, bash-only, an interface using shell commands for general workspace operations, improves Nemotron-3 550B’s success and cost on both benchmarks but lowers Mistral-Medium-3.5-128B’s SWE-Bench success from 68.60% to 45.40%.

1 Introduction

Comparisons between complete coding-agent systems combine changes in planning, tool interfaces, and context management, making individual contributions difficult to identify. The paper asks whether these components are generally useful or whether their effects depend on model capability, task type, and resource budget.

• The study holds the execution loop and components outside each comparison fixed, evaluating three Nemotron-3 sizes and one Mistral model on repository repair and Terminal-Bench 2.1, a benchmark of end-to-end command-line tasks.
• The contribution is an empirical characterization of specific implementations. The experiments do not establish a universally optimal harness.

The schematic connects three configurable harness components to the four-model, two-benchmark, four-window evaluation. Its conditional outcomes preview how component value varies with context pressure, model behavior, and workload.

2 Harness Design

The authors build a modular harness around ReAct, a loop in which each turn contains reasoning, an action, and an observation. Planning maintains task progress, the action space defines the operations available to the agent, and context management controls which history reaches the model.

• Workspace access controls, post-edit diagnostics, and repeated-action detection remain configured consistently across comparisons.
• The action-space intervention changes both available tools and their interface instructions, file-state tracking, and validation support.

The upper loop identifies where planning, action schemas, and managed history enter the model input. The lower flow shows how T4 can avoid some summarization calls: it first replaces bulky older outputs, then summarizes if the remaining history reaches the hard threshold.

2.1 Planning

Planning uses an externally stored task plan maintained through update_plan, the tool that replaces the current plan. A system instruction defines the protocol, an initial reminder requests a plan, and subsequent model calls receive the current plan without accumulating old copies in conversation history.

• Disabling planning removes its instructions, reminders, injected plan, and tool while preserving the remaining execution configuration.
• The comparison measures this persistent planning implementation; it does not isolate planning as a general reasoning strategy.

2.2 Action space

The predefined action space exposes read_file, write_file, and edit_file for file operations; list_files, glob_files, and grep_text for discovery and search; web_fetch for retrieving a specified web page; and bash for shell execution. The bash-only condition removes predefined file, search, and web tools, retaining shell commands for general workspace interaction and any enabled auxiliary planning or recall tools.

• Web search is excluded because it could reveal the public pull request or reference patch associated with a SWE-Bench issue.
• Predefined file operations require a recorded complete read before modifying an existing file, update file state, and trigger supported diagnostics; shell modifications bypass that path.
• This comparison measures the complete interface change, including tool availability, instructions, state tracking, and validation support.

The inventory separates general workspace operations from auxiliary planning and recall functions. Bash-only removes predefined workspace tools while retaining enabled auxiliary tools, so this comparison preserves some structured tool calls.

2.3 Context management

Context management combines three mechanisms: elision (M1), which replaces older tool-output bodies with short placeholders; recall (M2), which stores removed observations externally for recovery through recall_event; and summarization (M3), which compresses older messages into a running natural-language summary. Summarization uses a separate, tool-free call to the same evaluated model.

• T0 disables management and terminates on context overflow, meaning that history exceeds the available input window; T1 enables M1; T2 enables M1 and M2; T3 enables M3; T4 enables all three.
• T4 first elides bulky observations at a soft threshold and summarizes only if the remaining history reaches a higher hard threshold. T1–T3 act at the hard threshold.
• The system prompt, initial task description, and a recent window containing at least two turns remain verbatim; compaction affects the intervening history.

The mechanism matrix identifies T1 versus T2 as the matched comparison for recall availability. T4 differs from simpler tiers through both its enabled mechanisms and the earlier threshold at which elision occurs.

2.4 Other components

Supporting components restrict workspace access, return tool failures as recoverable observations, diagnose supported Python edits, and detect repeated calls. Path checks reject escapes from the project root, while file-state checks detect modifications made since the recorded read and a permission layer classifies actions as allow, ask, or deny.

• Post-edit diagnostics use ruff or pyflakes, tools for checking Python code, or a syntax-only fallback to report issues such as syntax errors and undefined names.
• Repeated-call detection injects a reminder after five consecutive identical calls or identical failing calls and terminates after eight consecutive identical failing calls. Permission denials do not count as failures for this mechanism.
• Automatic file tracking and diagnostics apply through predefined file tools; equivalent bash modifications do not enter that path.

3 Experiment

The experiments measure task success rate, the fraction of tasks resolved, and mean token-priced cost per task. A context-management sweep supplies the main comparison grid, with additional planning and action-space ablations—comparisons that remove or replace one component—evaluated against the T4/128k baseline.

3.1 Setup

The evaluated models are Nemotron-3 30B, 120B, and 550B, plus Mistral-Medium-3.5-128B, with B denoting billions of parameters. SWE-Bench Verified contains 500 human-verified GitHub issues, while Terminal-Bench 2.1 contains 89 end-to-end command-line tasks.

• Models run locally through SGLang, the model-serving system, in BF16, a 16-bit numerical format. Generation uses temperature 0, top-p 0.95, a cumulative-probability sampling cutoff, and at most 16,384 output tokens per turn.
• The harness uses LangGraph for its execution workflow and Harbor to manage task containers and verifiers. Each task permits at most 300 steps, tool results are truncated to 24k characters, and up to eight read-only tools may execute concurrently.
• Soft and hard context thresholds are 0.6 and 0.85 of the usable window; the recent verbatim region receives 0.3 of that window, with a minimum of two turns.

Five management strategies at 32k, 64k, 96k, and 128k tokens produce 20 settings per model–benchmark pair, all with planning and predefined tools. Removing planning or switching to bash-only at T4/128k adds two settings, producing 176 settings across four models and two benchmarks.

• Success comparisons use two-sided exact McNemar tests, which compare paired task outcomes. The Benjamini–Hochberg procedure adjusts multiple comparisons to control the false discovery rate, the expected proportion of false discoveries among rejected hypotheses, at 0.05 within each comparison family.
• Cost uses OpenRouter prices accessed in August 2026. Per 1M input/output tokens, rates are $0.05/$0.20, $0.08/$0.45, $0.50/$2.20, and $1.50/$7.50 for the four models in the order listed above.
• These dollar amounts price token consumption; the models themselves are served locally.

3.2 Main Results

Context management primarily prevents premature termination from exceeding the available history window. Across 32k, 64k, 96k, and 128k budgets, the model-averaged managed–T0 success gap decreases from 35.7 to 15.9, 5.5, and 2.7 percentage points on SWE-Bench, and from 9.5 to 7.5, 4.8, and 2.8 points on Terminal-Bench.

• Over the same budget range, T0’s mean overflow rate falls from 78.7% to 8.7% on SWE-Bench and from 61.0% to 12.1% on Terminal-Bench.
• Every managed strategy records zero overflow failures at every tested budget. The shrinking success gap and overflow rate support overflow prevention as the principal observed benefit at small windows.

As context capacity increases, unmanaged success generally approaches managed success while unmanaged overflow declines. Every managed strategy has zero overflow failures, supporting overflow prevention as the principal observed benefit at tight budgets.

T4 achieves broadly comparable success to the other managed strategies and the lowest cost averaged over window budgets in seven of eight model–benchmark panels. Averaged across all eight pairs, it also has the lowest managed-strategy cost and lowest normalized peak context, the maximum history size divided by the nominal window budget, at every tested budget.

• T4 invokes summarization less often than summarization-only T3 on average at every budget, consistent with inexpensive early elision reducing the need for model calls.
• T4’s efficiency advantage is aggregate: it does not achieve the highest success or lowest cost in every individual setting.
• Comparing T2 with T1 isolates recall availability: T2 wins in 15 settings, loses in 14, and ties in three, with an equal-weight mean success difference of −0.36 percentage points.

The success ranges summarize sensitivity to context capacity, rather than statistical uncertainty. Their means and the cost curves show T4’s aggregate efficiency advantage while preserving model-specific exceptions.

The panels associate T4’s smaller normalized peak history and fewer summarization calls with its lower aggregate managed-strategy cost. Equal-weight averages across four models and two benchmarks can conceal exceptions in individual settings.

Recall is seldom used even when available. Among 64 T2 and T4 settings, 36 (56.3%) never invoke recall_event, and the median invocation rate is zero.

• Mean calls per task decrease from 0.540 at 32k to 0.069, 0.011, and 0.007 at 64k, 96k, and 128k.
• All 16 additional planning and action-space ablation settings at T4/128k record zero recall calls.
• This recall implementation provides no consistent accuracy benefit over elision alone. Its limited use does not establish that external memory is unnecessary in other agent designs.

At T4/128k with predefined tools, planning improves Nemotron-3 30B’s success by 11.6 percentage points on SWE-Bench and approximately 4.5 points on Terminal-Bench, at higher cost. For Nemotron-3 550B and Mistral-Medium-3.5-128B, planning reduces cost on both benchmarks with small decreases in success.

• On SWE-Bench, planning reduces the stronger models’ costs by approximately 30% and 32%, while success decreases by 2.0 and 0.4 percentage points, respectively.
• Nemotron-3 120B has no consistent success gain: planning increases SWE-Bench cost but reduces Terminal-Bench cost.
• Trajectory analysis associates these results with longer execution before an initial edit for the weakest model and less post-edit verification for stronger models.

The paired bars show distinct planning trade-offs. Planning increases weak-model success with additional spending, while the stronger models show small success decreases and lower cost under the same T4/128k and full-tool configuration.

The preferred action interface depends on both model and benchmark. Under planning-enabled T4/128k, predefined tools raise Nemotron-3 30B’s success from 10.20% to 25.20% on SWE-Bench and from 3.37% to 13.48% on Terminal-Bench.

• On Terminal-Bench, 66% of this model’s bash-only trajectories terminate after emitting calls to unavailable tools, indicating a mismatch between its learned tool-call patterns and the exposed interface.
• For Nemotron-3 550B, bash-only raises success from 65.80% to 69.40% on SWE-Bench and from 44.94% to 50.56% on Terminal-Bench, while reducing cost by approximately 53% and 30%.
• Mistral shows a workload reversal: bash-only lowers SWE-Bench success from 68.60% to 45.40%, but raises Terminal-Bench success from 37.08% to 43.82%. Its Terminal-Bench cost rises from $2.22 to $2.75, so the latter improvement carries an additional cost.

The SWE-Bench results show large managed-strategy gains at 32k and smaller differences at 128k. The final ablations expose opposing interface effects: bash-only improves Nemotron-3 550B to 69.40% at $1.11 per task but reduces Mistral to 45.40%.

Interface preferences differ from those observed in repository repair. Mistral improves from 37.08% with predefined tools to 43.82% with bash-only, while cost rises from $2.22 to $2.75; many individual contrasts lack statistical significance on the 89-task benchmark.

The success, cost, and shell-use panels reveal model- and workload-specific interface preferences. Mistral’s greater shell reliance in terminal tasks is consistent with its benchmark reversal, while Nemotron-3 30B performs poorly with bash-only.

4 Analysis

Trajectory analysis, the study of agents’ sequences of model calls and actions, connects aggregate outcomes to workflow changes. Model-based annotations classify SWE-Bench turns into Localize, finding relevant code; Reproduce, recreating the bug; Fix, modifying source; Verify, checking results; and Other, auxiliary work, while Terminal-Bench actions use Understand, Write Code, Verify, and Other.

• At 32k, unmanaged SWE-Bench runs have median lengths of 20–30 turns and often stop during localization; managed runs extend to approximately 50–180 turns, allowing more runs to reach verification.
• At 128k, management-tier differences diminish: SWE-Bench medians span only 39–42 turns for Nemotron-3 30B and 70–74 for Nemotron-3 550B.
• These profiles describe longer execution under context pressure with broadly preserved workflow composition. They do not independently establish a causal behavioral mechanism.

Planning changes stopping behavior in different ways across models. For Nemotron-3 30B on SWE-Bench, enabling planning increases median trajectory length from 5 to 40 turns and reduces termination without an edit from 68.6% to 27.8%.

• The fraction stalled at localization falls from 58.4% to 10.4%, supporting the interpretation that planning keeps this model active through an initial edit attempt.
• For Nemotron-3 550B and Mistral, planning reduces median SWE-Bench length from 108 to 74 and from 68 to 53 turns, respectively; most removed activity is post-edit verification.
• On Terminal-Bench, planning can both extend prematurely ending runs and shorten the long-running tail, explaining why its net cost effect varies by model.

The active-run profiles distinguish early termination from extended verification. Removing planning reduces Nemotron-3 30B’s median SWE-Bench trajectory to five turns but lengthens the stronger models’ trajectories mainly through additional verification.

Bash-only changes action granularity, the amount of work expressed in an individual operation, allowing capable models to combine operations in larger shell commands or scripts. For Nemotron-3 550B on Terminal-Bench, it reduces the median trajectory from 47 to 31 actions while increasing the Write-code share from 16% to 27%.

• On SWE-Bench, that model’s mean re-patches, edits to already edited files, decrease from 4.6 to 1.5 per task, while its median largest edit grows from 18 to 54 lines.
• Every evaluated model performs fewer re-patches under bash-only, and its Terminal-Bench file-writing actions shift toward creating or replacing files. Largest-edit size does not increase for every model.
• These observations are consistent with operation bundling, but the interface ablation also changes prompts, tracking, and diagnostics, preventing attribution solely to action granularity.

The terminal-task profiles show changes in both early exits and long-running tails. For Nemotron-3 550B, bash-only produces a shorter trajectory with a larger share of code-writing actions, consistent with combining operations into fewer interactions.

Nemotron-3 550B’s re-patches fall from 4.6 to 1.5 and its median largest edit rises from 18 to 54 lines under bash-only. Other models also re-patch less, but their largest edits do not all increase, limiting a general claim that bash-only always produces larger edits.

5 Related Work

The related-work discussion covers coding models, coding harnesses, and history-management methods. The study examines implemented harness components across model sizes and explicit context-window budgets.

5.1 Coding Agents

Execution-based coding benchmarks evaluate complete agent systems, so their scores reflect both the language model and its surrounding control loop, tool interface, and history policy. The authors use this observation to motivate component comparisons within a fixed execution framework.

5.2 Coding Harness

Prior harness research includes integrated coding systems, constrained localization–repair–validation pipelines, modular agents, and studies of model-specific architecture preferences. The paper extends this work by testing implementation-level planning and workspace interfaces over long coding trajectories while explicitly varying context availability.

• The closest cited prompt-scaffolding study uses a full factorial design, testing all component combinations, on shorter reasoning tasks; the present study examines longer executions where context capacity can terminate a run.
• Its planning and interface experiments remain restricted to T4/128k, so coverage of component interactions is partial.

5.3 Context Management

The paper distinguishes fixed context policies, including removal, retrieval, and summarization, from methods that train models to manage their own context. It evaluates fixed policies across two model families and 32k–128k windows to characterize how their usefulness depends on available capacity and model behavior.

6 Conclusion

The study concludes that harness components should be selected for the target model, workload, and resource budget. Context management addresses overflow, persistent planning changes premature or delayed stopping, and the action interface affects how models express workspace operations.

• Staged elision and summarization provide the lowest aggregate cost among the tested managed strategies at broadly similar success rates.
• Planning and interface results support conditional choices, with Mistral’s benchmark-dependent preference demonstrating that one interface does not dominate across workloads.

Limitations

The results apply to particular implementations: one planning protocol, fixed compaction thresholds, and an action-space change that bundles tools, prompts, tracking, and diagnostics. Planning and action space are tested only at T4/128k, leaving their interactions with tighter windows and other management strategies unresolved.

• Each setting runs once per task. Terminal-Bench has only 89 tasks, and many contrasts do not reach statistical significance; directional patterns must be distinguished from significant individual comparisons.
• The model coverage comprises three sizes from one family and one cross-family model. Model size is an imperfect measure of capability because training and prior tool exposure can also affect interface proficiency.
• SWE-Bench Verified is Python-only. Transfer to other languages, model families, harness implementations, and non-software tasks requires additional validation.

Appendix

• A Harness Prompts documents how fixed instructions combine with runtime task information. A.1 System prompts records interface-specific workflow and tool-use instructions; A.2 Planning prompts specifies plan initialization and reinjection; A.3 Context-management prompts provides the running-summary and removed-output templates; A.4 Stuck-detection prompts defines reminders at five repeated calls and termination at eight identical failing calls.
• B Tool Descriptions provides protocols, errors, and side effects. B.1 File input and output describes line-numbered reads and recorded-read requirements for file changes; B.2 Search covers directory, filename-pattern, and content searches; B.3 Execution and web access documents /bin/sh -c execution and concrete-URL retrieval; B.4 Planning and context recall describes update_plan and recall_event as independently enabled auxiliary interfaces.
• C Trajectory Analysis defines the behavioral evidence. C.1 Annotation Schemes includes C.1.1 SWE-Bench Failure-Stage Attribution, which selects the earliest failure among finding the correct file, finding the relevant lines, implementing the patch, and verification; C.1.2 SWE-Bench Behavior Encoding assigns five workflow phases to turns; C.1.3 Terminal-Bench Action-Level Encoding assigns 10 action types grouped into four phases to individual tool calls.
• C.2 Judge Setup and Human Validation uses GPT-5.5 with high reasoning effort and temperature 0.6 for annotation. Three human annotators independently label non-overlapping portions of 200 trajectories, covering 15,610 units; aggregate judge–human raw agreement is approximately 94.2%, with weighted mean Cohen’s κ, a chance-adjusted agreement measure, of 0.929. The validation measures judge–human agreement under the stated labeling scheme; the non-overlapping splits do not measure agreement among humans on the same units.
• C.3 SWE-Bench Trajectory Results combines C.3.1 Failure-Stage Results and C.3.2 Behavior Profiles: weaker-model failures concentrate more on finding the correct file, while stronger-model failures more often concern patch implementation. Mistral’s unresolved-run file-localization share rises from 16.0% with predefined tools to 41.4% with bash-only; the behavior profiles show active-run proportions and workflow phases across all four context budgets. C.4 Terminal-Bench Trajectory Results provides corresponding profiles for command-line tasks with planning and predefined tools held fixed.
• C.5 Trajectory-Level Statistics at 128k reports full trajectory lengths, tool-call counts, edit granularity, and termination-stage statistics, distinguishing runs that never edit or write code from unresolved runs that do. C.6 Recall-Event Usage reports mean recall_event calls per task rather than the fraction of runs using recall. C.7 Judge Prompts reproduces the classification protocols used for workflow and failure-stage annotation.

Brief Thoughts

The paper connects component-level cost and success differences to observable execution behavior. Overflow measurements, termination before an edit, and verification profiles help explain when a harness component is useful beyond its aggregate benchmark score. Staged elision and summarization is a supported starting configuration under restrictive history capacity, with its aggregate cost advantage checked against the target model’s individual results. The recall result supports measuring actual retrieval use before accepting added implementation complexity; it does not support removing memory from every agent design. A broader interaction study and repeated runs would strengthen the planning and action-interface findings, particularly on the smaller Terminal-Bench evaluation.