Mid-Harness: Scaling Actions Between Model and Harness for Terminal Agents | Summary

This article explains the key points of Mid-Harness: Scaling Actions Between Model and Harness for Terminal Agents.

• 2026-09-30 (arXiv)
• Kang, Minki, Hachiuma, Ryo, Zhang, Shaokun, Radhakrishnan, Subhashree, Fu, Yonggan, Jiang, Jindong, Liu, Mingjie, Hosseini-Asl, Ehsan, Dong, Yi, Wang, Yu-Chiang Frank, et al.
• NVIDIA, KAIST
• Paper
• Project Page

Read this article in Korean

Summary

• Terminal agents, systems that use language models to execute commands, can derail later work by choosing a poor action even when the model could generate a useful alternative. With the TMAX-9B action generator and its execution harness, the software that runs commands and returns observations, held fixed on the TerminalBench-Lite benchmark of terminal tasks, Mid-Harness with GPT-5.6 Sol verification and eight candidate actions raises Pass@1, the average exact-success rate across three runs per task, from 50.00% to 68.03%.
• Mid-Harness allocates additional inference computation before each action: it samples alternatives from the same interaction history, verifies them, and executes one. When TMAX-9B also verifies its own actions without fine-tuning, comparing candidates in pairs achieves 54.76% Pass@1; supervised distillation, training the verifier to reproduce a stronger teacher’s responses, raises this to 57.14% without changing the generator.
• Action verification also improves selection among complete trajectories, the sequences of actions and observations produced by task executions: distilled Mid-Harness combined with selection among three trajectories reaches 66.33% Pass@1, versus 55.10% for trajectory selection alone. The study identifies verifier quality as a constraint on action scaling, but lacks gold action labels, uses estimated token costs, and reports uncertainty and task-specific regressions.

1. Introduction

Mid-Harness: Scaling Actions Between Model and Harness for Terminal Agents studies when additional computation before command execution improves task completion. Each executed command changes the environment for subsequent decisions, so the study examines both whether the generator supplies useful alternatives and whether verification can identify them. Its main comparisons vary candidate count, verification mechanism, and verifier capability while holding the generator and harness fixed.

• Action-level verification chooses the next command, whereas trajectory-level verification chooses among completed runs.
• The strong-verifier experiment indirectly probes whether useful alternatives exist. Self-verification, using the same model to generate and judge actions, tests how much of that opportunity the generator can recover. For Self-verification, see DuPO: Enabling Reliable LLM Self-Verification via Dual Preference Optimization | Summary.

The diagram locates candidate generation and verification before environment execution, showing how the method changes action selection while preserving the surrounding agent. Its numerical panels distinguish the alternatives exploited by a stronger verifier from the smaller gains recovered by self-verification and distillation.

2. Mid-Harness

Mid-Harness changes the model-call wrapper, the function through which the harness requests model outputs, to request several candidate actions and return the verifier’s selected candidate. The harness loop, action generator’s weights, and serving architecture remain unchanged; subsequent verifier distillation modifies only the verifier.

2.1. Scaling Actions Within a Trajectory

At step t, the generator samples N actions conditioned on the same history, and the verifier returns one member of that candidate set. The harness executes only that action, appends its observation to the history, discards the other candidates, and samples a fresh set at the next step. This procedure requires one environment instance.

• Candidates may serve different purposes, such as diagnosing a failed command or retrying it, so selection considers unresolved task requirements and the current state.
• The generator produces reasoning before each action using <think> tags. The verifier receives the task, observed history, and candidate actions, but does not receive the reasoning generated for those candidates.

2.2. Verification Mechanisms

Listwise verification selects from the full candidate set in one prompt; pointwise verification scores each candidate independently; pairwise verification compares two alternatives and aggregates their preferences. Listwise selection requires a joint ranking, pointwise scoring requires comparable scores across different action purposes, and pairwise comparison supplies a shared reference at the expense of more calls.

• The default pairwise mechanism ranks candidates by margin-weighted win rates, weighting each comparison by the difference between its two assigned scores, with a minimum weight of 0.1.
• For eight candidates, listwise verification uses one call and pointwise verification uses eight. Exhaustive pairwise comparison would use 28 calls; the implemented eight-candidate tournament instead compares neighbors in a ring, then compares candidates against four selected pivots.

3. Experimental Setup

The main experiments use TMAX-9B with the Vanillux2 execution harness on 98 TerminalBench-Lite tasks, excluding two tasks with sandbox service errors. TMAX-9B was trained from Qwen3.5-9B through reinforcement learning, training from task rewards, on synthetic terminal tasks; generation uses temperature 0.8, at most 64 steps, a 65,536-token context, and a 16,384-token output limit per step. Each task receives three runs.

• Pass@3 is the fraction of tasks solved by at least one of the three runs. Failures remain in the denominator, and fractional rewards do not count as exact success.
• Best-of-T Trajectories selects one output from T completed runs using a model-based tournament. Sequential Refine (SR) summarizes a previous run to guide a fresh run, repeated for R refinement rounds.
• Parallelized output tokens (POT) sum the longest output in each parallel generation or verification stage to approximate decoding latency under ideal parallel execution. Total verifier output tokens measure aggregate decoding volume, while reference-priced token costs apply fixed prices to counted input and output tokens.

4. When Does Action Scaling Work?

The action-scaling experiments hold the TMAX-9B generator fixed and examine candidate coverage, candidate width, verification mechanism, verifier distillation, and inference cost. A strong verifier provides an indirect probe of useful alternatives, while comparisons among TMAX-9B verifiers test how effectively self-verification selects them.

4.1. Candidate Coverage

Candidate coverage means that sampled actions contain useful alternatives. GPT-5.6 Sol listwise verification reaches 64.63% Pass@1 with N = 4 and 68.03% with N = 8, compared with 50.00% for the base agent. These task-level gains provide indirect evidence of exploitable alternatives; without action-level ground truth, they do not directly measure coverage or establish a perfect-verification ceiling.

4.2. Candidate Width and Verification

Increasing candidate width offers little benefit under weak verification: zero-shot listwise verification, using the verifier without fine-tuning, changes Pass@1 from 49.32% at N = 4 to 51.02% at N = 8, while Pass@3 changes from 66.33% to 67.35%. At N = 8, pointwise verification reaches 52.38% Pass@1 and 67.35% Pass@3; pairwise verification performs best among the evaluated zero-shot mechanisms at 54.76% and 71.43%, respectively.

Increasing width from four to eight candidates barely improves weak listwise selection, whereas GPT-5.6 Sol listwise verification reaches 68.03% Pass@1. Pairwise comparison and distillation recover part of this gap, showing that candidate count alone does not determine the return from action scaling.

4.3. Verifier Distillation

Verifier distillation uses approximately 117k GPT-5.6 Sol pairwise responses collected from 732 trajectories across 244 difficult tasks drawn from the TMAX-15k task collection. LoRA, low-rank adaptation for fine-tuning a model, trains the verifier to reproduce the teacher’s reasoning, scores, and preference label; its adapter is activated only for verification.

• At N = 8, distillation raises pairwise Pass@1 from 54.76% to 57.14% and Pass@3 from 71.43% to 75.51%. At N = 4, the corresponding changes are 54.42% to 55.44% and 68.37% to 70.41%.
• The distilled pairwise verifier remains below GPT-5.6 Sol listwise verification at 68.03% Pass@1 with N = 8. This comparison varies both verifier model and verification mechanism.

4.4. Cost for Action Scaling and Verification

Pairwise verification improves success but adds substantial decoding work. At N = 8, zero-shot pairwise verification adds an estimated 26.6k POT, compared with at most 1.3k for listwise and pointwise verification; total verifier output is 125.7k, 0.6k, and 7.0k tokens, respectively.

• Decision-only verification emits a single A/B preference token. At N = 8 with TMAX-9B, zero-shot and distilled decision-only variants improve Pass@1 while reducing reference-priced token cost by 20.9% and 24.1%, respectively.
• Decision-only verification lowers Pass@1 at 4B and 27B. With TMAX-9B at N = 4, both decision-only variants also have lower Pass@1 than their reasoning counterparts; the observed benefit depends on model scale and candidate width.
• POT is an idealized decoding proxy rather than measured end-to-end latency. Shorter verifier outputs leave the costs of repeated verifier inputs in place, and the decision-only variants also change aggregation and, for distillation, training targets.

The two panels separate idealized parallel decoding cost from aggregate verifier output. Reasoning-based pairwise verification improves success but incurs much more verifier decoding, while the decision-only points show higher TMAX-9B Pass@1 with shorter responses in the evaluated eight-candidate setting.

5. Analysis: What Still Limits Verification?

The offline analysis compares verifier outputs on stored TMAX-9B trajectories from 21 held-out TMAX-15k tasks with N = 8. Pairwise agreement measures matching A/B/TIE preferences, while verification agreement measures whether the verifier and teacher identify the same top-ranked candidate. These diagnostics measure teacher agreement under shared states, not independently established action correctness or online task success.

5.1. What Distillation Improves

Distillation reduces score mean absolute error (MAE), the average absolute teacher–student score difference, from 2.59 to 1.05. Pairwise agreement rises from 59.01% to 74.58%, and verification agreement rises from 38.52% to 57.79%. These results establish closer imitation of the teacher’s comparative judgments on stored states.

5.2. Remaining Verification Challenges

Distillation improves verification agreement in every episode-turn bin, but agreement remains lower later in the trajectory: the distilled verifier reaches 68.13% at turns 1–4 and 54.07% at turns 17–32. GPT-5.6 Terra’s review identifies 3,328 clear verifier failures before distillation and 1,810 afterward, with fewer cases in every category.

• Candidate semantics, judging what a command or edit does, and execution feasibility, judging whether it can run as intended in the current environment, account for 67.4% of reviewed distilled-verifier failures.
• These counts include only disagreements judged to have an unambiguous, evidence-supported teacher preference. They are model-judged failures relative to a teacher reference, rather than independently verified action errors.
• Turn-level agreement retains 1,355 of 1,624 states whose comparisons are valid for both models. It is conditional on this filtering, and the final turn bin covers fewer tasks than earlier bins.

Distillation improves teacher agreement across all turn bins, yet later states remain difficult. The category bars identify command semantics and execution feasibility as the largest remaining sources of reviewed failures; these counts concern model-reviewed teacher disagreements rather than independently established action errors.

Lower score error accompanies higher agreement on both pairwise preferences and selected candidates. These measurements establish closer teacher imitation on stored states, with online task success evaluated separately.

6. Composition and Transfer of Action Scaling

The composition experiments test whether action verification improves parallel trajectory selection and sequential refinement without increasing their number of environment executions. Both methods require fresh environment runs, which can be difficult outside benchmarks without reliable state serialization, the ability to save and restore environment state. Transfer experiments then vary model scale, model family, benchmark, and harness.

6.1. Composition with Trajectory Scaling

With TMAX-9B, Best-of-T at T = 3 reaches 55.10% Pass@1; generating those trajectories with zero-shot or distilled Mid-Harness raises success to 61.22% and 66.33%, respectively, using the same three environment executions. With SR at R = 1, applying distilled Mid-Harness to both source and refinement runs raises Pass@1 from 55.10% to 60.20% and Pass@3 from 71.43% to 75.51%.

• Distilled Mid-Harness alone matches Best-of-T at T = 5, both at 57.14% Pass@1, for about one-third of the reference-priced token cost.
• Combining distilled Mid-Harness with SR at R = 1 or Best-of-T at T = 3 exceeds Best-of-T at T = 7, which reaches 59.18%. The SR combination uses less than half its estimated token cost.
• The cost comparison uses OpenRouter Qwen3.5-9B reference rates of $0.08 per million input tokens and $0.13 per million output tokens. These are proxies for locally served models, with complete-pipeline accounting and shared generator prompts within each action step.

The component combinations show how action verification changes the trajectories supplied to selection and refinement. With TMAX-9B, distilled Mid-Harness raises three-trajectory selection from 55.10% to 66.33% Pass@1 without increasing its three environment executions; the 4B and 27B results show that outcomes also depend on model scale and composition.

The input-token, output-token, and reference-price panels compare complete pipelines under the paper’s accounting assumptions. Distilled action verification matches five-trajectory selection at 57.14% Pass@1 for about one-third of the estimated cost, and its compositions exceed seven-trajectory selection at lower estimated cost.

6.2. Transfer Across Models, Benchmarks, Harnesses

On TerminalBench-Lite, base, zero-shot, and distilled Pass@1 progress from 38.78% to 41.50% to 43.88% at 4B, and from 71.09% to 73.13% to 76.19% at 27B. Across seven additional benchmark–model–harness settings, zero-shot Mid-Harness improves Pass@3 and matches or improves Pass@1.

• On Terminal-Bench 2.1, a benchmark of realistic terminal tasks, TMAX-9B improves from 21.72% to 27.34% Pass@1 with zero-shot verification; distilled verification reaches 26.59%, below zero-shot verification.
• On the 23-task CPU-only FeatureBench-Mini subset for software feature development, TMAX-9B Pass@1 rises from 1.45% to 5.80% with zero-shot verification and 7.25% with distillation. Distillation nevertheless lowers Pass@3 relative to zero-shot verification, from 17.39% to 13.04%.
• The study also reports gains on the 50-task SWE-bench Verified Mini subset for resolving repository issues and with Qwen3.5-9B, Nemotron3.5 Lightning (30B), and Nemotron3 Ultra (550B) using the Terminus-2 execution harness.

Zero-shot verification improves Pass@3 in all seven transfer settings and matches or improves Pass@1. Distillation has mixed effects relative to zero-shot verification, including lower Terminal-Bench 2.1 Pass@1 and lower TMAX-9B FeatureBench-Mini Pass@3.

7. Related Work

The paper situates Mid-Harness among work on terminal-agent interfaces and harnesses, test-time compute scaling, process reward models that evaluate intermediate steps, and agent trajectory selection or refinement. Prior work already verifies per-step candidates and combines action ranking with trajectory selection; this study emphasizes controlled comparisons of verification mechanisms and generator-based self-verification across broader terminal tasks.

8. Conclusion

The experiments support action scaling as a complementary allocation of inference computation: useful alternatives can improve task success when verification identifies them, and verifier training recovers part of a stronger verifier’s advantage. The main limitations are the remaining gap to frontier verification, the paper’s strong-model reference, and the absence of gold action labels, which prevents direct measurement of verification correctness and candidate coverage.

AI use statement

The authors disclose generative AI use for synthetic datasets, implementation, methodology or experiment feedback, qualitative and thematic analysis, translation, figures, and readability editing. They state that AI was not used to propose or refine hypotheses, clean or reformat datasets, or interpret results, that all AI-assisted work was reviewed, and that two authors checked and tested generated code.

Ethics statement

The experiments run in isolated benchmark environments and measure successful task completion. The authors distinguish this action reliability from command safety or security and recommend retaining restricted permissions, environment isolation, and human approval for sensitive or irreversible actions.

Reproducibility statement

The reproducibility statement points to documented evaluation tasks, metrics, baseline procedures, timeout settings, verifier mechanisms and prompts, distillation data and training settings, inference cost calculations, and offline disagreement analysis. The PDF also links to the project page at https://byungkwanlee.github.io/MidHarness-page/.

Appendix

• A. Limitations and Future Work identifies unresolved verifier training, evaluation, and compute-allocation questions. The study does not compare action sampling against greater reasoning effort for one candidate, and it verifies at every action step rather than allocating effort adaptively. Changing a selected action changes later states, so replaying stored trajectories cannot establish online performance under a different verifier; intermediate-state branching would require environment cloning or restoration. Proposed extensions include learning from command outcomes, predicting action effects, and testing settings beyond terminal agents.
• B. Experimental Setup and Implementation and B.1. Evaluation and Baseline Procedures specify 98 TerminalBench-Lite tasks after excluding network-log-normalization and okhttp-trailers-crash, all 89 Terminal-Bench 2.1 tasks, 50 SWE-bench Verified Mini tasks, and 23 CPU-only FeatureBench-Mini tasks. All experiments use an agent-timeout multiplier of three. Best-of-T selects a completed output, whereas SR starts a fresh environment using a summary of the previous run; the SWE-bench subset is identified at https://github.com/mariushobbhahn/SWEBench-verified-mini.
• B.2. Verification Mechanisms and Hyperparameters specifies six possible unordered comparisons at N = 4 and a two-stage ring-and-pivot tournament at N = 8 with K = 4 pivots. Eight distinct valid candidates require 22–25 comparisons, depending on pivot positions. Candidates with identical tool names and arguments are deduplicated; invalid judgments are omitted, and specified fallback rules return a candidate when comparison evidence is insufficient.
• Default pairwise judgments contain a rationale, integer scores from 1 to 10, and an A/B/TIE preference. Their win-rate weights are the larger of 0.1 and the absolute score difference divided by 9. Decision-only verification instead normalizes the first-position A/B token probabilities and averages preference probabilities, changing aggregation as well as response format.
• B.3. Verifier Training Data and Settings reports 117,631 training inputs from 244 tasks and 10,543 analysis inputs from 21 disjoint tasks. Training retains 56,274 A wins, 56,274 B wins, and 5,083 ties; LoRA uses rank 64, alpha 128, dropout 0.05, learning rate 10⁻⁴, two epochs, per-device batch size 8, gradient accumulation 1, and four H200 GPUs. The analysis tasks are used only for final analysis, and the same training dataset is used at 4B and 27B. The separately generated decision-only corpus contains 113,728 training and 10,188 development examples and excludes ties.
• B.4. Inference Cost Accounting sums the longest output in each parallel stage for POT and sums all executed verifier outputs for total verifier output cost. Reference pricing counts a shared generator prompt once per action step, all candidate outputs, and every executed verifier input; complete trajectory-selection and refinement pipelines include their source runs and auxiliary calls. For TMAX-9B the cost is (0.08I + 0.13O)/10⁶ dollars, with I and O denoting counted input and output tokens. The 4B rates are scaled by 4/9, while 27B uses $0.30 and $2.00 per million input and output tokens; these fixed rates serve as proxies for locally served models.
• B.5. Evaluation Effort explains that the cost of repeated interactive execution constrains evaluation to three runs per task. For the TMAX-9B base agent, median task-mean execution times are 7.2 minutes on TerminalBench-Lite and 10.5 minutes on Terminal-Bench 2.1; median task-mean step counts are 21.0 and 28.3. These timings exclude environment setup and final evaluation, retain timed-out durations, and describe the authors’ serving setup.
• B.6. Verifier Prompt Templates provides B.6.1. Listwise Verification, B.6.2. Pointwise Verification, and B.6.3. Pairwise Verification. The templates evaluate commands against the original task, executed history, current state, unresolved requirements, repeated work, and evidence for completion. Listwise outputs a choice, pointwise outputs an independently assessed score and predicted next state, and pairwise outputs comparative reasoning, two scores, and a preference.
• C. Additional Experiments and Results and C.1. Numerical Performance Results provide the exact values behind the main plots. Sampling eight candidates and executing the first parsable action reaches 49.66% Pass@1, compared with 50.00% for the base agent, providing no observed benefit from this sampling-and-parsing procedure. At N = 8, distilled decision-only verification reaches 59.18% Pass@1 but 73.47% Pass@3, compared with 57.14% and 75.51% for reasoning-based distilled verification.
• C.2. End-to-End Costs with Decision-Only Verification reports that distilled TMAX-9B action verification reduces reference-priced cost from $0.207 to $0.157 per final output at N = 8. Its Best-of-T composition at T = 3 changes from 66.33% to 65.31% Pass@1 while cost falls from $0.797 to $0.608; its SR composition at R = 1 changes from 60.20% to 65.99% while cost falls from $0.399 to $0.309. Separate live runs, different training targets and aggregation, and missing usage limit attribution of these changes solely to response format.
• C.3. Additional Verifier Inference Compute finds no consistent benefit from generating more verifier output. Five responses per pair lower distilled Pass@1 from 57.14% to 53.40% with 4.99× verifier output; enabling thinking raises zero-shot Pass@1 from 54.76% to 56.12% with 4.86× output. Rubric scaling, the evaluated rubric-based verification variant, reaches 55.10% with 2.26× output.
• C.4. Uncertainty in Comparisons against Base Agent resamples 98 paired task blocks 100,000 times, keeping each task’s three runs together. The marginal 95% bootstrap confidence intervals for distilled 9B and 27B Pass@1 gains are [+1.36, +12.93] and [+0.34, +9.86] percentage points; the other four tested base-versus-Mid-Harness intervals include zero. These intervals support positive differences for those two distilled configurations under this analysis and do not establish improvement on every task or benchmark.
• C.5. Mid-Harness Across Task Groups shows that aggregate gains coexist with domain-specific regressions. Distilled verification improves software engineering, machine learning, debugging, and system setup at all three tested sizes, but lowers 27B scientific-computing Pass@1 by 10.0 percentage points. Improvements do not increase monotonically with observed execution length across models; the Extreme and build/dependency groups contain only three and four tasks, respectively.
• D. Verifier Analysis, D.1. Offline Verifier Agreement, and D.1.1. Score and Turn Diagnostics describe 20,394 score observations from 10,197 common valid pairs. State-level verification agreement retains 1,355 of 1,624 states and excludes 269 states with at least one invalid comparison. It uses saved ring-comparison win counts, unlike online margin-weighted aggregation; the comparative integer scores are not calibrated task-success probabilities. The score distributions and joint A/B score plots show that distillation brings the verifier’s ratings closer to the teacher’s distributions.
• D.2. Remaining Verifier Disagreements and D.2.1. Examples of the Two Largest Failure Categories illustrate model-judged errors in command semantics and execution feasibility relative to the teacher. One verifier incorrectly treats the equivalent C additions ‘a’ + s1 and s1 + ‘a’ as different in validity. Another rewards a planned service launch despite preceding process-matching cleanup that can terminate the command itself or its parent before launch.
• D.3. Qualitative Examples of Action Verification and D.3.1. Detailed Action-Verification Traces present one recovery and one error from TMAX-27B zero-shot runs with eight candidates and four pivots. Recovery Example: React and TypeScript Production Build selects diagnostics.ignoreCodes: [6137], after which all three Jest test suites, 16 tests, and 17/17 external tests pass; the base agent succeeds in 0/3 runs and Mid-Harness in 3/3. The successful preference still contains an incorrect claim that diagnostics.exclude is invalid: the paper explains that it filters file paths, whereas ignoreCodes filters diagnostic codes.
• Verifier Error: Build-System Alias Canonicalization shows the verifier favoring a rewrite over investigating a conflict between the explicit lexicographic rule and a worked example. The chosen heuristic excludes header names when selecting a canonical name, yielding compile where the evaluator expects build. The run passes 11/12 tests but receives final reward 0, illustrating how a preference for an immediate edit can overlook unresolved evidence about the required behavior.

Brief Thoughts

The paper’s most useful result is the difference between weak and strong action selection with the generator fixed. Verifier-only training improves online success, while simply increasing candidate count often achieves little. This provides an empirical basis for comparing spending on additional candidate commands, verifier training, and complete task runs.

The practical comparison needs verifier quality and complete-pipeline cost together. Pairwise verification adds substantial computation, teacher agreement measures imitation, several confidence intervals include zero, and decision-only results vary across model scales and candidate widths. The reported token-cost estimates support comparisons under the stated accounting assumptions; they do not establish measured deployment latency savings.