Mid-Harness: Scaling Actions Between Model and Harness for Terminal Agents

This research introduces Mid-Harness, a novel framework designed to systematically investigate how allocating test-time compute at the model-harness boundary can enhance the reliability of actions generated by terminal agents. The core problem addressed is that while stochastic model generations produce candidate actions, their execution success is not guaranteed; a poor command (e.g., an incorrect package installation) can derail subsequent progress, even when superior alternatives exist within the same generative model.

Mid-Harness operates by inserting a sampling and verification step directly into the model's action generation pipeline, without altering the underlying generator or harness architecture. The proposed mechanism modifies the standard model call wrapper as follows:

Model-call wrapper (Mid-Harness)

async def generate(messages):

• return await llm.generate(messages)

• candidates = await llm.generate(messages, n=N)

• return await verify(llm, messages, candidates)

This structure allows researchers to study the interplay between action candidate generation (sampling N actions) and subsequent verification before a single action is forwarded to the environment.

The study establishes a critical dependency: verification governs the benefit of action sampling. The effectiveness of generating multiple candidates is highly contingent upon the quality of the verifier employed.

1. Verifier Strength Dictates Benefit: When using a weak verification mechanism, wider action sampling yields negligible improvements in trajectory success. Conversely, a strong verifier enables substantially more successful trajectories by effectively exploiting useful alternatives from the same generator.

2. Optimal Verification Mechanism: Among various mechanisms evaluated—including Listwise, Pointwise, and Pairwise verification—Pairwise verification performs best. This suggests that comparing candidate actions against each other provides the most informative signal for identifying viable paths.

3. Leveraging Verifier Distillation: A significant performance boost is achieved by distilling the responses from a stronger verifier back into the original model (e.g., TMAX-9B). This distillation process improves Pass@1 success without requiring any modification to the action generator itself, demonstrating how knowledge transfer across verification layers can enhance agent performance.

4. Semantic Disagreements: Offline analysis revealed that command semantics and execution feasibility are persistent sources of disagreement between the distilled verifier and the teacher model, accounting for a substantial portion (67.4%) of verification failures.

Mid-Harness is not only effective in isolation but also demonstrates synergistic benefits when combined with other scaling techniques:

• Synergy with Trajectory Scaling: Combining Mid-Harness with established trajectory scaling methods, such as Best-of-T (T=3) and Sequential Refine (R=1), leads to higher overall success rates. Crucially, this combined approach achieves superior performance at a lower estimated token cost compared to simply generating a larger number of trajectories alone. This suggests that targeted action refinement is more computationally efficient than brute-force trajectory expansion.

• Cross-Model and Benchmark Robustness: The framework demonstrated improved performance across different underlying models, distinct benchmarks, and various harness configurations, indicating its generalizability as a powerful tool for improving terminal agent reliability.

The central conclusion of the paper is that action scaling represents a promising target for test-time compute scaling in terminal agents. Mid-Harness provides the necessary systematic study to determine precisely when additional computation (sampling and verification) translates into tangible improvements in trajectory success, proving its compatibility with both parallel and sequential trajectory scaling strategies.

Despite its strong findings, the authors acknowledge several limitations that point toward future research:

1. Gap to Frontier Verification: The distillation approach leaves a substantial gap when compared to frontier verification methods, motivating the need for better training algorithms, such as reinforcement learning specifically designed for verifiers.

2. Lack of Gold Labels: The evaluation inherently lacks gold action labels, which limits the direct measurement of verification correctness and candidate coverage—a crucial area for future rigorous validation.

3. Failure Analysis: The study notes that command semantics and execution feasibility are primary drivers of failure in distilled-verifier scenarios, reinforcing the need for verifiers to be highly attuned to these aspects.

The provided supplementary text from Candidate B describes a different type of problem-solving context—a code refactoring/debugging task involving transitive symmetric aliases and lexicographical ordering. While this is not directly related to the terminal agent paper, it illustrates the type of concrete, fix-oriented work that contrasts with purely observational debugging.

Here are specific improvements for AI systems based on the findings in Mid-Harness: Scaling Actions Between Model and Harness, detailing what these improved systems can achieve:

) Mid-Harness (Action Verification) Integration

The core improvement is integrating a dedicated action verification step at the model-harness boundary. This moves beyond simple generation to a structured process of Generate Candidates → Verify Candidates → Execute Best Candidate.

1. The AI system can be architected with a fixed Generator (e.g., TMAX-9B) and Harness, but an added Mid-Harness wrapper that samples multiple candidate actions from the generator based on the current history.

2. Before execution, this wrapper passes these candidates to a Verifier (which can be zero-shot or distilled). The Verifier's role is not just to judge plausibility but to explicitly evaluate the candidate against the task requirements, considering correctness, progress toward the goal, and risk of state change.

3. The system then executes only the single action chosen by the Verifier.

) Improved System Capabilities (What they can do)

This Mid-Harness architecture enables several high-value capabilities:

1. Active Error Correction and Recovery: The system can actively correct its own mistakes in real-time. If a generated command is flawed (e.g., wrong package install or incorrect code edit), the Verifier identifies this failure and selects a candidate that resolves the specific blocker, even if other candidates seem plausible in isolation.

2. State-Aware Search Space Pruning: Instead of blindly exploring all possible next actions, the system can use verification to prune irrelevant branches of the search tree. This means it spends its limited compute only on actions that demonstrably advance toward the task completion goal or fix known errors.

3. Enhanced Reliability on Complex Tasks: For long-horizon tasks (like software engineering), this system dramatically increases success rates (e.g., Pass@1 improvements of up to 80% compared to the base agent). It can reliably navigate complex dependencies and configuration changes where simple stochastic generation fails, as demonstrated by its ability to solve tasks where the base agent succeeds in less than 5% of runs.

4. Cost-Effective Scaling: The system proves that allocating test-time compute (for verification) is more effective than simply generating more trajectories or increasing reasoning effort for a single candidate. This allows developers to achieve higher reliability at lower estimated token costs by optimizing the action scaling axis.

) Advanced Verification Strategies (How to make it smarter)

To maximize performance, the following specialized verifier mechanisms should be employed:

1. Pairwise Comparison (The Gold Standard): The system should use a pairwise verifier mechanism. This involves comparing two candidate actions against each other to generate a preference score based on their likely impact on the task progress and state change, rather than scoring them in isolation.

2. Knowledge Distillation for Verifiers: The system can benefit from training the Verifier model using distilled knowledge from a powerful frontier model (like GPT-5.6 Sol). This allows the smaller, fixed generator to leverage the superior judgment of a much larger model without needing to serve that massive model at inference time, significantly narrowing the gap between weak and strong verification capabilities.

3. Decision-Only Verification for Efficiency: For high-throughput scenarios, using decision-only verifiers (which output only a single A/B token instead of a full rationale) can reduce inference costs by 20–24% while maintaining substantial gains in Pass@1, offering an optimal trade-off between performance and latency.

) Specific Application Examples:

The improved AI system can be used to:

• Automatically resolve complex debugging scenarios (e.g., fixing specific error codes via configuration changes rather than general code edits).

• Successfully complete multi-step software development tasks by correctly sequencing dependency installations and environment setups.

• Navigate ambiguity in ambiguous terminal environments where multiple valid commands exist, by using pairwise comparison to select the action that minimizes risk or maximizes progress toward a known sub-goal.

Terminal agents act through stochastic model generations, yet the ability to generate a useful action does not ensure its reliable execution. A poor command (e.g., wrong package install) can change the environment in ways that hinder subsequent progress, even when the model could generate a better alternative. We investigate whether allocating test-time compute at the model-harness boundary can improve action reliability and trajectory success, and what makes this allocation effective. To study these questions, we introduce Mid-Harness, which samples and verifies candidate actions before forwarding one for execution, while keeping the generator and harness unchanged. With a TMAX-9B generator, more action sampling yields little benefit under weak verification, whereas a capable verifier can exploit useful alternatives from the same generator. On TerminalBench-Lite, a GPT-5.6 Sol verifier raises Pass@1 from 50.00% for the base agent to 68.03% with 8 sampled actions. When the same TMAX-9B model serves as the verifier, pairwise verification performs best among the evaluated verification mechanisms. Distilling responses from the stronger verifier into TMAX-9B further improves Pass@1, while leaving the action generator unchanged. With TMAX-9B on TerminalBench-Lite, combining action and trajectory scaling reaches higher success at lower estimated token cost than generating more trajectories alone. Mid-Harness also improves performance across additional models, benchmarks, and harnesses. These findings identify action scaling as a promising target for test-time compute scaling in terminal agents.