SWE-Together: Evaluating Coding Agents in Interactive User Sessions

Most coding-agent benchmarks are static, but real coding assistance is interactive, requiring users to clarify goals and correct mistakes over multiple turns. This paper introduces SWE-Together, a multi-turn benchmark reconstructed from 11,260 recorded user–agent coding sessions to evaluate agents as collaborators by measuring both final correctness and the number of corrective feedback turns required during interaction.

How it works

SWE-Together transforms raw multi-turn coding agent sessions into reproducible interactive software engineering tasks through a three-component pipeline. First, the task construction pipeline filters and normalizes raw sessions, screens for local reproducibility, and converts viable sessions into sandboxed repository-level tasks with pinned environments and executable checks. This stage involves:

1. A deterministic rule-based collector that filters raw upstream sessions based on criteria like containing multiple genuine user messages, concrete agent actions or code edits, and sufficient repository signal to identify the working repository.

2. An LLM judge that determines whether the coding work can be reconstructed as a reproducible and verifiable task.

3. A sandbox orchestrator that runs task construction to produce a complete task directory inside an isolated sandbox, where a task-generation agent performs a stricter repository-grounded screen, writes tests, and constructs user-simulation prompts.

How it works (Continued)

Second, the user simulator replays the original user’s intent in a trajectory-conditioned manner, intervening only when conditions derived from the recorded session are satisfied. The simulator's action space includes no-op, question, redirect, new-requirement, and check-external. It follows two principles: interventions are trajectory-conditioned rather than scheduled and anchored to the original session rather than generic. This ensures that feedback timing is adapted to each agent's evolving trajectory while preserving the original user’s objectives and intervention order.

How it works (Continued)

Third, the evaluation framework separately measures correctness of the agent’s final repository state and user-simulator behavior. Task correctness is assessed using a combination of deterministic verifiers with an agentic rubric judge. The rubric is fixed offline across agents to ensure comparability: Phase 1 runs once per task to derive a weighted task rubric, and Phase 2 applies that same rubric to each candidate repository state. The final score is derived mechanically: score = round X / w g I[g met], 2, where weights are normalized.

How it works (Continued)

User-simulator behavior is characterized through two diagnostics: Intent Coverage and User Correction. Intent Coverage measures simulator fidelity by decomposing original session trajectories into atomic intents and matching the simulator’s messages against these intents, deriving IntentCoverage = round(0.70 Irecall + 0.30 Iprecision, 2). User Correction tests the hypothesis that stronger models require less intervention; it is computed as UserCorrection = Ncorrection + 0.2 Nnudge, where explicit corrections receive full weight and nudges receive a smaller weight.

How it works (Continued)

The evaluation reports four equally task-weighted correctness metrics: pass@1, SSR, pass2, and MeanJudge. The paper finds that User Correction is strongly inversely correlated with performance, suggesting that stronger models require less corrective intervention to reach the same outcome. Furthermore, Intent Coverage remains broadly stable across coding agents, indicating cross-agent comparability.

How it works (Continued)

The experiments evaluate seven frontier models on the full 109-task benchmark using the opencode harness with k = 2 replicates per task. The results show that Claude Opus 4.8 achieves the strongest overall performance, leading in pass@1 (63%), SSR (59%), pass2 (52%), and mean judge score (0.801), while requiring the least corrective steering with a mean User Correction of 1.38. The simulator quality study found that human annotators exhibited no statistically reliable ability to distinguish simulated users from real users, achieving a Turing pass rate of 46%.

How it works (Continued)

The paper distinguishes SWE-Together from prior work by combining repository-level agent-environment interaction, interactive user-correction replay, and provenance from real recorded user–agent coding sessions. This design choice is supported by the finding that simulators grounded in real human behavior yield substantially better downstream collaborative assistants than role-playing prompts. The limitation noted is that the simulator cannot interrupt the coding agent during its turns or rely on visual information from the interface.

How it works (Continued)

The conclusion emphasizes that evaluating coding agents requires accounting for interaction quality and the human effort needed for success, showing that User Correction is the one interaction signal that tracks capability. SWE-Together aims to evaluate agents in a way that more closely reflects the actual user experience. The overall finding is that stronger models need less steering to reach comparable performance.

Here are specific improvements to AI systems based on the SWE-Together benchmark and methodology described in the paper, along with what these improved systems can achieve:

The primary improvement is shifting from evaluating static, one-shot code generation to evaluating agents as active collaborators within complex, multi-turn engineering workflows.

A system that utilizes SWE-Together (or a similar interactive replay framework) can be trained or fine-tuned to exhibit superior behavior in environments requiring iterative refinement and clarification.

The improved AI system can be specifically engineered to reduce the number of required user interventions (User Correction) needed to achieve a high final success rate, indicating it is more capable of handling complex, evolving specifications autonomously.

By incorporating the metrics from the evaluation protocol, an AI system can be optimized not just for correctness but also for Intent Coverage. This means the system will be trained to ensure that every underlying user goal and constraint introduced across multiple turns is accurately reflected in its final output, reducing drift or omission of requirements.

The system can be designed to minimize unnecessary corrective feedback (User Correction) while maintaining high performance on task correctness. This suggests an improved user experience where the agent's initial understanding and trajectory are more aligned with the user's evolving intent, leading to fewer frustrating back-and-forth corrections.

An AI system can be evaluated for its Efficiency (Wall-clock time and output tokens per task). The goal is to develop models that achieve high correctness scores with lower computational cost, making them more practical for real-world deployment where latency and resource consumption matter.

The system can be fine-tuned using the feedback loop derived from the User Simulator. This allows the agent to learn a policy of when to proactively ask clarifying questions or provide updates based on its current progress, mimicking a highly effective human collaborator rather than just a reactive code generator.

A system can be developed with robust verification pipelines that use deterministic verifiers alongside agentic rubrics (as described in Section 2.3.1). This ensures that the final code not only looks correct but is also executable and satisfies the specific behavioral goals of the original session, preventing false negatives from superficially correct but functionally flawed solutions.

The system can be benchmarked against a baseline using metrics like Pass@2 (joint success across replicates), which heavily penalizes inconsistency between different agent runs on the same task. This drives development toward models with high reliability and stable performance rather than those that succeed only sporadically.

Most coding-agent benchmarks are static: an agent receives a complete task description up front and is judged only by its final code. Real coding assistance is interactive, with users clarifying goals, adding constraints, and correcting mistakes over multiple turns. We introduce SWE-Together, a multi-turn benchmark reconstructed from real user-agent coding sessions. To make real interactions verifiable, we curate 109 repository-level tasks from 11,260 recorded sessions, selecting sessions with recoverable repository states, clear user goals, and observable outcomes. To replay these interactions across agents, we build a reactive LLM-based user simulator that preserves the original users' intents and provides feedback when the coding agent's progress requires it. To evaluate agents as collaborators, we measure both final repository correctness and the number of corrective feedback turns required during the interaction. Experiments with frontier coding agents show that stronger agents generally achieve higher final success rates while requiring fewer interventions, suggesting an improved user experience.