Cochise: A Reference Harness for Autonomous Penetration Testing

Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.

Nadia: Today's paper: "Cochise: A Reference Harness for Autonomous Penetration Testing".

Elias: The gist The Cochise prototype is a minimal reference agent and harness designed to provide an execution interface, model abstraction, state handling,

Nadia: First, who's behind it and why it matters.

Title and authors: Nadia: So we're talking about "Cochise: A Reference Harness for Autonomous Penetration Testing". It’s a reference implementation, a six hundred thirty lines of Python code that lets you test agents against real target environments like Game of Active Directory <ref:2605.11671#pg1>.

Elias: That's the key part—it’s designed to be this reusable infrastructure. They aren't claiming it’s the best agent ever; they are positioning it as a way for researchers to compare different models and architectures fairly under one common protocol.

Priya: I see why that matters, because if you can swap out the LLM or change the planning logic easily without rewriting everything else, you can really isolate what part of the agent is doing what.

Nadia: Right. They connect this harness to a Linux execution host using SSH and let it run commands inside a controlled test environment reachable from that jump host. It’s about making the connection between the LLM brain and the actual attack surface very structured.

Elias: And they introduce this Planner-Executor architecture, where the planner keeps track of long-term state in a structured text representation, while the executor translates those high-level goals into concrete commands over SSH.

Priya: That decomposition is interesting because it addresses that multi-step nature of attacks—the agent isn't just guessing one thing; it’s building a path and then executing the steps along that path.

The paper's summary: Nadia: The paper summarizes Cochise as a way to handle the complexity of autonomous penetration testing by separating the strategic planning from the low-level execution. The planner handles the long-term state, and it feeds directives down to an executor that uses a ReAct style agent to issue commands over SSH.

Elias: They emphasize that this design tackles several hard problems inherent in security testing environments, like partial observability, where you don't see everything at once, and side-effecting actions where an exploit might crash the target system or trigger intrusion detection systems.

Priya: What I find important is how they handle those side effects; they force the agent to deal with them by needing autonomous error recovery and combining findings into successful attack paths over multiple steps. It’s not just about finding one vulnerability; it’s about chaining them together.

Nadia: Right. And that leads into their logging system, where every interaction—planner to executor, LLM calls, target network actions—is recorded in a JSON log file for reproducibility and analysis later on.

Elias: They detail how they log LLM invocations with specific event keys that name the architectural action, along with the exact prompt and completion received. And commands executed get their own identifiers showing the raw bash string run, along with the output streams.

Priya: So it’s not just a run log; it’s a rich dataset that lets you analyze things like cost and token usage for each step of the process, which is crucial for understanding how efficient those agents are.

The paper's improvements: Nadia: The authors point out several specific improvements they made to this baseline harness. They focused on making the executor state management very clean; they ensure a new executor instance is created for each task and then discarded once it’s finished.

Elias: That ephemeral state management is key because it bounds the context window size and the cost for any single task, which otherwise would grow indefinitely if you kept one giant context running for hours of operation.

Priya: That makes sense; if you don't bound the executor's memory, transient failures from one step could contaminate the reasoning for a completely different later step in the attack path. It keeps things focused and manageable.

Nadia: They also highlight that this structure forces the planner to become the integration point for cross-task knowledge, meaning it has to manage what happened in task one before it can properly plan task two.

Elias: That’s a structural change that makes the planner's job much harder and more meaningful than just letting the LLM handle everything sequentially without a high-level state manager guiding the flow.

Priya: And they mention using the Reflexion pattern within both components to actively detect and then repair invalid command invocations, which is another layer of self-correction built into the harness structure itself.

Conclusion: Nadia: So, to wrap up on "Cochise: A Reference Harness for Autonomous Penetration Testing", they’ve given us a minimal but functional system that structures agent behavior by separating planning and execution. It’s a six hundred thirty-line reference implementation connecting an LLM to a testbed like GOAD <ref:2605.11671#pg1>.

Elias: They stress that the real value isn't just the code itself, but how it forces you to think about penetration testing as a software engineering problem, handling all those stateful, multi-step issues with explicit planning and controlled execution.

Priya: The implication for me is that this tool gives researchers a solid foundation. It’s not trying to be the final agent; it’s providing the infrastructure so other people can build on top of it to compare different models and architectural variants systematically.

Nadia: Exactly. And they provide tools like cochise-replay and analysis scripts so you can actually look at the trajectories, analyze the cost, and see how it performed against a live testbed. It’s an experimental infrastructure for comparing things.

Elias: So, ultimately, this paper is about providing a standard way to run autonomous testing experiments safely and repeatably so we can properly evaluate different approaches to agent design.

Priya: It sets up the necessary framework so that the next step isn't just building another agent from scratch, but building an agent using this harness as its foundation.

Nadia: That’s it for Cochise. We’ll leave you with this reference infrastructure for autonomous penetration testing, built on a planner-executor design.