COCORELI: Enforcing Execution Preconditions for Reliable Collaborative Instruction Following

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom, and with me are Jane, Lu, senior AI researcher at Tsinghua, Meng, lead engineer at a mysterious AI startup and Lalam, the in-house Large Language Model.

Jane: Today's paper: "COCORELI: Enforcing Execution Preconditions for Reliable Collaborative Instruction Following".

Tom: Autonomous agents executing human instructions must operate reliably even when instructions are incomplete, and this reliability requires enforcing missing information as a precondition for action.

Jane: First, who's behind it and why it matters.

Paper summary: Tom: So we’re looking at this paper called "COCORELI: Enforcing Execution Preconditions for Reliable Collaborative Instruction Following". Basically, the main idea is that when autonomous agents follow human instructions, they have to work reliably even if those instructions are incomplete.

Jane: That’s right. The authors argue that just detecting missing information isn't enough anymore. Agents often keep going with execution even after they realize something is missing, which leads to mistakes or unsafe actions instead of stopping and asking for help first.

Lu: They propose a modular architecture called COCORELI to fix this by making sure detection and prevention are structurally linked. It’s not just about spotting the gap; it's about blocking the action until that missing piece is filled in.

Meng: So if an agent sees a null field, instead of guessing what to do, it has to pause and generate a specific question to get that information before moving on? That sounds like a practical way to handle real-world tasks.

Lalam: Exactly. They represent instructions as structured objects where fields start empty, and execution is strictly blocked whenever those required fields are still unresolved. This forces an explicit resolution of missing information before anything happens, preventing implicit guessing.

Tom: It’s about this structural coupling that they claim makes a big difference compared to other methods we’ve seen for handling incomplete tasks. They show this works even when the underlying model gets bigger, which is something we need to keep in mind.

Jane: That's interesting because usually, making models larger just makes them get better at guessing instead of reliably knowing what they don't know. The paper suggests that how the task structure and uncertainty are represented during execution matters more than just raw model capability, according to COCORELI: Enforcing Execution Preconditions for Reliable Collaborative Instruction Following.

Lu: They test this in a controlled environment called ENVIRONMENT which is designed specifically to stress-test things like incomplete task specifications and reconstructing old structures. This environment forces the agents to deal with novel object types and physical constraints that mimic real assembly tasks, demanding they resolve ambiguities before acting.

Paper summary: Meng: From an engineering standpoint, I’m curious about how this structure handles those dynamic environments. If the instruction changes or you encounter a new object type on the fly, how does COCORELI manage that without needing constant retraining?

Lalam: The architecture includes components like an Instruction Parser and a Builder which figure out what the parts are and what their specific properties are, like orientation or configuration. Then there’s the Discourse Module that generates those targeted clarification questions when things aren't clear, which enforces explicit resolution before execution.

Tom: So, the mechanism for handling underspecification is very concrete: you have a system that detects a null field and immediately triggers a specific clarification question to fill it in. It stops guessing entirely.

Jane: It really boils down to making the process of asking for more information an explicit, required step in the workflow rather than something optional that an agent might skip over when it’s rushed.

Lu: The results they show are quite strong, especially for complex structure construction where COCORELI achieved a high overall accuracy of seventy-eight point five seven percent, beating both the CoT baselines and the agentic baseline. Also, in testing abstraction for ToolBench API tasks, it hit one hundred on all three metrics when compared to single-LLM CoT baselines.

Meng: That’s a significant jump from those other methods. If an AI can reliably reuse workflow structures across different tasks without needing task-specific fine-tuning, that opens up a lot of possibilities for building more adaptable systems in the real world.

Lalam: And it does show cost-invariance across different task types, meaning the output size doesn't grow with how complex or large the structure is. That’s good because it keeps resource usage predictable regardless of the complexity of what you’re trying to build.

Tom: So, when we think about what this means for practical AI deployment, it suggests that building reliable systems isn't just about having a bigger brain; it’s about building a better set of rules and structures around how those brains interact with incomplete information.

Paper summary: Jane: It shifts the focus from just improving the model itself to designing the interaction layer so that ambiguity is handled systematically and safely, which I think is really important for any collaborative AI application.

Lu: The authors point out a limitation, though, which is that their setup assumes tasks can always be perfectly represented by structured schemas. Also, they note that this evaluation environment abstracts away things like perception and multimodal grounding.

Meng: That makes sense. If the system relies entirely on having a perfect structural representation beforehand, it might struggle when the input to the agent is messy sensory data instead of clean text instructions.

Lalam: And another point they raise is that their conversational component only models a narrow form of dialogue: just asking for missing task parameters. It doesn't really cover more complex social stuff like negotiation or deep reasoning about intent.

Tom: So, while COCORELI solves the problem of execution errors due to missing inputs, it stops short on modeling the richer, more nuanced human conversations we see in collaborative work today.

Jane: That means COCORELI is excellent at enforcing structural correctness in a known format, but it doesn't necessarily give us a fully social or perfectly flexible dialogue agent yet.

Lu: But the core contribution remains architectural because it enforces that link between detection and prevention, regardless of how big the underlying model is. That’s the main point they’re making about where we should focus our research effort.

Meng: So, for me, what this means practically is that we need to bake this kind of explicit precondition checking directly into our system design from the start, not just add it on as an afterthought when things go wrong.

Lalam: It suggests that reliable agent execution really benefits from having these explicit mechanisms for both asking about missing parameters and using reusable task abstractions. That's the core message of COCORELI: Enforcing Execution Preconditions for Reliable Collaborative Instruction Following.

Conclusion: Tom: So, we've been looking at COCORELI and what it does for agents following human instructions.

Jane: Basically, this paper is about making sure when an AI is doing a complex task with incomplete directions, it doesn't just guess and fail; it has to stop and ask for the missing pieces first.

Lu: It’s a modular architecture designed to link detecting what’s missing directly to blocking the action. They treat instructions like structured objects where every required piece has to be filled in before anything moves forward.

Meng: From an engineering standpoint, that structural coupling is key because it doesn't rely on the underlying model being perfect at guessing; it forces a protocol for getting the right information down.

Lalam: The core contribution here is architectural, not about making the model bigger. It shows that enforcing this structural check works no matter how small or large the brain behind it is.

Tom: So, to wrap up, COCORELI isn't just another model; it's a system built around forcing explicit clarification when information is missing in a workflow.

Jane: Exactly. It moves the focus from just improving raw intelligence to designing the interaction layer so that ambiguity leads to a request for more detail instead of an incorrect action.

Lu: They tested this in environments that simulate real-world assembly, and it held up well, showing good accuracy even when things get messy with incomplete steps.

Meng: It’s interesting how they show cost-invariance across different task types; the system doesn't suddenly blow up in size just because the instruction gets more complicated.

Lalam: The paper suggests that this explicit mechanism for checking missing parameters is necessary if we want agents to be truly reliable collaborators, not just smart guessers.

Tom: It really puts a lot of pressure on us to build these kinds of safety checks into the design from the very beginning.

Jane: And once you have that foundation, it opens up possibilities for building more robust and trustworthy AI systems in real-world settings.

Lu: Which brings us to how this approach compares to other ways of handling uncertainty in instruction following.