BASIS: Evaluating Belief-Aware Solutions in Interactive Settings beyond Instructions

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: Today's paper: "BASIS: Evaluating Belief-Aware Solutions in Interactive Settings beyond Instructions".

Jane: The gist The work formalizes Theory of Mind for Large Language Models as a mechanism for epistemic divergence detection and resolution,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: Alright team, we're looking at the paper "BASIS: Evaluating Belief-Aware Solutions in Interactive Settings beyond Instructions." The core idea here is that just following instructions isn't enough for agents anymore because users often misunderstand what they actually need or what the current situation is.

Jane: So, the paper introduces this concept of Theory of Mind, or ToM, not just as a theoretical thing models might have, but as a way for an AI to detect and fix when the user's understanding doesn't match reality.

Lu: Exactly. They formalize ToM as a mechanism for detecting and resolving epistemic divergence between what the user believes and what is actually happening in the environment, which is important because previous work often just focuses on inferring those mental states in isolation > Page one of that work reads: "Resolving this epistemic divergence requires Theory of Mind (ToM), yet existing ToM evaluations for LLMs primarily focus on isolated belief inference, overlooking its functional utility in real-world interaction <ref:2602.13832#pg1,Resolving this epistemic divergence requires Theory of Mind (ToM), yet existing ToM>."

Meng: Functional utility is what matters to me. So, the paper claims they're building a benchmark called SynchToM to see if agents can actually use this understanding to solve problems better when interacting with users > Page one of that work reads: "To this end, we formalize ToM for LLMs as a mechanism for epistemic divergence detection and resolution, and propose a benchmark, SynchToM, to assess how models reconcile user beliefs and profiles in practice <ref:2602.13832#pg1,To this end, we formalize ToM for LLMs as a mechanism for>."

Lalam: So if we put it simply, they want to see if the AI can figure out what the user *thinks* versus what's actually true so it can help them succeed > Page one of that work reads: "The agent constructs an inferred belief ˆb by maximizing the posterior P(b o, g) = P(τ o, g, b) · P(b o, g)" <ref:2602.13832#pg1>

Tom: Right. And they aren't just asking explicit questions to see if the AI knows what the user thinks; they are evaluating how useful that inference is for actually solving the task > Page two of that work reads: "Rather than assessing mental-state inference solely through explicit questions, BASIS evaluates both the utility of such inferences and the quality of downstream solutions in user-agent interactions <ref:2602.13832#pg1>."

Jane: And their findings suggest that current models often fail to account for users' mental states when they are generating solutions, and that solution quality is consistently higher when this information is explicitly given during problem solving > Page two of that work reads: "Evaluation results show that current models often fail to account for users' mental states when generating solutions, and that solution quality is consistently higher when such information is explicitly elicited and made available during problem solving <ref:2602.13832#pg1>."

Paper summary: Lu: They also looked at how they could make this more functional by comparing it to standard Chain-of-Thought prompting, controlled interventions on the mental state info, targeted training, and even multi-agent collaboration experiments under partial observability > Page three of that work reads: "We further investigate the practical value of mental-state consideration through comparisons with generic Chain-of-Thought prompting, controlled interventions on mental state information, targeted training, and multi-agent collaboration experiments under partial observability <ref:2602.13832#pg3>."

Meng: From a production standpoint, I'm interested in how they tie this to actual deployment. They show that trajectory-based reinforcement learning and a training-free multi-agent framework can improve the functional ToM part > Page four of that work reads: "Reinforcement Learning with Trajectory Data involves inserting specific tokens, “…” and “…”, for guiding models to focus on the cognitive gap between user belief and the true state"

Lalam: So, a key finding is that these ToM reasoning chains are transferable across different models, meaning one model's good mental state inference can guide another model in a multi-agent setup > Page four of that work reads: "These findings confirm that ToM reasoning chains are transferable across models, and a high-quality mental state inferred by one model can serve as an guidance for another to enable multi-agent collaboration for downstream tasks."

Tom: The overall message from BASIS is that it's not just about inferring someone's thoughts; it's about using those inferences to provide assistance that actually hits the mark, especially when the user’s beliefs are shaky > Page one of that work reads: "To bridge this gap, we further curate a trajectory-based ToM dataset linking belief" <ref:2602.13832#pg1,To bridge this gap, we further curate a trajectory-based ToM dataset>

Jane: It really frames user-agent interaction as a problem of calibrating beliefs against reality, which is a concrete way to look at it > Page three of that work reads: "In contrast, our work frames user-agent interaction as a belief calibration problem, grounding ToM in interaction trajectories not only to infer and align with internal user beliefs but also resolve these conflicts <ref:2602.13832#pg3,In contrast, our work frames user-agent interaction as a belief calibration>."

Lu: The paper sets up this framework by simulating scenarios where the user has limited observations and the agent needs to navigate those partial observations to achieve a goal > Page three of that work reads: "We simulate interaction scenarios where the local observations of a user provide incomplete view of the environment (e <ref:2602.13832#pg3,We simulate interaction scenarios where the local observations of a user provide>.g., un unseen system configuration or a hidden error)."

Meng: From my side, what I'm worried about is the cost and complexity of running this kind of deep mental state modeling at scale; they haven't really shown how much it adds to the inference latency > Page four of that work reads: "The final reward computes as r = λformatrformat + λbrb + λprp + λsolrsol, where we set λformat=zero point one, and... lambda sol=zero point four"

Paper summary: Lalam: So they are showing that accurate epistemic alignment is a critical prerequisite for resolution, especially in environments where information is sparse > Page two of that work reads: "Results across eleven leading models reveal a significant limitation to identify underlying cognitive gaps that impede task success <ref:2602.13832#pg1,Results across 11 leading models reveal a significant limitation to identify underlying>."

Tom: I think the title itself says it all—BASIS Evaluating Belief-Aware Solutions in Interactive Settings beyond Instructions. It’s about moving past just following commands to actually understanding the user's internal world > Page one of that work reads: "Beyond Words: Evaluating and Bridging Epistemic Divergence in User-Agent Interaction via Theory of Mind Minyuan Ruan * one Ziyue Wang * one Kaiming Liu one Yunghwei Lai one Peng Li two Yang Liu one" <ref:2602.13832#pg1,Beyond Words: Evaluating and Bridging Epistemic Divergence in User-Agent Interaction via>

Jane: It means that for AI to be truly helpful, it needs to be able to recognize when the user's input is based on a flawed mental model of the world and correct its own path > Page three of that work reads: "By reconciling latent user beliefs with factual environment states, the agent can guide users toward correct resolutions and prevent the persistence of erroneous mental states <ref:2602.13832#pg3>."

Lu: The implication for future work seems to be that functional ToM is a measurable component we need to focus on developing, not just theoretical reasoning > Page two of that work reads: "Moreover, much of this work conceptualizes ToM as an isolated" <ref:2602.13832#pg1>

Meng: So if you're building a system today, the paper says you need to consider how user beliefs might be wrong and design a mechanism for correcting that mismatch > Page four of that work reads: "The study indicates that functional ToM is a critical, measurable component for the development of more collaborative and resilient models and AI agents."

Lalam: The work establishes this robust ToM-resolution linkage, showing that attributing the right mental state is a causal driver for task success in environments with limited history > Page one of that work reads: "The study indicates that functional ToM is a critical, measurable component for the development of more collaborative and resilient models and AI agents <ref:2602.13832#pg1>."

Tom: So we've seen how they formalize it, what the benchmark looks like, and why it matters for actual problem solving in interaction. That’s where we’ll end this part of our discussion > Page one of that work reads: "Beyond Words: Evaluating and Bridging Epistemic Divergence in User-Agent Interaction via Theory of Mind Minyuan Ruan * one Ziyue Wang * one Kaiming Liu one Yunghwei Lai one Peng Li two Yang Liu one" <ref:2602.13832#pg1,Beyond Words: Evaluating and Bridging Epistemic Divergence in User-Agent Interaction via>

Conclusion: Tom: So we’ve looked at how this paper, BASIS, uses Theory of Mind to make AI agents better at following user requests when users are actually confused or misinformed about the task state.

Jane: Right, so instead of just checking if an AI followed instructions perfectly, they’re testing if the AI can figure out what the user *thinks* versus what’s actually happening in the world.

Lu: The authors argue that this isn't just a theoretical exercise; it’s about creating a real benchmark called SynchToM to see if models can actually reconcile those differences in practice.

Meng: I wonder how much extra computation this mental state tracking adds to the runtime when you’re deploying these things at scale.

Lalam: The core finding is that accurate alignment between what the user believes and reality is a critical requirement for successful task resolution, especially when things are unclear.

Tom: They show that current models often fail to account for these user mental states during solution generation, and the quality of those solutions goes way up when you explicitly give the AI that belief information during the solving process.

Jane: So what does this mean for us? It suggests we can’t just rely on generic instruction following anymore; we need mechanisms that detect when a user's understanding is flawed and try to correct it.

Lu: The authors also looked at ways to make this functional, proposing methods like trajectory-grounded reinforcement learning and multi-agent collaboration frameworks where one model's good belief inference can guide another model.

Tom: That transferability of reasoning chains across different models sounds interesting from a mechanical standpoint; if one agent learns how to fix a belief gap, that knowledge can help another agent solve a different problem.

Jane: It really shifts the focus from just building smarter models to building agents that are more collaborative and resilient in messy human interactions.

Lalam: In short, functional Theory of Mind is now seen as a measurable component for making AI assistants genuinely helpful in complex settings where information is sparse.