Ego2World: Compiling Egocentric Cooking Videos into Executable Worlds for Belief-State Planning

arXiv:2605.13335 · cs.AI, cs.CV · Submitted 2026-05-13 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Ego2World: Compiling Egocentric Cooking Videos into Executable Worlds for Belief-State Planning".

Jane: Embodied agents in household environments must plan under partial observation, requiring them to remember objects, track state changes, and recover from action failures.

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So, to recap, we're looking at "Ego2World: Compiling Egocentric Cooking Videos into Executable Worlds for Belief-State Planning," which is fundamentally about building a benchmark that forces agents to handle partial observation in cooking scenarios by using real videos.

Jane: Exactly; the title highlights the main achievement: compiling those dense egocentric cooking videos into symbolic worlds where agents have to remember things and track changes without seeing everything at once. It’s about testing if an agent can maintain a useful belief state over time when it's not seeing the whole picture.

Lu: What strikes me is that they are using HD-EPIC annotations as the source material, which means the environment isn't just synthetic; it has actual cooking activity baked into its rules. That grounding is important for testing real-world applicability.

Meng: I wonder how much effort goes into transforming those dense narrations from those videos into executable actions and rules; that compilation pipeline must be quite involved to ensure it's usable.

Lalam: It makes me think about how we structure our internal models; if an agent can actually separate its current perception from the underlying reality, that structural separation could help us design more robust belief maintenance systems for complex tasks.

The paper's summary: Tom: Moving on to what they actually did in "Ego2World: Compiling Egocentric Cooking Videos into Executable Worlds for Belief-State Planning," the paper explains that it builds these symbolic worlds using a specific process. They take the dense narrations from HD-EPIC and normalize them into primitive actions and semantically coherent action groups.

Jane: That normalization step is key because it translates those very fine-grained video descriptions into a set of one hundred fifty-five executable action types, which makes the environment runnable for an agent to interact with directly. It’s not just watching a video anymore; it's interacting within a symbolic structure.

Lu: The compilation process then moves to deriving reusable transition rules from these annotations, which they use to build a hidden world graph, Gwt. This graph dictates how the environment actually changes based on the actions taken by the agent.

Meng: So, they're essentially creating a simulator where the hidden state is controlled by those video annotations, and that's what we need to look at for practical implementation; it’s less about just looking at data and more about building a runnable system.

Lalam: The paper emphasizes that this structure allows the simulator to maintain Gwt while the agent plans over its own separate belief graph, Gbt, which is built only from what the agent observes locally and gets feedback on.

The paper's improvements: Tom: Now, let's talk about what they found as improvements in their approach to belief-state planning within "Ego2World: Compiling Egocentric Cooking Videos into Executable Worlds for Belief-State Planning." They focused heavily on how the agent should plan and how it recovers from mistakes.

Jane: The main suggestion is that the agent plans by selecting an executable skill conditioned on its current belief, which means it uses its own belief graph to decide what to do next. Crucially, they introduce specific conditions for replanning when execution fails or when a required piece of information becomes too old in the agent's memory.

Lu: The diagnostic evaluation suite they use is pretty clever because it specifically checks things like whether planner backbones can be compared under the same executable protocol, and if action overlap actually aligns with final physical-state success, which is a really nuanced check.

Meng: That diagnostic finding about action overlap overestimating physical state success is something I can't ignore; it tells us that just because an agent *thinks* an action looks plausible doesn't mean the underlying physics or the world rules will actually allow it to succeed.

Lalam: And another improvement they highlight is that persistent belief memory helps with task completion while reducing the need for repeated visual exploration, which points toward a better way to manage long-horizon planning without getting stuck in endless visual searching.

Conclusion: Tom: So, wrapping up on "Ego2World: Compiling Egocentric Cooking Videos into Executable Worlds for Belief-State Planning," the paper shows that by turning real video annotations into executable worlds, we get a much better way to test if agents can handle partial observation in complex tasks.

Jane: They've successfully demonstrated that separating the simulator's hidden state from the agent’s belief state allows us to measure belief maintenance and replanning directly, which is a big step forward for understanding how these systems actually reason.

Lu: The implication for future research is clear: we need benchmarks that jointly score action plausibility, executable grounding, belief maintenance, and final-state correctness so we aren't just looking at one piece of the puzzle.

Meng: From an engineering standpoint, it confirms that memory selection is as important as memory capacity; you can have a huge memory if you don't know which pieces of information to keep or discard efficiently.

Lalam: I think the reliance on source-grounded construction rather than just direct LLM synthesis for building the environment ensures a level of provenance alignment and replayability that makes these results more trustworthy for real-world deployment.

Xi’an Jiaotong University · Nankai University · *A*STAR

cs.AI, cs.CV

Submitted: 2026-05-13

Updated: 2026-10-06

Comments: I have a new version(huge different from previous one),and already put it in arXiv, so I need to withdraw previous version to avoid two articles held on arXiv in the sometime which may cause confusing

Project page: https://sj-li.com/PROJ/Ego2World

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 87/100

The gist: Embodied agents in household environments must plan under partial observation, requiring them to remember objects, track state changes, and recover from action failures.

Key concepts

Gwt (Hidden World Graph)
This is the simulator's secret map of the environment, derived from video annotations. It dictates what actions are physically possible and how the world changes internally. The agent never sees this graph directly; it only infers its state based on observations.
Gbt (Belief Graph)
This represents what the agent actually knows about the world, built solely from partial observations, feedback from actions, and local state changes. It is the information the agent uses to make decisions and plan its next move.
Action Groups
Instead of analyzing every frame in a video, this process aggregates consecutive steps into meaningful 'action groups,' like 'retrieving an object.' These groups are mapped to a set of 155 executable operations, simplifying the complex video data for the planning system.
Replanning Triggers
The agent must replan when it experiences execution failure or when its belief becomes stale (a required piece of information hasn't been seen recently). This mechanism ensures the agent adapts its plan based on real-time feedback and changing circumstances.

Terminology

Summary

Embodied agents in household environments must plan under partial observation, requiring them to remember objects, track state changes, and recover from action failures. This work introduces EGO2WORLD, an executable benchmark that transforms egocentric cooking videos into symbolic worlds governed by graph-transition rules to explicitly test the agent's ability to maintain a useful belief state over a partially observed and ever-changing environment.

How it works

EGO2WORLD functions by compiling real egocentric cooking video annotations from HD-EPIC into executable graph-transition worlds. This process involves several stages:

  1. Normalization: HD-EPIC narrations are temporally dense and often too fine-grained for direct execution, so consecutive steps are aggregated into action groups, each corresponding to an executable operation such as retrieving an object, transferring an ingredient, or producing a symbolic state change. These are mapped into a set of 155 executable action types by mapping over 300 HD-EPIC verb classes.

  2. Compilation: The validated groups are merged and aligned with high-level task structures to form executable task units, where each task is defined by an instruction τ and a goal predicate Φτ.

  3. Rule Derivation: A Video-Compiled Symbolic Simulator (VCSS) maintains a hidden world graph, Gwt, which governs state transitions via a set of world rules R = 1, k=1 to K, rk = (pre(rk), eff(rk)). These rules are extracted from video annotations.

World and Belief Separation

A central design choice is the separation between the simulator's hidden state and the agent's available information. The simulator maintains Gwt, which determines action validity and final success. In contrast, the agent maintains a separate belief graph, Gbt, built only from partial observations, local state-change feedback, and execution feedback. The agent never observes Gwt directly, forcing it to plan over its own belief graph using only local observations and execution feedback.

Agent Planning and Replanning

The agent plans by selecting an executable skill st ∈ S conditioned on its current belief: st = πθ(τ, Gbt). To execute a skill, the simulator checks the preconditions against the hidden world graph Gwt. If preconditions are not met, it returns a failure signal. Replanning is triggered by two conditions: (1) execution failure (ft = FAIL(c)) and (2) stale belief, when a required node has not been observed for more than ∆stale steps. The agent updates its belief graph using the operator U: Gbt+1 = U(Gbt, olt t, ft).

Diagnostic Evaluation Suite

EGO2WORLD employs a diagnostic evaluation suite to target main failure modes of belief-state planning. This suite asks questions such as whether planner backbones can be compared under the same executable protocol, and whether action overlap agrees with final physical-state success. Key findings from these diagnostics include:

action-level overlap scores overestimate physical state success

persistent belief memory improves task completion while reducing repeated visual exploration

Key Contributions and Findings

The main contributions are:

  1. An executable benchmark grounded in real video, compiling HD-EPIC annotations into hidden-world graph-transition environments.

  2. A video-to-world compilation pipeline and hidden-world protocol that separates Gwt from Gbt, making belief maintenance and replanning directly measurable.

  3. A diagnostic evaluation suite showing that action overlap overestimates physical state success, and that belief representations reduce repeated visual exploration.

Furthermore, experiments reveal trade-offs:

memory selection is as important as memory capacity

local action plausibility and final executable outcome should not be conflated

The benchmark exposes capabilities that static action-prediction datasets obscure, suggesting that robust embodied planning requires benchmarks that jointly score action plausibility, executable grounding, belief maintenance, and final-state correctness. The results indicate that belief representations matter more than action vocabulary, and memory selection is crucial for long-horizon performance. The paper concludes by suggesting future research should focus on belief maintenance architectures and co-optimizing planners for both local executability and global task completion. The compilation pipeline's reliance on source-grounded construction, rather than direct LLM synthesis, ensures provenance alignment and executable replayability. (598 words)

The gist

EGO2WORLD is an executable benchmark that compiles egocentric cooking video annotations into hidden-world graph-transition environments for embodied planning under partial observation.

Improvements for AI systems

As a fastidious and diligent researcher, I have analyzed the EGO2WORLD paper to identify concrete, high-leverage improvements for embodied AI systems. The core finding is that current benchmarks fail because they conflate action plausibility with final physical state success, and they do not adequately test for belief maintenance under partial observation.

Here are the specific improvements and what a system employing these principles can achieve:

)Specific Improvements & System Capabilities:

  1. Shift from Action-Prediction to Belief Maintenance Architectures

  2. Implement Explicit World/Belief Graph Separation in Planning Loops

  3. Develop Uncertainty-Aware, Contextual Memory Selection Mechanisms

  4. Integrate Diagnostic Replanning Triggers Based on Stale Belief and Execution Failure

)What the Improved AI System Can Do:

  1. Enhanced Robustness to Partial Observation (Belief Maintenance):

The system can maintain a complex, evolving internal model of the environment's state (the belief graph, Gbt), which is separate from its immediate visual perception or simulator state (Gwt). This allows the agent to reason about unobserved objects, inferred states (e.g., the cup must be full because I just poured water in), and object locations across multiple tasks without needing a perfect, real-time view of the entire scene.

  1. Correcting Plausibility Errors (Separating Action Type from State Success):

The system will stop selecting an action simply because it looks right based on its current belief state. Instead, it must explicitly verify that the chosen action's preconditions are met in the hidden world graph (Gwt). This prevents plausible but incorrect actions (e.g., picking up the wrong cup or placing a knife on an open stove) by forcing a check against executable rules, directly addressing the finding that action overlap overestimates physical-state success.

  1. Adaptive and Efficient Memory Management:

The system will move beyond simple capacity limits to employ sophisticated memory selection strategies. It can dynamically prioritize updating or retaining belief nodes based on their metadata (source, confidence, last observed step). This allows the agent to discard stale or low-confidence memories while preserving critical state information necessary for long-horizon task completion—directly addressing the finding that memory selection is as important as memory capacity.

  1. Automated and Targeted Replanning:

The system will implement a two-pronged replanning trigger:

a) Triggered by explicit execution failure (simulator returns FAIL(c)), where the agent uses the failed precondition to hypothesize a corrected state update and generate a repair action.

b) Triggered by stale belief (when a required object hasn't been observed for too long), which automatically initiates a memory refresh skill (e.g., navigate to) before attempting to plan again, ensuring the agent doesn't act on outdated information.

  1. Improved Efficiency via Targeted Visual Querying:

When planning requires grounding an action but the belief is insufficient (confidence < 0.6), the system will trigger a highly targeted VLM query based on a localized anchor selection protocol. This ensures visual exploration is used precisely to fill specific belief gaps, rather than being a general, costly search across the entire scene, as demonstrated by the VLM Anchor Discovery Protocol.

Sources

Related papers