Hierarchical Planning with Latent World Models

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom, and with me are Jane, Lu, senior AI researcher at Tsinghua, Meng, lead engineer at a mysterious AI startup and Lalam, the in-house Large Language Model.

Jane: Today's paper: "Hierarchical Planning with Latent World Models".

Tom: The gist The Hierarchical Planning with Latent World Models (HWM) framework enables zero-shot, non-greedy manipulation on a real robot from a single goal image, where single-level world-model planners fail.

Jane: First, who's behind it and why it matters.

Title and authors: Tom: We've looked at the title and authors of "Hierarchical Planning with Latent World Models." It’s a mouthful, but it sums up the core idea: using hierarchy to plan with latent world models.

Jane: The authors are Zhang, Terver, Zholus, Chitnis, Sutaria, Assran, Balestriero—they bring together expertise from Meta and NYU and other places.

Lu: What this means in simple terms is that they are proposing a new way to structure the planning process itself.

Meng: Instead of one giant planner trying to figure out everything at once across a long sequence of steps, they break it down into two or more specialized planners working together.

Lalam: It’s like having a manager who sets big, abstract goals while a worker handles the fine details of getting there.

Tom: That's exactly right, Jane. The core idea is decomposition. They are taking the complexity of long-horizon tasks and splitting it up into shorter subproblems that are easier to solve individually.

Jane: And they do this all within a shared latent space, which is important because it keeps the information consistent across both levels of planning.

Lu: The paper introduces this specific coupling mechanism where predictions from the coarser model feed as subgoals for the finer model to optimize primitive actions in latent space.

Meng: So the high-level planner figures out a macro-action, and that prediction acts as a target point for the low-level planner to hit with actual movement commands.

Lalam: It’s about making sure the high-level plan is actually achievable by ensuring that there's a detailed, low-level plan matching its latent prediction.

Tom: It’s about creating this feedback loop where both levels replan as new observations come in, which is crucial for real robot control.

Jane: So, when you look at the overall structure of "Hierarchical Planning with Latent World Models," it’s a system designed to manage complexity by layering planning decisions.

The paper's summary: Tom: We're now going over the actual summary of "Hierarchical Planning with Latent World Models." This paper outlines the architecture and the specific planning paradigm they are using.

Jane: They describe how they train these world models solely via next-latent prediction, which is a key part of their training strategy.

Lu: The training involves learning world models at multiple temporal resolutions within that shared latent space, so they have different levels of detail available.

Meng: This means the low-level model F(one) predicts the very next state based on current state and action, while the high-level model F(two)phi predicts dynamics over longer stretches <ref:2604.03208#pg1>.

Lalam: And they use a learned action encoder to compress those long sequences of primitive actions into these latent macro-actions for the high-level planner.

Tom: The planning procedure itself is hierarchical: a high-level planner optimizes abstract macro-actions using the long-horizon world model to reach the final goal embedding.

Jane: The output of that high level plan, which is the first predicted latent state from the rollout, serves as the subgoal for a low-level planner.

Lu: That low-level planner then optimizes primitive actions to reach that specific subgoal using its short-horizon world model.

Meng: The energy functions they define are E2 for the high level and E1 for the low level, which guide each planner toward its respective objective.

Lalam: This structure is what allows them to achieve things like seventy percent success on Franka pick-and-place from a single goal image, compared to zero percent for VJEPA2-AC one <ref:2604.03208#pg1>.

Tom: The paper also emphasizes that this entire framework works in a closed-loop MPC fashion, meaning it replans every k interactions using the hierarchical planner.

Jane: So, the core summary is this: training multiple temporal models at different scales, coupling them via latent subgoal matching to allow for two levels of planning that work together.

The paper's improvements: Tom: Let’s talk about what the authors claim are the actual improvements they’ve achieved with this framework. They aren't just presenting a new idea, they are showing results compared to existing methods.

Jane: The main improvement is that HWM solves pick-and-place from a single goal image at seventy percent success where prior single-level planning failed completely, achieving zero percent on VJEPA2-AC.

Lu: They also show consistent gains across three latent world model backbones—VJEPA2-AC for manipulation, DINO-WM for push manipulation, and PLDM for maze navigation.

Meng: And these gains are significant; they show that hierarchical planning improves both performance by up to plus forty percent and efficiency by up to plus thirty-nine percent absolute improvement on long-horizon tasks.

Tom: The efficiency gain is really notable here because it shows that this structure helps reduce search complexity, requiring up to three times less planning compute under a fixed planning budget.

Jane: That computational saving means we can actually run these more complex plans on hardware that might otherwise struggle with the computation needed for single-level planners.

Lu: The analysis of the components is interesting because it shows that manual subgoals isolate planning from representation, and HWM automatically recovers most of that oracle performance, getting seventy percent for pick-and-place from a single goal image.

Meng: They also found that the learned macro-actions they produce are consistently better than both concatenating primitive actions and hand-crafted summaries like net end-effector displacement.

Lalam: The paper suggests that using a macro-action dimension of four seems to strike the right balance for tasks like Franka tasks, which is useful information for how we structure these latent summaries.

Conclusion: Tom: We’re wrapping up with the conclusion of "Hierarchical Planning with Latent World Models." This paper sets a lot of expectations for what's possible in embodied AI.

Jane: Essentially, HWM is presented as the first world-model planner to demonstrate zero-shot, non-greedy real-robot manipulation from pixels using only a single goal image.

Lu: The authors suggest that hierarchy is a practical mechanism for extending latent world-model MPC beyond short, greedy behaviors by coupling multiple latent world models through shared latent subgoal matching.

Meng: It’s about enabling zero-shot planning with learned world models trained from offline trajectories, which is a big step for making these systems more general.

Lalam: This framework opens up possibilities where the AI can plan much more robustly than before because of this hierarchical structure, especially when dealing with long sequences of observations.

Tom: The results show that even though success rates degrade with task horizon for every method they evaluate, HWM still keeps performing better on those longer tasks compared to single-level planners.

Jane: So, the big implication is that for anyone working on embodied control, hierarchy seems like a practical way to move beyond simple greedy behaviors by managing the planning complexity more effectively.

Lu: The paper shows consistent gains across VJEPA2-AC, DINO-WM, and PLDM backbones across different task suites, proving that the hierarchical approach is robust across different types of tasks.

Meng: We’re just waiting to see how much this translates into real deployment on a robot that isn't in a perfect simulation environment.

Lalam: It’s an interesting piece of research because it shows how structure in the AI system can directly translate into better performance on the actual physical world.