DIRECT: When and Where Should You Allocate Test-Time Compute in Embodied Planners?

summary

Video file (mp4)

The gist

Detailed Research Summary of DIRECT: Dynamic Compute Allocation for Vision-Language Models in Embodied Planning As a diligent researcher, I have thoroughly analyzed the provided excerpts from the

In short

Vision-Language Models used for robot planning often waste compute by scaling test-time resources uniformly. DIRECT introduces a dynamic routing framework that uses scene context to decide which specific planner to use for each subtask. This approach shows that allocating compute based on task needs—rather than blindly increasing it—improves performance while significantly reducing latency and cost.

Key concepts

Test-Time Compute Scaling
This is the practice of increasing the computational resources (like FLOPs or token usage) used by a model during inference to try and improve its output. The paper argues this scaling is often inefficient, leading to high latency without guaranteeing better results.
Dynamic Routing Framework (DIRECT)
DIRECT is a system that uses information from the scene and instructions to dynamically select the best planning model for every single step of an agent's plan. It acts as a smart traffic controller, sending tasks only to the planner whose specific skills match what is needed at that moment.
Routing Efficiency Score ($\eta(\beta)$)
This metric measures how well DIRECT allocates compute. It balances two competing goals: achieving a high task success rate ($q_s$) while keeping the latency cost ($c_s$) low. A higher score means the system is making smarter, more efficient choices about which planner to use.

Terminology used across episodes

This episode discusses

The paper

DIRECT: When and Where Should You Allocate Test-Time Compute in Embodied Planners? · Read on arXiv

Jadelynn Dao, Milan Ganai, Yasmina Abukhadra, Ajay Sridhar, Mozhgan Nasr Azadani, Katie Luo, Clark Barrett, Jiajun Wu, Chelsea Finn

Stanford University · University of Waterloo

Vision-Language Models (VLMs) are increasingly deployed as high-level planners for embodied agents, with an emerging strategy of scaling test-time compute to improve capability. However, we observe that doing so increases latency, token usage, and FLOPs while yielding uneven, often diminishing gains in downstream success, limiting where embodied agents can be deployed. We argue that choosing when and where to spend test-time compute is central to bringing frontier performance to the real world, and we can budget inference at the planning level. We introduce DIRECT, a routing framework that uses multimodal scene context to allocate compute at each step of a closed-loop agent, selecting a planner for every subtask, improving the success--cost Pareto frontier over fixed model selection. Across three dominant scaling axes, namely chain-of-thought depth, model size, and memory history, our experiments on VLABench and RoboMME show that test-time compute is not a uniform lever: different axes yield qualitatively distinct capability gains. We validate these insights on a physical Franka arm in a DROID setup spanning zero-shot manipulation and long-horizon chaining, where our router matches or exceeds a stronger model's success rate at up to 65% lower average latency. Ultimately, our results show that naively scaling test-time compute is wasteful, and DIRECT can provide frontier-level embodied planning at a fraction of the cost. Project page can be found at jadee-dao.github.io/direct/.

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: I'm Rosa.

Dev: And I'm Dev, here with Taro.

Rosa: Today's paper: "DIRECT: When and Where Should You Allocate Test-Time Compute in Embodied Planners?".

Dev: Detailed Research Summary of DIRECT: Dynamic Compute Allocation for Vision-Language Models in Embodied Planning As a diligent researcher,

Rosa: First, who's behind it and why it matters.

Title and authors: Rosa: We've seen that DIRECT is about dynamically allocating compute per prompt instead of picking a fixed model, and now we need to look at the mathematical structure behind that decision-making process.

Dev: Right, they formalize it with this utility function notation where k star i equals arg max over k U of q i k c i k. It basically says for every task demand q i inferred from context c i you want to pick the planner k that maximizes that utility function U of the task, planner, and context.

Taro: So what does that actually mean when we're talking about matching capability profiles? How do they define those candidate planners and how does the utility function really capture reasoning depth versus model size versus memory history?

Rosa: The paper states that this allows DIRECT to be agnostic to the specific form of the utility function, meaning it doesn't care if you’re prioritizing one thing over another as long as you match capability profiles to inferred needs.

Dev: It's about matching what the task requires—like deeper reasoning or longer-horizon memory—to what each planner inherently provides in terms of compute configuration.

Taro: So, if a task demands deep spatial reasoning, the router should infer that and route it toward a planner that handles those subtle physical constraints well, even if it's slightly more expensive than a simpler model.

Rosa: Precisely. It’s about the inference of task demands from scene and instruction context driving which planner gets invoked for that specific query.

Dev: And they mention that this whole routing framework is trained offline over a fixed pool of candidate planners, so the router itself doesn't have a huge inference cost at all during deployment.

Taro: That’s good for real-time loops because we don't want the router adding another significant latency spike to our control loop.

Rosa: It keeps the decision-making process fast while ensuring that you get that quality-cost tradeoff we talked about in the first place.

The paper's summary: Dev: Now let's look at how they summarized their findings in "DIRECT: When and Where Should You Allocate Test-Time Compute in Embodied Planners?"

Rosa: They summarize that test-time compute is not free in robotics because it compounds across a hierarchical stack, so costs from chain-of-thought, larger models, or denser memory can multiply the base model latencies.

Dev: And they showed that cutting compute uniformly lowers latency but risks failure on harder tasks, so the solution is to study where compute actually yields gains across three axes: reasoning depth, model size, and memory history.

Taro: So they are looking at those three specific areas because that’s where the real trade-off happens in embodied planning compared to just trying to cut everything equally.

Rosa: They examine chain-of-thought reasoning on VLABench and find that high-level planners with deliberate thinking chains can reason through subtle physical, semantic, and spatial constraints, but this reasoning comes at a heavy cost in token generation latency.

Dev: And they also look at model size and find that model size governs the breadth of skills a planner can reliably command; larger models offer greater breadth but introduce unnecessary overhead for simpler tasks.

Taro: So it’s not just about which model is biggest, but whether that extra size is actually needed for the specific skill command required by the task.

Rosa: They also looked at memory context and found that memory-oriented compute is only valuable when a task genuinely depends on distant history, otherwise paying for recall overhead across all tasks is wasteful.

Dev: So the summary boils down to this: effective allocation requires routing policies that infer task demands from scene and instruction context and match them to the compute configuration with the best quality–cost tradeoff.

Taro: That really frames it as an intelligent way to navigate the success–cost frontier instead of blindly scaling up compute.

Rosa: It means we move away from a one-size-fits-all approach to test-time compute allocation in embodied planning and toward something context aware.

The paper's improvements: Dev: Moving on to the proposed improvements in "DIRECT: When and Where Should You Allocate Test-Time Compute in Embodied Planners?"

Rosa: One main improvement they suggest is deploying DIRECT as a routing framework that uses multimodal scene context to allocate compute per prompt, which is designed to improve the success–cost Pareto frontier over fixed model selection.

Dev: And they show that this dynamic allocation allows embodied agents to match or exceed a stronger model’s success rate at up to sixty-five percent lower average latency.

Taro: That sixty-five percent reduction in latency is what makes it really compelling for real-world deployment because it shows we can get frontier performance without the usual massive compute penalty.

Rosa: They also suggest improving task performance by implementing per-step allocation on long-horizon tasks, which means DIRECT decides at each step whether routing to the thinking model is worth its cost.

Dev: And that per-step decision making showed highest progress success in physical validation results and recovers the Thinking model’s success at much lower latency for those critical steps.

Taro: So they aren't just allocating compute once, but deciding dynamically at each step whether the cost of routing to a specific planner is justified for that particular step.

Rosa: And regarding skill command across varied capabilities, they enable dynamic scaling based on task demands, which means model size primarily governs the breadth of skills a planner can reliably command.

Dev: So instead of deploying the largest models everywhere, you can reserve them only for tasks that explicitly demand that increased capacity for complex skill commands.

Taro: That lets you have a system commanded on more skills without having to deploy every single model at its maximum potential capability all the time.

Rosa: And finally, they improve memory utilization for long-horizon tasks by employing task-aware memory architectures, which identifies that memory-oriented compute is only worth its cost when a task genuinely depends on distant history.

Dev: So instead of paying the overhead on every single prompt for historical context, you can allocate it selectively based on whether the task actually requires that distant history.

Conclusion: Rosa: So to wrap up, this paper "DIRECT: When and Where Should You Allocate Test-Time Compute in Embodied Planners?" suggests that naively scaling test-time compute is fundamentally wasteful.

Dev: It provides a principled path to achieving frontier-level embodied planning performance at a significantly reduced computational cost by treating the high-level planner as a dynamic variable inferred from scene and instruction context.

Taro: The implication for us is that we should stop thinking about just scaling up compute and start thinking about matching capability profiles to task demands.

Rosa: It means we can achieve much better performance, matching the strongest model at up to sixty-five percent lower latency while keeping the system viable for real-world deployment.

Dev: The key is using DIRECT to intelligently distribute test-time compute based on inferred demands from scene and instruction context so we can navigate the success–cost Pareto frontier effectively.

Taro: It’s about being adaptive, knowing when to spend more compute and when to save it for the next step.

Rosa: So that's what DIRECT is all about: a dynamic inference router for embodied planning that makes sense of where we should allocate our test-time compute.

More episodes

← Home