RMBench: Memory-Dependent Robotic Manipulation Benchmark with Insights into Policy Design

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: I'm Rosa.

Dev: And I'm Dev, here with Taro.

Rosa: Today's paper: "RMBench: Memory-Dependent Robotic Manipulation Benchmark with Insights into Policy Design".

Dev: The gist: This paper introduces RMBench, a new simulation benchmark for memory-dependent robotic manipulation, and Mem-0,

Rosa: First, who's behind it and why it matters.

Paper summary: Rosa: So we're talking about this paper called RMBench, and the main idea is that most robotic manipulation policies don't actually think about memory at all. Dev Right. They struggle with tasks that need to remember things from way back, or keep track of stuff over a long time, which is super common in real life manipulation scenarios.

Taro: So it sounds like they are trying to figure out how much memory a robot actually needs for a task?

Rosa: Exactly. The paper introduces Task Memory Complexity, or TMC, which is this number that tells you the smallest integer m such that an optimal policy only needs to look at its memory state M(m)t. Dev It’s a systematic way to classify tasks based on how much past info the robot needs.

Taro: So if it's M(one), it means it only needs very recent observations, right <ref:2603.01229#pg1>? Like, what's immediately happening now.

Rosa: That’s correct for M(one) tasks like Observe and Pick Up or Rearrange Blocks Dev But then they have M(n) tasks, which are much harder because they require remembering information from much further back in time <ref:2603.01229#pg1>.

Taro: So the paper builds this RMBench benchmark on the RoboTwin two point zero platform, and it has nine tasks spanning different levels of that complexity <ref:2603.01229#pg1>?

Rosa: Yep. There are five M(one) tasks and four M(n) tasks, covering things like Battery Try or Press Button Dev This helps them systematically evaluate what policies can actually handle at different memory needs <ref:2603.01229#pg1>.

Taro: And they also propose a policy called Mem-zero which is modular with explicit memory components <ref:2603.01229#pg2>?

Dev: That's the second big part of the paper. Mem-zero has a dual system architecture with this task-phase classifier that explicitly separates different stages of a task Rosa This allows for structured memory usage over long horizons <ref:2603.01229#pg2,task-phase classifier that explicitly>.

Rosa: So they are trying to give the policy a clear structure for when to use and how to manage its memory?

Dev: Right. The planning module in Mem-zero uses a vision language model conditioned on observations and that structured memory state, predicting the next subtask based on all of that history Taro This design helps reduce planning invocations to O(N), where N is much smaller than the total time T because it only plans when a subtask ends <ref:2603.01229#pg2>.

Taro: That sounds efficient, but what happens when the world misbehaves? If the classifier messes up, how does Mem-zero handle that failure <ref:2603.01229#pg2>?

Dev: The Subtask End Classifier is key for connecting the planning and execution modules because it only triggers high-level reasoning when necessary to reduce inference cost Rosa But they found that failures in those M(n) tasks often come from poor classifier performance, like not accurately detecting when an Uncover subtask is finished.

Taro: So even with this structure, there are still things that can go wrong when the system encounters unexpected situations?

Rosa: That's what they found. For example, in the Press Button task, the visual tokens from the vision language model don't have enough detail to capture fine-grained information needed for that classifier Dev They suggest a need for richer representation fusion to improve how both anchor and sliding memory are used.

Dev: And look at their ablation studies on Mem-zero <ref:2603.01229#pg2>. They removed the anchor memory module and saw a substantial drop in success rates across most tasks Rosa It shows that having that explicit retention of task-critical info is really important for execution.

Taro: What about the sliding memory window? Does removing it hurt as much?

Dev: It still degraded performance on most tasks, with success rates falling below even the vanilla model Rosa But on the Swap T task, they actually found that without the sliding setting it performed better than a regular model.

Rosa: So what are the big implications here for us working with real robots?

Dev: The main implication is that explicit memory modeling yields significant improvements on most memory-dependent tasks in RMBench Taro It shows how architectural choices, like having both anchor and sliding components, directly influence how well the policy performs under long-term memory requirements.

Rosa: So it’s not just about throwing more data at the robot; it’s about designing a system that knows what to remember and when to forget Dev The RMBench benchmark gives us a way to test these design choices systematically.

Taro: And for future work, they are looking into better memory representation and fusion mechanisms, as well as incorporating proprioceptive or tactile feedback for the classifier Rosa That would give it non-visual information.

Dev: It seems like the paper gives us a solid framework to start building more robust robotic policies that can actually handle the long-term reasoning required in complex manipulation tasks.

Conclusion: Rosa: So we're wrapping up this look at RMBench and Mem-zero which are these two things: a benchmark and a policy design tool for robotic manipulation that deals with memory.

Dev: Yeah, the title itself is pretty straightforward, it’s about how memory affects robotic manipulation policies. The authors are Tianxing Chen and his team.

Taro: They built this whole setup on the RoboTwin two point zero platform, so we know this is grounded in actual robotic systems rather than just a computer simulation.

Rosa: Exactly, and they showed that existing policies often fail because they don't handle tasks that need remembering things over long periods well.

Dev: They introduce Task Memory Complexity, or TMC, which is this way to measure exactly how much memory a policy actually needs to succeed for a given task.

Taro: So it sorts the tasks into M(one) stuff and M(n) stuff, giving us a clear map of the memory requirements.

Rosa: And they tested their proposed Mem-zero policy against these tasks, showing that it can do much better than the baseline policies we've seen before.

Dev: The results suggest that explicit memory modeling really makes a difference for most of these complex manipulation tasks.

Taro: But they also pointed out where the current system breaks down, especially with how well the classifier handles those high-level task changes in M(n) scenarios.

Rosa: It’s clear that designing a robot policy isn't just about getting good vision input; it’s about making sure the robot knows what to keep track of across time.

Dev: And they show us how different architectural choices, like having both anchor and sliding memory modules, directly change the performance numbers we see in the experiments.

Taro: It gives us a roadmap for where to focus our next research if we want to build robots that handle really long-term reasoning reliably.