Value as Memory: Online Value-Guided Post-training for General Robot Policies

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: Today's paper: "Value as Memory".

Dev: The gist The Value as Memory (VaM) framework addresses challenges in real-world post-training for generalist robot policies by using value caching and online reinforcement learning to mitigate partial observability and…

Rosa: First, who's behind it and why it matters.

Title and authors: Rosa: So we're looking at this paper called "Value as Memory: Online Value-Guided Post-training for General Robot Policies." It sounds like it tackles the problem of training robots in the real world after they've been trained in simulation or offline data.

Dev: Yeah, and the authors are Yu, Chen, Xiong, Han, Chen again. The title really tells you what it’s about: using value as memory to guide online learning for general robot policies.

Rosa: Basically, it’s tackling those two big headaches in real-world post-training: partial observability makes picking actions ambiguous and offline methods delay getting feedback from actual deployment experience.

Dev: Exactly. And this paper proposes Value as Memory, which uses value caching and online reinforcement learning to fix both of those issues by combining offline demonstrations, off-policy data, and on-policy rollouts for sample-efficient adaptation.

The paper's summary: Rosa: So the summary says that VaM introduces value caching where actions are conditioned on recent value predictions as task progress memory. This is supposed to help the system distinguish between states that need different actions, even if they look visually similar because it tells you whether estimated progress improves, stalls, or declines.

Dev: It's like giving the robot a kind of short-term memory about how well it’s actually doing on the task. And then they use this to calculate an advantage estimate using that cached value function to help improve the policy in real time.

Rosa: The system moves from just offline RL warmup into online autonomous learning with sparse rewards, which is a big deal because it means the robot can actually learn complex, long-horizon tasks without needing a ton of dense reward signals.

Dev: The core idea is this hybrid training data scheme—combining offline demonstrations, off-policy trajectories, and current on-policy rollouts—which lets them jointly adapt the policy and value models with high sample efficiency.

The paper's improvements: Rosa: The paper highlights a few key improvements for VaM. First, it shows that value caching helps reduce the representation-induced lower bound on policy prediction error by defining a difference between action distributions as epsilon p, which suggests because epsilon p is greater than or equal to epsilon obs, the value cache can weaken that aliasing lower bound by making two states easier to separate.

Dev: That means it makes distinguishing those tricky states much more effective. Then they show that across long-horizon tasks involving deformable objects and contact-rich manipulation, VaM gets a mean absolute success-rate gain of twenty-nine point four percent with value caching, and a thirty-two point eight percent gain with online autonomous learning over matched offline training <ref:2610.22368#pg1,across long-horizon tasks involving deformable objects and contact-rich manipulation, VaM>.

Rosa: And the speed is impressive too; the evaluated online schedule performs eight times as many policy iterations under the same gradient-step budget, and they recorded more than two times lower time per gradient step when data collection and training were happening at the same time.

Dev: It’s about making real-time adaptation much faster, which is crucial for safety in deployment scenarios. But they also have a limitation to mention: this framework depends on a value function that closely tracks actual task progress, so they need that offline RL warmup phase to keep things stable before the online learning starts.

Conclusion: Rosa: So, to wrap up, the paper on "Value as Memory: Online Value-Guided Post-training for General Robot Policies" shows how value caching can support progress-aware recovery by showing more progress-ordered PCA trajectories and recovering from stuck states in qualitative comparisons on tasks like Laundry Folding and Magnetic Bus Assembly.

Dev: It’s about using that memory to give the robot a better sense of where it is on the task, which leads to more robust behavior under partial observability. The results show a significant improvement in success rates over long-horizon manipulation tasks compared to just offline training alone.

Rosa: And they show that when you run data collection, value training, and action training concurrently, you cut down the mean experimental time per gradient step by fifty-seven point eight percent across all tasks. It’s about making closed-loop policy updates much more efficient in practice.

Dev: So for someone listening who just cares about the practical application, this paper means we can transition from a slow offline training process directly into online learning with sparse rewards, and it does that faster than before.

Taro: From my side, I’m interested in how it handles the world misbehaving scenarios when it's already in deployment; specifically how that value caching helps with progress-aware recovery when things get stuck.

Rosa: That’s a good point, Taro, because those qualitative results on stuck states are really important for real-world reliability.

Dev: And from a control loop standpoint, the speed gains you mentioned—the eight times more policy iterations—means we have much lower latency in getting corrections applied to the robot's actions.

Taro: I just want to make sure that this reliance on a value function closely tracking actual progress doesn't introduce instability when the robot encounters something completely unexpected during those online updates.