RARM: Confidence-Gated Progress Reward Modeling for RL in Manipulation

arXiv:2606.22027 · cs.RO, cs.AI · Submitted 2026-06-20 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "RARM: Confidence-Gated Progress Reward Modeling for RL in Manipulation".

Jane: The paper was written by Pengzhi Yang, Pengyu Jing, Xinyu Wang, Kehan Wen, Zhenhao Huang et al. from National University of Singapore (NUS) and Booking.com and School of Artificial Intelligence, Nanjing University and Shanghai Jiao Tong University.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Summary: Jane: Okay, so if we’ve grasped that RARM is about giving robots better ways to understand progress, let’s look at how they summarize its approach in the paper. They seem to be introducing a specific mechanism—the confidence gating—that acts like a filter on the reward signal.

Tom: It's not just adding a reward; it's modifying *how* the agent perceives the reward, which is much more sophisticated. Essentially, the system is designed to quantify how confident it should be in its own actions and progress metrics at any given time step.

Meng: I appreciate that they aren't just giving us a high-level concept; they are describing a functional model. If the agent’s confidence dips low, does RARM automatically temper the reward signal? That sounds like it could prevent catastrophic failure during training.

Lu: Precisely, Meng. The gating mechanism acts as a form of self-correction or skepticism within the learning process. Instead of treating every observed change as equally valuable progress, it weights the rewards based on the model's internal assessment of certainty about that progress.

Lalam: That concept—self-skepticism—is incredibly powerful for AI development because it moves past simply optimizing for a reward and starts optimizing for *reliable* optimization. It makes the learning process itself more trustworthy.

Jane: Thinking about this in simpler terms, imagine you're trying to teach a robot to pick up an egg. If it bumps into the egg hard, a traditional reward might just say "Negative Reward!" But RARM seems to be able to say, "You failed, but given your previous movements, we were actually pretty confident you were close; let's adjust the penalty based on that confidence."

Tom: That’s a fantastic way to put it. It turns a binary success/failure system into a gradient of reliable progress. So we’re moving toward models that aren't just *good* at tasks, but are also *good at knowing* how good they are while performing the task.

Lu: This ability to model uncertainty and progress simultaneously is what makes RARM such a breakthrough for real-world deployment, especially when the environment is noisy or unpredictable.

Meng: If we could integrate this confidence-gating principle into industrial robots, it would drastically reduce the amount of data needed to train them in variable environments. That's a massive cost saving right there.

Lalam: It fundamentally changes how we view robotic intelligence; it’s not just about achieving the task, but about building a model that constantly updates its understanding of its own limitations and successes.

Improvements: Tom: We've talked through the general mechanism, but what really excites me is how they detail the improvements—how RARM specifically enhances traditional RL frameworks. They aren't just suggesting a new component; they’re proposing a systemic improvement to the reward definition.

Jane: Right, because simply adding confidence gating isn't enough; they are addressing the underlying issue of sparse and non-dense rewards which have plagued imitation learning for years. The progress modeling part is key here.

Lu: What I take away from this segment is that RARM seems to provide a robust framework for defining "progress" itself, rather than just defining the endpoint. This allows it to handle tasks where the success isn't immediately obvious or quantifiable by a simple binary check.

Meng: From an implementation viewpoint, this means we don't have to hand-engineer complex, multi-stage reward curves for every single task. We define the progress model, and RARM helps us navigate the messy middle ground of learning.

Lalam: It introduces a level of abstraction that is really exciting—it’s modeling the *potential* for success rather than just rewarding it after the fact. That allows for much more generalizable skills across different manipulation tasks.

Jane: So, they're making the reward signal richer and more informative throughout the entire episode, not just at the end when everything is done. It gives gradient feedback even when things are going wrong in subtle ways.

Tom: Exactly! It fills in those critical moments where traditional RL agents often get lost because they aren't given enough meaningful signals to tell them what direction to go next.

Lu: And this framework seems highly adaptable, meaning that once you've built the progress model for one type of manipulation, the core principles can be applied and fine-tuned for a totally different set of objects or environments.

Meng: If we could modularize reward engineering this way, it would drastically accelerate the development cycle for new robotic applications—we

Paper discussion segment 3: Tom: So, we've established that RARM uses a confidence-gated approach to map progress against a reference demonstration, but how does this fundamentally improve upon traditional reinforcement learning methods?

Jane: It moves beyond just rewarding success or failure; it gives the robot a continuous sense of how much progress it's actually making along the path to completion.

Lu: That continuous signal is incredibly powerful because it allows the policy to learn from partial successes, which is critical in long-horizon tasks like folding cloth.

Meng: From an engineering perspective, this means we don't need huge datasets of successful demonstrations for every single task configuration anymore. One robust reference demonstration might be enough.

Lalam: And because the system can tolerate ambiguity by gating low-confidence matches, it makes the learned behavior much more resilient to real-world environmental noise and variability.

Tom: That’s a huge leap in reliability; if the reward function is stable, we aren't constantly training against spurious or misleading signals.

Jane: Exactly, so we’ are effectively turning a single reliable demonstration into a dense feedback loop that helps the agent learn from its own mistakes without being wildly distracted by noise.

Lu: The implication for complex manipulation is that RARM gives us an entire class of scalable reward models, rather than just task-specific ones.

Meng: Imagine deploying this across multiple industries; we could apply a similar progress model to assembly lines where the sequence is known, but the exact path isn's not.

Lalam: It allows AI systems to learn with a sophisticated level of self-awareness about their own performance limits, making them far more robust companions in society.

Tom: That’s a great way to frame it; they' learning from their own uncertainty.

Jane: And because the whole system is designed to handle uncertainty, it’ inherently leads to better generalization when we can't perfectly supervise every single state.

Meng: We need tools that work even when we don't have perfect supervision, and RARM provides that tool.

Lu: It suggests a future where the definition of task completion is not just a final state, but a continuous trajectory of confidence.

Lalam: That shift allows us to build systems that are both powerful and trustworthy.

Conclusion: Tom: So, wrapping up our deep dive into RARM: Confidence-Gated Progress Reward Modeling for RL in Manipulation, it really boils down to this idea that standard rewards just aren't enough for complex physical tasks, right?

Jane: Exactly, Tom. It shows that if you can make the reward signal actually guide the *process* of getting somewhere—not just judge the final outcome—you unlock a whole new level of robotic dexterity.

Meng: I keep thinking about how much time we spend engineering those reward functions; they're notoriously fragile. If RARM makes that progress scaffolding robust, it cuts down years off the development cycle for physical AI systems.

Lu: And think about what that means for scientific discovery! Suddenly, robots aren't just executing pre-programmed routines; they’re assisting human scientists by handling materials in ways we barely knew how to teach them before.

Lalam: It feels like a massive leap toward embodied intelligence that can truly interact with messy, unpredictable real-world environments, not just perfect simulations.

Tom: You hit it on the nail head with that, Lalam; the unpredictability is where these systems usually fall apart. So, Lu, you mentioned scientific discovery—what's the wildest application you can picture right now?

Lu: Well, if we combine this progress reward structure with things like microscopic surgery or deep geological sampling, we're talking about AI guiding instruments through environments that are incredibly small and incredibly hard to model perfectly.

Jane: That’s such a vast jump from folding clothes to microsurgery; it really shows the generality of the approach, doesn't it?

Meng: From an engineering standpoint, I’m most interested in the generalization across modalities—can this framework easily adapt if we swap out cameras for tactile sensors or haptic feedback arrays?

Tom: That’s a solid question, Meng. It suggests the reward structure itself is portable, which is huge. But Lalam, circling back to the impact, what does this mean for how people interact with AI in the future?

Lalam: It shifts the dynamic from 'AI tells you what to do' to 'AI helps you *do* it,' improving human autonomy by giving us reliable tools in our hands.

Jane: I think that’s the perfect way to put it—it’s about partnership, making AI a genuine extension of human capability rather than just a sophisticated gadget.

Tom: It's incredible how far we've come just by refining the reward signal for tasks like picking up an eraser. We gotta take a quick breather before we jump into next week’s paper, but I have to say, RARM really sets the bar high for real-world robotics research.

Lu: Seriously, this architecture is a game changer for physical AI understanding the 'how' behind movement.

Meng: It gives us a much more reliable benchmark for assessing robotic proficiency outside of clean lab settings.

Lalam: Because it models progress, it naturally improves how we build trustworthy and useful cultural artifacts using AI.

Jane: We’ll definitely be watching the next evolution of this work closely; thanks so much to everyone for joining us today!

Pengzhi Yang, Pengyu Jing, Xinyu Wang, Kehan Wen, Zhenhao Huang, Xin Liu, Yiduo Qu, Minghao Fu, Yaheng Shen, Fan Shi

National University of Singapore Human-Centered Robotic Lab (for authors marked with †) · Booking.com (for author Xinyu Wang) · University of Cambridge (for author Minghao Fu)

cs.RO, cs.AI

Submitted: 2026-06-20

Updated: 2026-08-25

Project page: https://rarm-robotics.github.io

Importance score: 80/100

The gist: The paper introduces RARM, a novel reward model designed for reinforcement learning (RL) in manipulation tasks, which provides a "highly scalable paradigm for visual reward design." The core

Key concepts

Confidence Gating
A mechanism that acts as a filter on the reward signal. It quantifies how confident the agent should be in its own actions and progress metrics, tempering rewards if confidence dips low to prevent catastrophic failure during training.
Progress Modeling
The core function of RARM, which provides a robust framework for defining 'progress' itself rather than just defining the task's endpoint. This allows the system to handle tasks where success is not immediately obvious or quantifiable by a simple check.
Reinforcement Learning (RL)
A machine learning method where an agent learns optimal behavior by interacting with an environment and receiving rewards. RARM improves this by giving continuous, reliable feedback throughout the task's process.

Terminology

Summary

The paper introduces RARM, a novel reward model designed for reinforcement learning (RL) in manipulation tasks, which provides a highly scalable paradigm for visual reward design. The core mechanism involves using a cross-attention comparator that treats the reference trajectory as a coarse temporal scaffold for progress localization, while simultaneously employing a calibrated confidence gate to suppress artifacts inherent to generative video models. This approach means that RARM does not require pixel-perfect alignment with the reference video.

Robustness and Generalization:

A key aspect of the research is demonstrating the model's stability in uncontrolled environments. In a controlled perturbation study using Task 4 (Bimanual Clothes Folding), RARM was tested against four distinct real-world failure modes:

  • Augment 1 (Glare): synthetic white spots and halos that partially occlude the rollout frames, simulating specular highlights and lens flare.

  • Augment 2 (Color shift): a green tint applied across the frame, simulating changes in ambient lighting and white balance.

  • Augment 3 (Occlusion): a black box covering part of the image, simulating a foreground obstruction.

  • Augment 4 (Viewpoint): a perspective warp introducing pitch and yaw, simulating camera misalignment.

The results showed that For every augmentation, RARM continued to recover a coherent progress signal, with the estimated progress closely tracking the ideal linear trend across all tested perturbations.

Efficiency and Performance:

In terms of computational efficiency, RARM was benchmarked against various baseline reward models. The model demonstrated strong performance metrics: our RARM performs in the same band as the rest, being able to process each frame of the 125 in under 10ms. Furthermore, it was noted that RARM uses much less GPU memory compared to VLM-based Reward Models (GVL, Robometer, RoboDopamine).

Integration into RL Frameworks:

RARM is integrated into state-of-the-art RL algorithms for both simulation and real-world deployment:

  1. Simulation (Drq-v2): In the DrQv2 loop, RARM functions as a drop-in dense reward function. The model scores buffered rollout frames to produce progress-aware scalar rewards, which are then used as the active environment reward r t in the standard DrQ-v2 Bellman update.

  2. Real-Robot Adaptation (DSRL): For real-robot experiments using Diffusion Steering RL (DSRL), RARM is utilized to adapt a frozen pi 0 diffusion policy. In this setup, RARM produces one reward per pi 0 query step from the captured rollout frames, replacing the default −1 rewards in default DSRL implementation.

Overall, the paper validates RARM as a highly reliable and efficient tool for generating progress-aware rewards that can drive downstream policies using automated pipelines based on text-to-video foundation models.

Improvements for AI systems

As a diligent researcher, I have analyzed the RARM paper and identified three critical areas where its core methodologies can be generalized to significantly improve current AI systems—specifically those in Reinforcement Learning (RL) for robotic manipulation.

The following improvements focus on replacing brittle, hand-crafted reward functions with a robust, data-efficient, confidence-gated progress estimation mechanism.

Improvement: Replace traditional task-specific reward engineering (e.g., proximity to target, joint angle thresholds) with an architecture that leverages a single successful demonstration as a dynamic progress anchor, implemented via a lightweight visual comparator.

  • Mechanism: The AI system uses a pre-trained cross-attention model (trained contrastively on general video data) to map the current rollout clip against all clips within the reference demonstration. Instead of assigning reward based on simple similarity, it determines the optimal temporal alignment (argmax g theta(clip current, reference j)).

  • What the Improved System Can Do:

  • Handles Long-Horizon Tasks: It provides a stable, dense signal for complex, multi-stage tasks (e.g., folding, assembly) where success is only defined by the final state. This solves the sparse reward bottleneck that plagues long-horizon RL agents.

  • Reduces Data Dependency: The system can be deployed in environments where human demonstrations are scarce or impossible to collect, requiring only one successful trajectory as a progress scaffold.

Improvement: Integrate a dynamic confidence gate (delta j) into the reward calculation, ensuring that the RL agent is only rewarded for movement that is both forward (pi j(t) > pi t-1) and trustworthy.

  • Mechanism: The system calculates a confidence threshold based on the self-comparison statistics within the reference demonstration. At runtime, if the current rollout's best match to any reference clip falls below this calibrated threshold, all progress is suppressed (reward = 0). If it exceeds the threshold and represents a forward movement along the reference timeline, it is rewarded.

  • What the Improved System Can Do:

  • Prevents Reward Hacking: It eliminates false-positive rewards—where a visually plausible but physically incorrect state receives high reward. This prevents agents from exploiting noisy or misleading visual cues to achieve high scores without actual progress.

  • Increases Stability in Noisy Environments: The system can be deployed in real-world settings where lighting, sensor noise, or minor visual perturbations occur; the confidence gate ensures that only robust, forward-moving progress is utilized for policy updates.

Improvement: Utilize the cross-attention mechanism not just to measure similarity, but to localize the agent's current state relative to a reference trajectory, allowing for non-synchronized movement patterns.

  • Mechanism: The system treats the reference demonstration as a continuous progress map (pi j in [0, 1]). The agent’s current clip is mapped onto this map based on the best match. This is distinct from simple trajectory matching; it allows for temporal misalignment—the agent can pause, recover, or move slower than the reference and still have its progress accurately localized.

  • What the Improved System Can Do:

  • Enables Flexible Exploration: The RL agent is no longer penalized solely for deviating from a pre-recorded path. It is rewarded for progress, allowing it to learn optimal, robust, and flexible strategies while still being guided by the successful reference trajectory.

  • Scales to Complex Motion: It allows the system to track progress in complex tasks where intermediate steps are ambiguous or where movement speed varies significantly from a single static demonstration.

Sources

Related papers