BORA: Bridging Offline Reinforcement Learning and Online Residual Adaptation for Real-World Dexterous VLA Models

arXiv:2605.30226 · cs.RO, cs.AI · Submitted 2026-05-28 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: I'm Rosa, and with me are Dev and Taro, guest researcher.

Dev: Today's paper: "BORA: Bridging Offline Reinforcement Learning and Online Residual Adaptation for Real-World Dexterous VLA Models".

Rosa: Vision-Language-Action (VLA) models face significant challenges in real-world dexterous manipulation due to high degrees of freedom and compounding execution errors,

Dev: First, who's behind it and why it matters.

Title and authors: Rosa: Moving on, let's talk more specifically about what the paper actually summarizes as the BORA framework and how it works in practice. Essentially, we’re looking at how they structured this offline-to-online RL post-training pipeline for these VLA models.

Dev: The summary emphasizes that the offline phase is dedicated to distilling intent comprehension from offline data by constructing an action-conditioned critic that takes both the VLM's cognition tokens and action chunks.

Taro: That fusion of semantic tokens with continuous actions in the critic seems to be the central innovation for extracting those foundational manipulation skills before we even get to online adaptation.

Rosa: Right, and they use a consistency policy as the action expert during this offline phase to generate these action chunks in just one or three steps, which helps truncate the computation graph for efficient gradient backpropagation.

Dev: That’s smart because it directly tackles the problem of long denoising chains causing noisy gradients when dealing with diverse and potentially redundant micro-actions in offline data.

Taro: So the summary paints a picture of an offline phase that is designed to be computationally efficient while still ensuring the resulting policy is informed by both language and physical action structure.

Rosa: Then, during the online phase, they introduce this lightweight, Human-in-the-Loop chunk-wise residual adaptation mechanism to correct for real-world execution deviations.

Dev: The structure of that residual actor is defined by the formula Afinal = Abase + λres · πres(sprop, Abase, zVLM), which shows it’s generating compensations specifically at the action chunk level.

Taro: That residual mechanism is what allows the system to safely extract corrective priors from human intervention data while freezing the main VLA base to prevent catastrophic feature drift.

Rosa: And they pair this with Critic Inheritance, initializing the online value function with that offline critic, which is supposed to provide a stable value estimation.

Dev: That inheritance is crucial because it ensures that even during online fine-tuning, the system has a baseline for what constitutes a good or bad action based on the physically grounded prior from the offline phase.

Taro: So, in summary, the core idea is building a strong offline foundation through action conditioning and then layering a lightweight, human-guided correction mechanism on top for real-world execution.

Rosa: It’s a very structured approach that moves away from purely visual imitation learning toward something that explicitly models both the intent and the physical dynamics of dexterous tasks.

Dev: Exactly, and it’s trying to solve the sample inefficiency problem inherent in online RL by leveraging that robust offline knowledge to guide the adaptation process.

Taro: It seems like they are systematically addressing the biggest hurdles in deploying VLA models into physical reality by separating the intent learning from the real-time error correction.

Rosa: So, BORA is essentially a post-training method that takes a pre-trained VLA model and makes it robust enough for real, dexterous manipulation by injecting structure derived from offline data and human feedback.

The paper's summary: Dev: Now let’s look at the specific improvements they propose in the BORA framework, because those are the technical details that really show how they achieve their results.

Rosa: The main improvement is definitely the Action-Conditioned Critic for Dexterous Manipulation, which they design to fuse continuous action chunks with the VLM’s cognition tokens.

Taro: So, this critic isn't just looking at what’s on screen; it't explicitly grounded in what the VLM understands about the task and where the physical interaction should occur.

Dev: That means the value estimation is fundamentally tied to actual physical interactions rather than relying solely on visual context, which is a big step toward reliability.

Rosa: Then they introduce the Lightweight Residual Online Adaptation mechanism, which involves freezing the VLA base and leveraging intervention-driven rewards during deployment.

Taro: That mechanism is what allows for sample-efficient adaptation by focusing only on correcting execution errors at the chunk level, rather than retraining the entire model from scratch.

Dev: And they couple that residual actor with a Critic Inheritance strategy to stabilize value estimation and provide discriminative guidance for that residual policy.

Rosa: Plus, they use an asymmetric Intervention-Driven Reward function during adaptation to guide the RLPD pipeline by imposing an instant penalty upon OOD drift and granting a positive recovery reward upon human corrective action.

Taro: That penalty for OOD drift is important because it actively steers the residual policy away from risky states that the offline critic identified as problematic.

Dev: It sounds like they’ve built a very specific control loop where the offline knowledge sets the stable baseline, and human feedback guides safe, targeted adjustments in real-time.

Rosa: This entire suite of improvements is what makes BORA a unified framework designed to significantly enhance real-world deployment robustness.

Taro: It’s a comprehensive set of techniques that systematically handles the challenges we see when deploying VLA models in physical systems, from initial intent generation to final error correction.

Dev: So it’s not just one fix, but a combination of fusing different elements—critic design, policy truncation, and intervention-driven rewards—to achieve stability.

Rosa: It really shows how you can bridge the gap between learning abstract visual concepts and executing precise physical movements reliably through this layered approach.

The paper's improvements: Rosa: So, to wrap up this discussion on BORA, we’ve discussed how it combines offline learning with online adaptation to create a more robust system for dexterous VLA models.

Dev: We’ve covered the key improvements like the action-conditioned critic and the residual adaptation mechanism that stabilize value estimation during real-world use.

Taro: I think what stands out is how it tackles credit assignment failure by making sure the critic is grounded in physical consequences rather than just visual context.

Rosa: And I feel that the BORA Unified Framework really succeeds by achieving a thirty-three percent absolute increase in average success rate and up to a forty-three percent improvement in unseen object generalization across five complex real-world tasks.

Dev: That level of success suggests that this method is genuinely effective for pushing VLA models toward reliable deployment, provided the sample efficiency gains translate well into real-world scenarios.

Taro: The implication for autonomy is that we can expect these systems to be much more capable of handling dynamic environments and unexpected physical disturbances without needing constant retraining.

Rosa: It seems like this paper, "BORA: Bridging Offline Reinforcement Learning and Online Residual Adaptation for Real-World Dexterous VLA Models," provides a very practical path forward for making these AI systems capable of handling the physical demands of real-world tasks.

Dev: It’s a framework that moves beyond just imitation by incorporating RL post-training to address execution discrepancies in high-DOF systems.

Taro: We’re really excited about the potential for this to make embodied AI much more dependable when interacting with the physical world, even if we still need to figure out how long it can run reliably in truly unstructured conditions.

Conclusion: Rosa: So we’ve walked through the BORA framework, which is essentially an offline-to-online RL post-training method designed for real-world dexterous VLA models.

Dev: Exactly, and it’s really smart how they structure the pipeline to address both the knowledge extraction in the offline phase and the necessary error correction during online deployment.

Taro: I think what we saw was their Action-Conditioned Critic, which is designed to fuse VLM cognition tokens with continuous action chunks, providing a physically grounded value estimation.

Rosa: That’s the core idea, and it really does seem to solve the problem of relying too much on raw pixels when evaluating actions in physical space.

Dev: And then during the online phase, they use that inherited critic to stabilize things while adding a lightweight residual actor for chunk-wise adaptation, which is pretty clever for managing latency and failure modes.

Taro: The intervention-driven reward function guiding the RLPD pipeline seems key there, especially how it punishes OOD drift instantly while rewarding human corrective actions.

Rosa: It really shows how they’ve managed to bridge that gap between learning abstract intent and ensuring reliable physical execution, which is what we need for real-world applications.

Dev: From an engineering standpoint, the focus on freezing the VLA base prevents catastrophic feature drift, which is a huge concern when you're trying to fine-tune models in a live setting.

Taro: I just think the implications for autonomy are pretty big; if this works reliably outside the lab with minimal human intervention, it opens up a lot more possibilities for complex, unstructured environments.

Rosa: It certainly makes me wonder how long these models can stay reliable in truly messy, dynamic settings before they start needing constant updates.

Dev: That’s the million-dollar question for any deployment scenario, Rosa; we need to nail that loop rate and ensure those residual adjustments don't introduce new instabilities.

Taro: I agree with Dev on the stability point; if it’s robust enough to handle execution failures, we’ll see it perform much better when the world misbehaves unexpectedly.

Rosa: Well, that wraps up our discussion on BORA, this offline-to-online RL post-training framework.

Dev: Yeah, it’s a solid piece of work that shows how structured offline training can make online adaptation much safer and more efficient.

Taro: It’s exciting to see researchers moving toward methods that explicitly model the physical consequences during the learning phase, rather than just relying on visual heuristics.

Shanghai Jiao Tong University (SJTU) · CASIA Institute of Artificial Intelligence Laboratory at Shanghai AI Laboratory at Shanghai Jiao Tong University (SJTU) · USTC

cs.RO, cs.AI

Submitted: 2026-05-28

Updated: 2026-10-01

Comments: 9 pages,7 figures

Project page: https://chenzhongxi-sjtu.github.io/BORA

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 91/100

The gist: Vision-Language-Action (VLA) models face significant challenges in real-world dexterous manipulation due to high degrees of freedom and compounding execution errors, necessitating a framework that

Key concepts

Action-Conditioned Critic
This is a tool used in the offline phase that estimates the value of an action by explicitly combining continuous action chunks with the VLM's semantic tokens. This ensures that the critic's value judgments are based on what actually happens during physical interaction, not just what looks good visually. It provides precise guidance for learning how to move.
Residual Chunk Adaptation
In the online phase, this is a lightweight correction mechanism applied chunk-by-chunk to fix execution errors. The main VLA model is frozen, and this residual actor generates small adjustments based on the offline policy and the current state. This allows for safe, sample-efficient fine-tuning that compensates for real-world discrepancies without corrupting the original learned skills.
Human-in-the-Loop (HiL) Adaptation
This refers to the online process where human intervention is used to guide and correct the model's actions. The framework uses an asymmetric reward function that penalizes out-of-distribution errors and rewards positive human corrections. This makes the online adaptation safe, as it only learns from successful human interventions, stabilizing the policy during real-world deployment.
Critic Inheritance
This technique involves initializing the value function used in online training with the critic trained during the offline phase. By inheriting this pre-trained critic, BORA ensures that value estimation remains stable and consistent throughout the adaptation process. This prevents catastrophic feature drift while allowing the residual actor to focus only on necessary local improvements.

Terminology

Summary

Vision-Language-Action (VLA) models face significant challenges in real-world dexterous manipulation due to high degrees of freedom and compounding execution errors, necessitating a framework that bridges offline learning with online adaptation for reliable deployment. The gist: BORA is an offline-to-online RL post-training framework designed for real-world dexterous VLA models that constructs an action-conditioned critic in the offline phase and introduces a lightweight, Human-in-the-Loop (HiL) chunk-wise residual adaptation mechanism in the online phase to correct execution discrepancies.

The BORA Framework Overview

BORA is a comprehensive offline-to-online reinforcement learning framework tailored for Vision-Language Models (VLAs) in dexterous manipulation. Its core philosophy establishes a two-stage pipeline: first, extracting foundational manipulation skills via offline RL, and subsequently compensating for real-world execution errors through Human-in-the-Loop (HiL) online residual RL adaptation on the physical robot. The framework is designed to address challenges such as credit assignment failure in action generation and visual occlusion during evaluation.

Offline Phase: Action-Conditioned Critic and Consistency Policy

In the offline phase, BORA deploys a Consistency Policy as the action expert to generate continuous action chunks in just 1–3 steps, thereby truncating the computation graph for efficient gradient backpropagation. To mitigate visual overfitting and handle high dimensionality, BORA designs an Action-Conditioned Critic that explicitly fuses the continuous action chunks with the VLM’s cognition tokens. This critic outputs k-dimensional value vectors conditioned on VLM semantic tokens and relative position embeddings, ensuring that value estimation is fundamentally grounded in actual physical interactions rather than visual context alone. The actor is optimized using a validity-masked clipped PPO surrogate objective, combined with Behavior Cloning regularization to ensure policy conservatism and prevent out-of-distribution deviation.

Online Phase: Lightweight Residual Chunk Adaptation

During the subsequent online phase, BORA freezes the VLA base to prevent catastrophic feature drift. It introduces a lightweight, Human-in-the-Loop (HiL) chunk-wise residual adaptation mechanism. This mechanism is formulated as:

Afinal = Abase + λres · πres(sprop, Abase, zVLM)

The residual actor generates compensations at the action chunk level. To stabilize optimization dynamics, BORA couples Critic Inheritance with an Intervention-Driven RLPD pipeline. The offline-trained critic initializes the online value function. The residual actor optimizes a conservative improvement hinge loss that enforces a local policy improvement bound: Qϕ(zt, aˆt+i) ≥ Qϕ(zt, abase t+i) + δ. Furthermore, an asymmetric Intervention-Driven Reward function guides the RLPD pipeline by imposing an instant penalty upon OOD drift and granting a positive recovery reward upon human corrective action.

Key Contributions and Results

BORA's main contributions are threefold: (1) Action-Conditioned Critic for Dexterous Manipulation, which enables precise, action-conditioned value guidance evaluated on physical execution consequences rather than visual context alone. (2) Lightweight Residual Online Adaptation, which achieves safe and sample-efficient online adaptation by freezing the VLA base and leveraging intervention-driven rewards. (3) The BORA Unified Framework, which significantly enhances real-world deployment robustness, achieving a 33% absolute increase in average success rate and up to a 43% improvement in unseen object generalization. Experiments across five complex real-world tasks demonstrate that BORA significantly outperforms pure imitation learning and traditional decoupled RL baselines.

Mechanistic Insights

The inherited action-conditioned critic provides stable value estimation during online execution, maintaining high confidence along successful trajectories while remaining consistently low along failed trajectories, even under severe visual occlusion. This allows the residual policy to mitigate superficial visual ambiguities and guide adaptation by penalizing high-risk states. The framework effectively bridges the gap between offline intent learning and online physical execution through this progressive optimization strategy.

Limitations

The paper notes two main limitations: first, the framework relies on visuo-proprioceptive inputs and lacks dense tactile feedback, suggesting that integrating high-fidelity tactile arrays could further tighten the perception-action loop. Second, the current physical evaluation is constrained to a single arm-hand topology; verifying BORA’s cross-embodiment generalization remains an important future direction.

Conclusion

BORA presents a comprehensive offline-to-online RL post-training framework that seamlessly bridges offline alignment with efficient online fine-tuning, addressing critic overfitting, execution discrepancies, and catastrophic feature drift in dexterous VLA models. Moving forward, the goal is to extend this framework toward high-precision skills and investigate its scalability to structurally complex tasks.

Improvements for AI systems

Here are the specific improvements that can be made to existing AI systems by implementing the BORA framework, and what these improved systems will be capable of:

The core improvement is a shift from brittle, purely imitation-based or standard RL fine-tuning models to a robust, two-stage offline-to-online reinforcement learning pipeline.

Here are the specific improvements:

  1. [Improvement] Implement an Action-Conditioned Critic that fuses VLM cognition tokens with continuous action chunks during offline training.

  2. [Improvement] Utilize a Consistency Policy for action generation in the offline phase to efficiently handle high-dimensional, multimodal continuous action spaces without accumulating severe gradient noise from long denoising chains.

  3. [Improvement] Employ a Lightweight, Human-in-the-Loop (HiL) Chunk-wise Residual Adaptation mechanism during online deployment.

  4. [Improvement] Integrate an Inherited Action-Conditioned Critic into the online phase to stabilize value estimation and provide discriminative guidance for the residual policy.

  5. [Improvement] Use an Asymmetric Intervention-Driven Reward function during adaptation to guide the residual policy toward high-quality recovery trajectories while aggressively penalizing out-of-distribution (OOD) drift.

The improved AI system (the BORA model) will be capable of the following specific capabilities:

  1. [Capability] Extrapolating Dexterous Intent from Semantic and Physical Context: The model will be able to generate high-fidelity, physically grounded action chunks by explicitly conditioning its value estimation on the VLM's semantic tokens. This means it won't just guess an action based on a vague visual feature; it will evaluate the action based on what it knows about the task and where the physical interaction should occur.

  2. [Capability] Robust Generalization to Novel Objects: By leveraging offline RL alignment with object-unseen data, the system will significantly improve its ability to perform tasks on objects or in environments never seen during training (up to a 43% improvement in unseen object generalization). It will not rely solely on visual similarity but on learned physical principles.

  3. [Capability] Safe and Sample-Efficient Real-World Deployment: Unlike traditional online RL, which is sample-inefficient and risky, this system requires only minimal human intervention (1–2 times per task) to correct execution errors. This makes real-world deployment significantly safer, faster to deploy, and cheaper in terms of required physical interaction data.

  4. [Capability] Error Correction Under High Occlusion: The integrated critic is specifically designed to be invariant to visual occlusion by focusing its evaluation on the semantic tokens and continuous action chunks rather than raw pixel artifacts. This means the system can maintain high performance even when critical parts of the hand or object are hidden from view.

  5. [Capability] Adaptive Correction of Execution Drift: When deployed in a real environment, if execution deviates due to friction or contact dynamics, the residual actor will use the inherited critic's stable value landscape to learn and apply corrective actions derived from human demonstrations (HiL). This allows the model to adapt its physical control strategy instantly without catastrophic forgetting of its core vision-language understanding.

Sources

Related papers