RankQ: Offline-to-Online Reinforcement Learning via Self-Supervised Action Ranking
cs.AI, cs.RO
Submitted: 2026-05-11
Updated: 2026-09-29
Comments: Project page: https://horizonrobotics.github.io/gail/projects/rankq/
Project page: https://horizonrobotics.github.io/gail/projects/rankq
License: http://creativecommons.org/licenses/by/4.0/
The gist: Offline-to-online reinforcement learning (RL) improves sample efficiency by leveraging pre-collected datasets prior to online interaction.
Terminology
Abstract
Offline-to-online reinforcement learning (RL) improves sample efficiency by leveraging pre-collected datasets prior to online interaction. A key challenge, however, is learning an accurate critic in large state--action spaces with limited dataset coverage. To mitigate harmful updates from value overestimation, prior methods impose pessimism by down-weighting out-of-distribution (OOD) actions relative to dataset actions. While effective, this essentially acts as a behavior cloning anchor and can hinder downstream online policy improvement when dataset actions are suboptimal. We propose RankQ, an offline-to-online Q-learning objective that augments temporal-difference learning with a self-supervised multi-term ranking loss to enforce structured action ordering. By learning relative action preferences rather than uniformly penalizing unseen actions, RankQ shapes the Q-function such that action gradients are directed toward higher-quality behaviors. Across sparse-reward D4RL benchmarks, RankQ achieves competitive overall performance against seven baselines. In vision-based robot learning, RankQ enables effective offline-to-online fine-tuning of a pretrained vision-language-action (VLA) model in a low-data regime, achieving an average simulation success rate 38.2 percentage points higher than the next best method. In a high-data setting, RankQ improves simulation performance by 13.7 percentage points over the next best method and demonstrates strong sim-to-real transfer, increasing real-world cube stacking success from 43.1% to 88.9% relative to the VLA's initial performance.
Sources
- Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
- Conservative Q-Learning for Offline Reinforcement Learning
- D4RL: Datasets for Deep Data-Driven Reinforcement Learning
- SimpleVLA-RL: Scaling VLA Training via Reinforcement Learning
- Off-Policy Deep Reinforcement Learning without Exploration
- Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction
- Behavior Regularized Offline Reinforcement Learning
- AWAC: Accelerating Online Reinforcement Learning with Offline Datasets
- Improving TD3-BC: Relaxed Policy Constraint for Offline Learning and Stable Online Fine-Tuning
- Revisiting the Minimalist Approach to Offline Reinforcement Learning
- Offline Reinforcement Learning with Implicit Q-Learning
- Efficient Online Reinforcement Learning with Offline Data
- ENOTO: Improving Offline-to-Online Reinforcement Learning with Q-Ensembles
- Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble
- Decision Transformer: Reinforcement Learning via Sequence Modeling
- Scaling Sim-to-Real VLA Reinforcement Learning with Generative 3D Worlds
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems
- VSAL: A Vision Solver with Adaptive Layouts for Graph Property Detection