FOCUS: From Privileged States to RGB-D with Controlled Modality Switching and Representation Alignment

arXiv:2610.11119 · cs.RO · Submitted 2026-10-08 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: Today's paper: "FOCUS: From Privileged States to RGB-D with Controlled Modality Switching and Representation Alignment".

Dev: The gist FOCUS, a single-stage PPO framework,

Rosa: First, who's behind it and why it matters.

Paper summary: Rosa: So we're talking about this paper called "FOCUS: From Privileged States to RGB-D with Controlled Modality Switching and Representation Alignment". It tackles the problem that training a robot with high-resolution RGB-D cameras is super slow because those inputs are just too noisy and have too much data to handle.

Dev: Exactly. And you get this privileged state information, which is basically what you see in simulation, which makes training way faster, but then when you take that trained robot outside to test it, it can't use that privileged state anymore because there isn't any privileged state there.

Taro: So the core problem they’re looking at is this gap between the training environment where we have this extra helpful data and the real world testing environment where we only have RGB-D. It’s how to bridge that gap so the robot learns well in training but works on its own when it only sees what's out there.

Rosa: And they propose FOCUS, which is a single-stage PPO framework designed to train the critic using that privileged state while automatically deciding whether the actor should collect rollouts from either that privileged state or the actual RGB-D data.

Dev: The big claim here is that they regulate this switching based on how much the actor's behavior changes when you swap out one input for the other, which is measured through KL divergence between those two inputs.

Taro: So it’s not just about switching randomly; they use this measure of disagreement to decide if it’s time to collect data from RGB-D or stick with the privileged state latents. That sounds like a smart way to control exposure during training.

Rosa: And they also have this representation alignment objective, which is designed to make sure that even when the actor is using different inputs, those inputs result in compatible action distributions.

Dev: They use a policy-output alignment loss, specifically KL divergence between the policy induced by the privileged state and the one induced by RGB-D observations. This loss helps keep things consistent across both modalities during training.

Taro: So it’s not just controlling *when* to use RGB-D, but also making sure that when it does use it, the resulting actions look like they belong with what the robot is already trained to do from the privileged state.

Rosa: Right. And they find that these two things—the switching mechanism and the alignment objective—work together to limit RGB-D rollouts when those distributions don't match up well, which means it only increases RGB-D exposure when it makes sense for learning.

Dev: The experimental results show this works well across five different manipulation tasks, raising average test success from zero point seven one to zero point nine three compared to a baseline that just used RGB-D at test time on those tasks.

Paper summary: Taro: And looking at the budget normalized training success AUC, it jumps from zero point four seven up to zero point six five across all the pipelines they tested, which shows a real improvement in how much learning happens for the same amount of data used in training.

Rosa: On a specific task called Pick-and-Place, they see test success go up from zero point four seven to zero point eight six and the AUC increase five times, from zero point one two to zero point six one, which is pretty significant for that kind of manipulation work.

Dev: Now looking at the ablation studies, they showed that a fixed schedule ramp without the KL feedback from the switching mechanism still helps, but it just lets the RGB-D probability increase based on a predefined warmup schedule w(r).

Taro: And when they looked at FOCUS-simple, which only kept that KL-guided switch and the image-encoder policy output KL term, it suggested that having controlled visual exposure and aligning the image encoder through policy output KL are the most reliable parts of this system.

Rosa: So, to put it plainly, FOCUS uses a single PPO setup to manage whether you collect data from privileged states or RGB-D latents during training, using disagreement as a signal for switching and alignment to keep things consistent.

Dev: It’s about solving that training versus testing gap by making sure the actor is adapted from the privileged state rollouts toward actual RGB-D control, without letting it see too much of the real world too early.

Taro: And what this means for us is that we can get better sample efficiency in simulation while still having a robot policy that actually works when it’s interacting with real visual data at test time.

Rosa: It also suggests that the way we structure the training signal—how we blend simulated knowledge with real-world sensor data—is a key factor in making these kinds of skills more robust.

Dev: The paper also points out a limitation, which is that they don't fully address how this system would perform when you move it to physical robots or more complex manipulation suites outside of simulation.

Taro: That makes sense. They focused on the latent space dynamics in simulation, and the real world adds a whole new layer of uncertainty that’s hard to model perfectly with just these switching rules.

Rosa: So, while this is a strong result for achieving high performance metrics in simulated settings with limited real-world data, it leaves open questions about its direct applicability to physical robots where sensor noise and dynamics are much messier.

Dev: The name of the paper, "FOCUS: From Privileged States to RGB-D with Controlled Modality Switching and Representation Alignment," really sums up the core idea of this paper.

Taro: It’s a controlled system that manages the flow of data based on how well the learned actions from different sources align.

Rosa: So, if you're interested in how we can build systems that learn from simulation but perform reliably in reality without needing perfect real-world data for every step, this paper is worth checking out.

Conclusion: Rosa: So we're wrapping up this look at FOCUS, which is this PPO framework that manages how the robot gets its training data from privileged states versus actual RGB-D images during learning and testing.

Dev: Yeah, Rosa, so the main point is this single-stage setup that uses a switching mechanism based on policy disagreement to decide when to use the RGB-D data instead of those simulation latents.

Taro: It means they're not just blindly collecting data from one source; they are actively regulating that exposure based on how much the robot’s behavior changes between those two inputs.

Rosa: Right, so for someone just listening, it boils down to this system making smart choices about which input to use during training so the robot can actually perform well when it only sees real-world camera feeds at test time.

Dev: And they're saying this improves sample efficiency because you get the benefit of fast simulation training while still hitting better performance metrics on the real hardware.

Taro: The authors focused on getting these two things—the switching and the representation alignment—working together to ensure that whatever input is being used, it leads to compatible actions.

Rosa: It seems like their conclusion is that this controlled approach successfully bridges that gap between where training happens and where the robot actually operates.

Dev: And they show strong results across five different manipulation tasks, with test success jumping up significantly compared to just using RGB-D at test time on its own.

Taro: But they also pointed out something important, which is that while this works well in simulation for these specific tasks, it’s still not fully tested on physical robots or much more complex scenarios outside of what they set up.

Rosa: So the big implication is that this shows a really clever way to structure training signals when you have access to high-fidelity simulation data but need it to translate into good real-world robot skills.

Dev: It suggests that how you blend simulated knowledge with real sensor data is a critical design choice for making those kinds of skills robust.

Taro: And the next thing we need to look at is whether this kind of controlled switching can handle the messy, unpredictable nature of a completely physical robot interacting with a real room.

Filip Grigorov, Kourosh Darvish, Nandita Vijaykumar

University of Toronto

cs.RO

Submitted: 2026-10-08

Updated: 2026-10-08

The gist: The gist FOCUS, a single-stage PPO framework, trains an actor to use privileged state information during training while automatically regulating whether it collects rollouts from RGB-D or privileged

Key concepts

KL-Guided Modality Switching
This mechanism uses cross-modal policy disagreement to decide the probability of using RGB-D rollouts. It measures how much the actor's actions change when switching from privileged state input to RGB-D input, converting this disagreement into a probability ($\lambda_r$) for collecting an RGB-D rollout. The switch is smoothed over time to ensure regulated exposure.
Representation Alignment Objective
This objective ensures that the representations of the privileged state ($z_{t,s}$) and RGB-D observations ($z_{t,o}$) are compatible during training. It minimizes policy-output KL divergence between these two modalities, forcing them to induce similar action distributions when fused into the actor's input. This alignment is crucial for making training inputs work together.
Single-Stage PPO Framework
FOCUS is structured as a single-stage Proximal Policy Optimization (PPO) framework. It trains an actor that utilizes privileged state information during training but employs a controlled switching mechanism to balance the use of this information against RGB-D data for rollout collection. This structure allows it to manage the trade-off between training efficiency and test-time requirements.

Terminology

Summary

The gist FOCUS, a single-stage PPO framework, trains an actor to use privileged state information during training while automatically regulating whether it collects rollouts from RGB-D or privileged state latents, thereby improving sample efficiency and achieving superior test performance without requiring privileged state at test time.

Problem and Motivation

Reinforcement learning for robotic manipulation is sample-inefficient because RGB-D observations are high-dimensional and noisy. Privileged state information available in simulation can accelerate training but its absence at test time creates a train–test modality gap. The difficulty is addressing the modality gap between privileged-state training and RGB-D-based action selection at test time.

FOCUS Framework Overview

FOCUS is formulated as a single-stage PPO framework that makes privileged-state and RGBD actor representations compatible while adaptively regulating whether on-policy rollouts are generated from privileged-state or RGB-D latents. It combines two complementary ideas: cross-modal representation alignment to reduce mismatch between training-time and test-time inputs, and an online KL-guided switching mechanism that regulates when the actor uses RGB-D rather than privileged-state latents

KL-Guided Modality Switching

FOCUS controls which observation pathway generates each actor rollout by using cross-modal policy disagreement to determine the probability of selecting the RGB-D pathway. It measures how much the actor’s action distribution changes when the privileged-state input is replaced by the RGB-D input, and converts this disagreement into an RGB-D rollout probability λr. The controller state is smoothed at each controller update j via λ(j+1) = (1 − α)λ(j) + α λtarget(r), where λtarget(r) = min[w(r)λmax, λmean(r), λq(r)]. This mechanism keeps the selected modality fixed throughout the rollout, providing regulated RGB-D exposure without mixing privileged-state- and RGB-D-based actions within the same trajectory.

Representation Alignment Objective

FOCUS aligns privileged-state zt,s and RGB-D zt,o representations during training so that their fused actor inputs induce compatible action distributions. The primary term is a policy-output alignment loss Lπ,repr = 1/2KL(πs∥πo) + 1/2KL(πo∥πs). This objective reduces cross-modal policy disagreement and includes auxiliary representation regularizers such as Lalign, Lraw, and Ldec. The full objective is defined as Lrepr = λπ,reprLπ,repr + Lalign + Lraw + Ldec.

Experimental Results

Across five manipulation tasks, FOCUS raises average test success from 0.71 to 0.93 relative to the strongest RGB-D-at-test baseline on each task. Across complete training pipelines, budget-normalized Training Success AUC increases from 0.47 to 0.65. On Pick-and-Place, test success rises from 0.47 to 0.86 and AUC increases from 0.12 to 0.61 (5.0×).

Ablation Insights

A fixed schedule ramp removes KL feedback from the modality switch and increases the RGB-D rollout probability according only to the predefined warmup schedule w(r), while retaining rollout-level modality sampling. FOCUS-simple keeps only the KL-guided switch and image-encoder policy-output KL term, suggesting that controlled visual exposure and image-encoder alignment through policy-output KL are the most reliable contributors.

Conclusion

FOCUS introduces a single-stage PPO framework that regulates whether privileged-state or RGB-D latents generate on-policy actor rollouts during training, while using RGB-D for action selection at test time. The method addresses the privileged-stateto-RGB-D observation gap and is orthogonal to the sim-to-real gap. Future work should evaluate FOCUS on physical robots, broader manipulation suites, and less engineered training signals.

The gist FOCUS, a single-stage PPO framework, trains an actor to use privileged state information during training while automatically regulating whether it collects rollouts from RGB-D or privileged state latents, thereby improving sample efficiency and achieving superior test performance without requiring privileged state at test time.

How it works

FOCUS uses a shared fusion module gψ that combines a modality latent with proprioception and the previous action before it is passed to the actor or critic. The actor receives the selected modality latent, while the critic always receives the privileged-state latent.

KL-guided Modality Switching

For each rollout, it uses cross-modal policy disagreement to determine the probability of selecting the RGB-D pathway. It measures how much the actor’s action distribution changes when the privileged-state input is replaced by the RGB-D input, and converts this disagreement into an RGB-D rollout probability λr. The controller state is smoothed at each controller update j via λ(j+1) = (1 − α)λ(j) + α λtarget(r), where λtarget(r) = min[w(r)λmax, λmean(r), λq(r)]. This mechanism keeps the selected modality fixed throughout the rollout, providing regulated RGB-D exposure without mixing privileged-state- and RGB-D-based actions within the same trajectory.

Representation Alignment Objective

FOCUS aligns privileged-state zt,s and RGB-D zt,o representations during training so that their fused actor inputs induce compatible action distributions. The primary term is a policy-output alignment loss Lπ,repr = 1/2KL(πs∥πo) + 1/2KL(πo∥πs). This objective reduces cross-modal policy disagreement and includes auxiliary representation regularizers such as Lalign, Lraw, and Ldec.

Ablation Insights

A fixed schedule ramp removes KL feedback from the modality switch and increases the RGB-D rollout probability according only to the predefined warmup schedule w(r), while retaining rollout-level modality sampling. FOCUS-simple keeps only the KL-guided switch and image-encoder policyKL term, suggesting that controlled visual exposure and image-encoder alignment through policy-output KL are the most reliable contributors <ref:4.

Improvements for AI systems

  1. Based on Proposition 1, implement KL-guided modality switching to regulate RGB-D exposure using running statistics: When disagreement is at or below the corresponding ε, the exposure factor is one and when above it, exposure decreases inversely with disagreement. This limits RGB-D rollouts when the actor’s action distributions from RGB-D and privileged-state latents disagree, shifting training toward test-time inputs while retaining value estimation.

  2. Integrate cross-modal representation alignment to reduce mismatch: The objective Lπ,repr = 1/2KL(πs∥πo) + 1/2KL(πo∥πs) is used to ensure that privileged-state zt,s and RGB-D zt,o representations induce compatible action distributions. This directly addresses the modality gap by encouraging consistent action selection across them.

  3. Implement auxiliary control regularization to stabilize learning: Augment the PPO objective with LRL = LPPO + λπ(t)Lπ + λV(t)LV, where Lπ is defined as E[KL(πs∥πo)] and LV is the value agreement term, to encourage pathways to induce compatible policy outputs and value predictions.

  4. Utilize a state decoder objective for latent reconstruction: Employ the term Ldec = Lstate dec + λconfLconf, which preserves control-relevant state information by minimizing the difference between reconstructed privileged-state targets and RGB-D targets, ensuring control-relevant state information is preserved in both modalities.

  5. Employ a multi-stage teacher-student baseline for knowledge transfer: Utilize a 3rd stage teacherstudent baseline that trains a privileged-state PPO teacher, distills an RGB-D student from simulator rollouts, and then fine-tunes the student with PPO to leverage pre-trained privileged information effectively.

Sources

Related papers