CoRE-VLA: Towards Scalable and Robust Vision-Language-Action Modeling via Conditional Routing of Experts

summary

The gist

The gist Vision-language-action (VLA) models have advanced generalist robotic manipulation, yet real-world deployment reveals a fundamental challenge: robots are equipped with diverse and

In short

CoRE-VLA addresses brittle Vision-Language-Action (VLA) models by making action generation context-conditioned sparse computation. It introduces Conditional Routing of Experts, which allows robots to selectively use auxiliary sensors when available and fall back reliably when they are missing or absent. This enables graceful degradation and improves performance across diverse tasks.

Key concepts

Context-Conditioned Sparse Computation: Instead of using a single dense computation path for all actions, CoRE-VLA treats action generation as sparse computation guided by context. This means the model only activates the specific parts of its network that are relevant based on the current situation, such as which sensors are present and what the task intent is. This makes it efficient and adaptable.
Sensor Availability Gating: This mechanism uses sensor availability to control which specialized experts are used. If an auxiliary sensor, like depth, is present, its corresponding expert is activated. If the sensor is missing or unavailable during deployment, that expert is masked off. This allows the policy to gracefully degrade and rely only on available information without needing retraining.
Task Intent Routing: The framework routes action-side representations to experts based on task intent derived from the VLM's text embeddings. This ensures that the computation focuses on what is relevant for the current goal. By routing based on intent, the model can specialize its action generation for different long-horizon subgoals effectively.

Terminology used across episodes

Reinforcement Learning (RL) — explained in 25 episodes
Federated Learning — explained in 16 episodes
Retrieval-Augmented Generation (RAG) — explained in 16 episodes
Interpretability — explained in 15 episodes
Sample Complexity — explained in 15 episodes
Generalization — explained in 13 episodes

This episode discusses

The paper

CoRE-VLA: Towards Scalable and Robust Vision-Language-Action Modeling via Conditional Routing of Experts · Read on arXiv

Haozhe Zhang, Sixian Li, Yifei Zhang, Zezheng Huai, Hao Chen, Chunhua Shen, Jingjing Gong, †Xipeng Qiu

Zhejiang University · Shanghai Innovation Institute (Shanghai) · Fudan University

Vision-language-action (VLA) models have advanced generalist robotic manipulation, yet real-world deployment reveals a fundamental challenge: robots are equipped with diverse and heterogeneous sensor configurations, auxiliary sensors can fail unexpectedly during operation, and different robot embodiments often lack certain sensors by design. A unified policy that can exploit auxiliary perceptual inputs when available while remaining reliable under sensor absence, whether incidental or by design, is therefore essential for practical deployment. However, existing VLA policies couple action generation to a fixed sensor set through shared dense computation, making them brittle when sensors are missing and limiting their ability to specialize across diverse tasks and long-horizon behaviors. We propose CoRE-VLA, a scalable and robust VLA framework that formulates action generation as context-conditioned sparse computation. Sensor availability gates modality-specialized experts, enabling graceful degradation under missing sensors without retraining. Task intent further routes action-side representations to task-relevant experts, improving specialization across diverse tasks and long-horizon subgoals. While the framework is designed to accommodate different auxiliary sensors, we focus on depth as a representative and practically important auxiliary modality in our experiments. Experiments on LIBERO, RoboCasa GR1 Tabletop, and real-world dual-arm manipulation show that CoRE-VLA achieves strong results on long-horizon and multi-task benchmarks, and outperforms both a dense-action-generator ablation and a strong pretrained VLA baseline, including in zero-shot generalization to unseen scenarios. Modality analysis shows that CoRE-VLA can exploit auxiliary depth when available while remaining robust when depth is unavailable during deployment.

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: Today's paper: "CoRE-VLA: Towards Scalable and Robust Vision-Language-Action Modeling via Conditional Routing of Experts".

Dev: The gist Vision-language-action (VLA) models have advanced generalist robotic manipulation, yet real-world deployment reveals a fundamental challenge: robots are equipped with diverse and heterogeneous sensor configurations,

Rosa: First, who's behind it and why it matters.

Paper summary: Rosa: So we're looking at this paper called "CoRE-VLA: Towards Scalable and Robust Vision-Language-Action Modeling via Conditional Routing of Experts," and the main idea is that robots in the real world have all sorts of different sensors, some might fail, and some robots just don't have certain sensors built in.

Dev: Exactly. The core problem they're tackling is how to make a unified policy that can use those extra sensory inputs only when they're there, without breaking when they aren't available at deployment time.

Taro: So the authors are saying that existing VLA policies couple action generation to a fixed set of sensors through this shared dense computation path, which makes them brittle if those sensors go missing and limits their ability to handle different tasks well.

Rosa: That’s right. They propose CoRE-VLA as a way to fix that by formulating action generation as context-conditioned sparse computation instead of one fixed dense path for everything.

Dev: It uses task intent and sensor availability as two main pieces of context to guide this routing mechanism, which allows it to handle missing sensors gracefully without needing retraining.

Taro: So the framework splits the action representations into general experts and modality-specialized experts, and it routes them based on both what we want to do with a task and what sensors are actually present.

Rosa: That’s the key mechanism they focus on, using conditional routing guided by task intent and sensor availability to select the most relevant parts of the model.

Dev: And they use a specific process where intent conditions route action-side representations, and then those representations get sent to experts that are specialized for that particular modality or general patterns.

Taro: I think what’s interesting is how it prevents the model from just relying too much on one sensor input by using modality dropout and availability-based expert masking when a sensor isn't there.

Rosa: Right, so they treat the auxiliary sensor availability as a condition at the action generator level, which lets it exploit those depth cues when available while falling back to an RGB language path when depth is gone.

Dev: They train it using a flow matching objective over action chunks, and they even have this selection regularizer to make sure the average representation selection mass matches some target capacity ratio rho.

Taro: I see how that helps with efficiency too, because the paper shows that by controlling that capacity ratio rho, you can keep the number of computations low for smaller inputs while still keeping the expert computation scaled correctly.

Rosa: It really makes sense when you think about deployment; being able to rely on depth only when you have it but not crashing when it's missing is a big deal for real-world robotics.

Dev: So, to sum up this paper called "CoRE-VLA: Towards Scalable and Robust Vision-Language-Action Modeling via Conditional Routing of Experts," they claim they are the first to explicitly formulate how to deploy unified VLA policies under sensor heterogeneity and missing auxiliary sensors as a joint architecture-and-training problem.

Taro: The implication for us is that scalable VLA policy learning shouldn't just be about scaling up the perception backbones, but it also has to make action generation adaptive to both the task intent and the physical sensory context.

Rosa: That’s a big picture idea. It suggests that if we want these models to work outside of a perfect lab setting, they need this kind of dynamic way of allocating their computational resources based on what's actually happening around them.

Conclusion: Rosa: So we're wrapping up this look at CoRE-VLA, which is titled "Towards Scalable and Robust Vision-Language-Action Modeling via Conditional Routing of Experts."

Dev: It’s a framework that handles sensors not by just stacking everything together, but by making the action generation sparse and context-conditioned.

Taro: Basically, instead of one big dense computation path for every single sensor setup, they route the computation based on two things: what the task is trying to do and what sensors are actually available.

Rosa: So it’s about being smart about when to use which brain parts depending on whether you have the right inputs or not.

Dev: The authors show that this approach lets the robot degrade gracefully when a sensor fails, meaning it doesn't just crash or give a bad action because one input is missing.

Taro: It changes things for autonomy because it means we don't have to build every single robot model from scratch for every possible sensor combination.

Rosa: Right, so if you’re looking at how this scales to different robots—like the LIBERO or those dual-arm setups they tested—it shows it can actually handle long-horizon tasks better than just a standard dense generator.

Dev: They show strong results on multi-task benchmarks, and when you look at real-world testing, it beats existing baseline models, even in zero-shot scenarios where the robot hasn't seen that specific setup before.

Taro: The numbers they shared suggest that this routing mechanism doesn't just give a slight bump; it really improves how the AI allocates its processing power across a long sequence of actions.

Rosa: It’s about making the policy more adaptive to both what you’re trying to achieve and what the physical world is actually showing you at any given moment.

Dev: So, this moves us past just building bigger models toward building models that are robust enough to operate in messy, real-world environments where things go wrong.

Taro: And it opens the door for future work on how we can make those task-intent signals even richer so the routing gets even smarter.

More episodes

← Home