ResCue: Residual Spatial Cueing for Language-Conditioned Imitation Learning

arXiv:2606.25360 · cs.RO · Submitted 2026-06-24 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: Today's paper: "ResCue: Residual Spatial Cueing for Language-Conditioned Imitation Learning".

Dev: End-to-end Vision-Language-Action (VLA) models often suffer from an alignment bottleneck where semantic reasoning and spatial control are coupled, leading to poor target disambiguation in data-constrained imitation learning.

Rosa: First, who's behind it and why it matters.

Title and authors: Rosa: So, moving beyond the setup, the core summary of this paper is about how they propose decoupling semantic reasoning and geometric grounding to solve that alignment bottleneck in standard VLA models. They argue that monolithic models struggle when they have to simultaneously learn abstract language meanings and precise spatial control from limited data.

Dev: Essentially, the paper summarizes their method as translating high-level natural language instructions into explicit Spatial Visual Prompts, or SVPs, which are then fed into a feature-level fusion mechanism inside the continuous action generator.

Taro: I see how that translates to something actionable; they take words and turn them into a spatial map that guides the robot's movement directly through its internal features.

Rosa: Right, and the key part is this intermediate feature-level fusion where they add this mask information element-wise to the primary visual backbone's intermediate features, which provides explicit spatial gradient guidance during fine-tuning.

Dev: That mechanism is what makes it different from previous attempts because it acts like a rigid structural bias rather than relying on complex learnable gating mechanisms for how much of that prompt to use.

Taro: So, the summary emphasizes that this approach avoids the optimization instability often associated with low-data regimes because it forces immediate spatial attention.

Rosa: It really focuses on providing targeted spatial gradient guidance during fine-tuning while completely avoiding input-level domain shifts and ensuring stable model convergence, which is a huge win for imitation learning.

Dev: If we look at the practical application, the paper shows that this architecture significantly improves success rates on highly ambiguous tasks when tested on benchmarks like RoboTwin two point zero <ref:2606.25360#pg0>.

Taro: The summary highlights how SVP-IL dramatically improves success rates on highly ambiguous tasks using as few as fifty to one hundred demonstrations, which speaks directly to data efficiency.

Rosa: It confirms that this decoupled architecture significantly outperforms standard end-to-end models and pure visuomotor baselines, showing improved performance with very little training data.

Dev: So the summary boils down to: they decouple semantics from geometry, use vision-language tools to generate spatial priors, and inject those priors directly into the policy via feature fusion for stable control.

Taro: It’s a clear statement on how architectural separation can lead to more robust learning in embodied AI systems that are operating in complex physical environments.

Rosa: And they show it works well even when dealing with visual clutter and variable lighting, which is important for real-world deployment scenarios.

The paper's summary: Dev: Now let's talk about what the paper specifically suggests as its improvements, and it seems centered around refining that initial decoupling strategy. They focus on making the spatial prompting mechanism more robust and efficient.

Rosa: They suggest a few key areas for improvement, starting with addressing how the system handles challenging visual properties, like transparent surfaces, because right now it's bounded by the upstream vision-language model's capability in those cases.

Dev: That means they think we need to find a way to make the spatial cueing more resilient even when the input image itself is ambiguous or contains things that confuse the initial perception step.

Taro: I agree, and another point they flag is their reliance on explicit object-centric instructions for initializing those prompts, which limits how much implicit intent the system can infer on its own.

Rosa: They suggest exploring implicit inference of target objects from more abstract user intent instead of just relying on specific object names to start the process.

Dev: That would be a big step toward making the system more general, but I have to wonder how we'd design that translation pipeline without losing the precision we got from the SAM three mask generation <ref:2606.25360#pg0>.

Taro: Also, they suggest substituting models like SAM three with more lightweight segmentation models to optimize inference efficiency, which is definitely something we need for practical deployment <ref:2606.25360#pg0>.

Rosa: So, while the core idea is strong, they are acknowledging that the current implementation isn't fully general because it's tethered to specific object names and potentially computationally heavy.

Dev: I think optimizing the prompt generation step is crucial because if generating those spatial priors becomes too slow or complex, it defeats the purpose of having a low-latency control loop.

Taro: If they can make the prompt generation lighter, it would significantly enhance the system's deployability in real-world scenarios where speed matters for fast reactions.

Rosa: So, these improvements point toward making the system more robust against visual noise and more efficient computationally while expanding its ability to handle diverse instructions.

Dev: It sounds like they are focused on moving from a specialized, high-performance setup toward something that’s more generalized and practical for continuous operation.

The paper's improvements: Rosa: So, wrapping up the discussion on "ResCue: Residual Spatial Cueing for Language-Conditioned Imitation Learning," the main implication is that by explicitly decoupling semantic reasoning from spatial control, we can build systems that are much more stable and data-efficient when learning complex manipulation skills.

Dev: I agree; this separation helps mitigate the alignment bottleneck in VLA models, leading to policies that have stronger inductive biases and less instability during fine-tuning.

Taro: From an autonomy research standpoint, this means we can expect robots to perform tasks with much higher precision under conditions where the visual input is messy or ambiguous.

Rosa: In short, it gives us a way to achieve robust manipulation using far fewer demonstrations than previously required for comparable performance on challenging tasks.

Dev: We're talking about a system that can reliably execute instructions in cluttered environments without needing massive datasets to learn those spatial relationships from scratch.

Taro: It’s about building systems that are more capable of handling the real-world mess, which is what we need for true autonomy outside the lab.

Rosa: So, "ResCue: Residual Spatial Cueing for Language-Conditioned Imitation Learning" demonstrates a powerful technique for injecting explicit visual priors into continuous control loops to guide robot actions effectively.

Dev: It's a solid architectural contribution because it provides targeted spatial gradient guidance that doesn't require additional optimization phases for attention weights.

Taro: I think the focus on data efficiency is the most practical aspect here, showing that structural improvements can yield significant performance gains with minimal training examples.

Conclusion: Rosa: So, to wrap up our discussion on "ResCue: Residual Spatial Cueing for Language-Conditioned Imitation Learning," this paper shows how decoupling semantic reasoning from geometric grounding through explicit spatial prompts really helps stabilize learning in these complex VLA systems.

Dev: I agree, Rosa; the way they handle that intermediate feature fusion, adding the prompt to the backbone features element-wise, seems like a very stable way to inject those spatial priors without messing up the original visual distribution.

Taro: I’m interested in how this stability translates when things go wrong in physical space; for instance, what happens when the world presents something truly unexpected that isn't covered by the initial object prompt?

Rosa: That’s a fair question, Taro; their limitations point out that performance can be bounded by the upstream vision-language model when it encounters challenging visual properties like transparent surfaces.

Dev: Exactly, and they also noted that the current approach relies on explicit, object-centric instructions for prompt initialization, which limits how much implicit intent the system can infer on its own.

Taro: So while it’s great for known objects, the paper suggests future work should look at ways to move toward implicit inference of target objects from more abstract user intent instead of just specific object names.

Rosa: It sounds like they’re aiming for a more general system that doesn't need perfect prior knowledge about every single object before it can start acting.

Dev: And on the computational side, they acknowledge the substantial overhead of using models like SAM three to generate those prompts, so making those spatial priors lighter is definitely something they’ll need to focus on for wider deployment.

Taro: If we can make that prompt generation more lightweight, it would dramatically enhance the system's deployability in real-world scenarios where fast reactions are essential for physical interaction.

Rosa: It really shows that structural separation of concerns, even with current limitations like the reliance on specific object names, provides a significantly stronger inductive bias for data efficiency.

Dev: I see it as a strong architectural choice because it avoids those complex, parameterized gating mechanisms that often introduce instability when you’re training on very little data.

Taro: Ultimately, "ResCue: Residual Spatial Cueing for Language-Conditioned Imitation Learning" provides a solid framework for making VLA policies more robust by explicitly separating what the robot needs to *understand* from what it needs to *do*.

Tsinghua University of Technology and Science and Huawei Technologies Co. Ltd. · National University of Singapore

cs.RO

Submitted: 2026-06-24

Updated: 2026-10-03

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 87/100

The gist: End-to-end Vision-Language-Action (VLA) models often suffer from an alignment bottleneck where semantic reasoning and spatial control are coupled, leading to poor target disambiguation in

Key concepts

Spatial Visual Prompts (SVP)
These are explicit geometric masks generated by a vision-language model to represent the target object described in a language instruction. Instead of relying on implicit visual understanding, the SVP provides a clean, pure spatial representation of what needs to be targeted, effectively stripping away distracting visual noise.
Feature-Level Fusion Mechanism
This is a lightweight side-stream network that takes the sparse SVP mask and dynamically encodes it into dense feature maps matching an intermediate layer of the main visual encoder. This allows the spatial guidance to be added directly to the primary visual features, ensuring uncorrupted gradient flow without needing complex optimization.
Decoupling Semantics and Geometric Grounding
The core idea is separating what the language means (semantics) from where that object is located in space (geometry). By explicitly translating text into a geometric prompt, the system reduces the difficulty for the continuous control policy, allowing it to focus purely on executing precise spatial movements.
Diffusion Policy (DP)
This is a standard architecture used by ResCue to generate continuous actions. The enhanced visual features derived from ResCue are concatenated with robot states and text embeddings, serving as global conditioning input for the noise prediction network. This guides the policy to produce robust action sequences based on the provided spatial priors.

Terminology

Summary

End-to-end Vision-Language-Action (VLA) models often suffer from an alignment bottleneck where semantic reasoning and spatial control are coupled, leading to poor target disambiguation in data-constrained imitation learning. The proposed SVP-IL architecture decouples these components by translating language instructions into explicit Spatial Visual Prompts (SVP), which are then injected directly into the continuous action generator via a feature-level fusion mechanism, providing uncorrupted spatial gradient guidance.

The gist

SVP-IL is a decoupled perception-control architecture that mitigates the semantic-spatial alignment bottleneck in standard VLAs by translating language instructions into spatial visual prompts for imitation learning.

Decoupling Semantics and Geometric Grounding

The paper addresses the inherent coupling in monolithic end-to-end VLA models, which struggle to implicitly learn the mapping from abstract semantic tokens to precise physical coordinates under low-data regimes. The core idea is a paradigm shift that posits explicit separation of semantic reasoning and geometric grounding. This is achieved by leveraging vision-language foundation models to parse instructions into zero-shot geometric masks, which are then translated into explicit Spatial Visual Prompts (SVP). This process reduces the cognitive load on the continuous control policy from implicit visual grounding to pure spatial execution.

Translating Language into Spatial Visual Prompts (SVP)

The framework employs a two-stage pipeline to generate the SVP. First, a Large Language Model (LLM) is utilized as an instruction parser to perform semantic reasoning and extract the core name of the target object, denoted as 'c'. Second, an off-the-shelf vision-language foundation model, specifically SAM 3 [3], is used to act as a zero-shot semantic extractor. Given an RGB observation It and the extracted object name c as a text prompt, this model generates the binary geometric mask Mt:

Mt = SAM 3(It, c) (4). This mask effectively translates complex language instructions into a pure geometric representation of the target object by stripping away texture, lighting, and distractor noise.

Intermediate Feature-Level Fusion Mechanism

A critical challenge addressed is how to effectively inject these spatial priors into a continuous action generator without corrupting the original visual distribution. The authors introduce a direct feature-level fusion mechanism that acts as a lightweight side-stream Convolutional Neural Network (Side-CNN), denoted as Φprompt. This stream dynamically encodes the sparse mask Mt into a dense feature map F(i) prompt that matches the spatial dimensions of an intermediate layer i in the base encoder:

F(i) prompt = Φ(i) prompt(Mt; θprompt) (5). The fused feature F(i), which is then added element-wise to the primary visual backbone's intermediate feature F(i), is computed as:

F(i) = Φ(i) rgbIt + F(i) prompt (6). Empirically, this direct addition strictly outperforms complex, parameterized gating mechanisms and provides immediate, uncorrupted spatial gradient guidance without requiring additional optimization phases for attention weights.

SVP-IL Instantiation and Performance

The SVP-IL framework instantiates its continuous action generator using the standard Diffusion Policy (DP) architecture [7]. The geometrically enhanced visual features F are flattened and concatenated with the proprioceptive states of the robot qt and the original text embedding l. This combined vector serves as the global conditioning input for the 1D U-Net noise prediction network of the DP [7], guiding robust action chunking with spatial priors. Extensive experiments on RoboTwin 2.0 demonstrate that SVP-IL significantly outperforms state-of-the-art VLAs and pure visuomotor baselines, achieving an average success rate of 67.8% on highly ambiguous language-conditioned tasks using as few as 50 to 100 demonstrations. Furthermore, the framework shows superior data efficiency compared to Domain Randomization techniques.

Real-World Validation

The system is deployed on a physical Aloha-AgileX robot in unstructured environments, validating its robustness and data efficiency. In real-world evaluations against dense clutter and variable lighting, SVP-IL achieves a dominant 60.0% average success rate, significantly outperforming the vanilla DP baseline (28.3%) and the language-conditioned model π0 (31.7%). This confirms that explicitly decoupling semantic reasoning from spatial grounding provides a significantly stronger, more data-efficient inductive bias than standard Domain Randomization techniques in low-data regimes.

Limitations and Future Work

The limitations identified include performance being bounded by the upstream vision-language model when challenging visual properties (like transparent surfaces) are encountered, the current reliance on explicit, object-centric instructions for prompt initialization, and the substantial computational overhead of using models like SAM 3. Future work is suggested to explore implicit inference of target objects from abstract user intent and substituting SAM 3 with more lightweight segmentation models to optimize inference efficiency.

Improvements for AI systems

Here are the specific improvements that can be made to existing AI systems by implementing the SVP-IL framework, and what these improved systems will be capable of doing:

  1. The ability to perform high-precision, language-conditioned manipulation in highly ambiguous, cluttered environments using extremely limited demonstrations (as few as 50–100).

  2. The capacity for a robotic system to reliably disambiguate target objects based on complex natural language instructions even when multiple visually similar distractors are present (e.g., distinguishing between two red apples in a cluttered basket).

  3. The development of robust sim-to-real transfer capabilities, ensuring that policies trained with minimal real-world data generalize effectively to physical hardware in unstructured environments, as demonstrated by the robust performance on the Aloha-AgileX robot.

  4. The creation of a more efficient and stable end-to-end Vision-Language Policy (VLA) architecture by decoupling semantic reasoning from spatial control, thereby mitigating the semantic-spatial alignment bottleneck that causes instability in low-data regimes.

  5. The enhancement of visual feature extraction capabilities within robotic policies, as the intermediate feature fusion mechanism allows the primary visual backbone to retain its pre-trained low-level texture filters while being guided by explicit spatial priors, leading to more stable learning during fine-tuning.

  6. The creation of a plug-and-play framework for vision systems where complex language instructions are translated into explicit binary geometric masks (Spatial Visual Prompts), allowing the policy to focus solely on precise execution rather than implicit cross-modal alignment.

Sources

Related papers