RoboAug: One Annotation to Hundreds of Scenes via Region-Contrastive Data Augmentation for Robotic Manipulation

arXiv:2602.14032 · cs.RO · Submitted 2026-02-15 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: Today's paper: "RoboAug: One Annotation to Hundreds of Scenes via Region-Contrastive Data Augmentation for Robotic Manipulation".

Dev: The gist The proposed RoboAug framework, a region-contrastive data augmentation framework,

Rosa: First, who's behind it and why it matters.

Paper summary: Rosa: So we're looking at RoboAug: One Annotation to Hundreds of Scenes via Region-Contrastive Data Augmentation for Robotic Manipulation. Basically, this paper claims you don't need massive pretraining or perfect vision recognition anymore if you only get a bounding box annotation from just one image during training #pg2.

Dev: It’s about getting those real robots to work in messy, unpredictable environments without needing mountains of perfect data for every single scenario #pg2. It tackles the problem of environmental interference like background shifts and lighting changes that make policies brittle when they are actually deployed #pg2.

Taro: What I find interesting is that it’s trying to solve this fragility by focusing on what's actually relevant to the task, not just throwing in random data #pg2. It suggests you can build robust skills with much less training data than before five eighty seventy-seven forty-one four eighteen <ref:2602.14032#pg2,5, 80, 77, 41, 4, 18>.

Rosa: Exactly. The core idea is a three-phase process that starts with extracting task-relevant regions and then using generative models to create diverse backgrounds for augmentation #pg2. It’s trying to make the learning process more efficient by focusing on the key parts of the scene #pg3.

Dev: And that region extraction part, they use a lightweight two-step pipeline where they first find key elements in one frame and then use something like SAM2 to turn those bounding box ideas into dense pixel masks called Mtask #pg5. It’s about turning sparse information into something usable for training #pg5.

Taro: That sounds smart because if you can get a good mask, the next step, the semantic data augmentation, gets a lot more powerful because it can composite objects onto truly diverse scenes #pg6. They even use a Large Language Model like ChatGPT to generate five hundred background description templates categorized by material type like wood or stone #pg7.

Rosa: Which means you get this augmented dataset called Dfnl that is much richer than what you could collect manually, and that’s the input for the final part of RoboAug #pg8. It seems they are trying to make the data itself more representative of what a robot will actually see out there.

Dev: Then we get to this region-contrastive policy learning objective where they use a contrastive loss directly in the visual encoder without changing the architecture at all #pg9. During training, for every image from that augmented set Dfnl, they isolate the task-relevant objects using that Mtask mask #pg10.

Taro: So they extract features from those isolated objects to get zobj, and then they use a spatial self-attention mechanism with the full image features to sharpen those object embeddings #pg11. This helps them get a better signal for what matters most in the scene #pg11.

Rosa: And the policy is optimized using this Region-Contrastive Loss, or LRC, which forces the representation to be consistent even when objects are manipulated or when backgrounds change #pg11. It’s essentially telling the model, "focus on this object no matter what it's sitting in" #pg11.

Dev: The experimental validation shows they tested this across over thirty-five thousand rollouts on three different robots: UR-5e, AgileX, and Tien Kung two point zero #pg2. The results are pretty telling because the success rates went up significantly from starting points like zero point zero nine to zero point four seven on the UR-5e robot #pg2 <ref:2602.14032#pg2,0.09 to 0.47 on>.

Taro: And what stands out is that they tested how it handles those triple-factor variations, which includes background shifts, lighting changes, and distractors together #pg2. They achieved average success rates of zero point six seven, zero point four seven, and zero point six zero across those three robots #pg2 <ref:2602.14032#pg2>.

Rosa: That performance on the challenging scenarios is what really sells the idea; it shows that RoboAug outperforms the baseline methods even without any extra augmentation #pg2. Plus, when testing for generalization in totally unseen scenes with mixed backgrounds and lighting, it still showed substantial gains over just using the baseline #pg2.

Dev: Theoretically, they ground this in Rademacher complexity analysis to show how it tightens the generalization bound by increasing the effective sample size through semantic augmentation #pg19. They also reduce the hypothesis space complexity by forcing feature invariance with respect to task-irrelevant regions #pg19.

Taro: So, you’ve got this expansion of data and this reduction of complexity happening at the same time to make the bound smaller in two different directions #pg19. It sounds like a solid way to push generalization without needing massive datasets #pg19.

Rosa: The implication is that for real-world manipulation, we might not need those impossibly large, perfectly labeled datasets if we use a technique that intelligently focuses on the task region and learns invariance through contrastive learning #pg2. This moves us closer to deploying generalist robots in truly unstructured settings #pg2.

Dev: It’s important to remember that they flagged a limitation: the method relies on getting those region annotations from just a single frame during training, which is something you have to manage carefully during the initial setup #pg8.

Taro: That’s fair; you still need that one good starting point for every trajectory image #pg3. But the overall message of RoboAug is that we can get much better results by focusing on task relevance and using contrastive learning to handle the mess of real-world scenes #pg2.

Rosa: So, to wrap up, RoboAug uses a region-contrastive data augmentation framework to boost robotic generalization across varied scenes by only needing single image annotations #pg2. It’s a big step toward making robots that can actually handle the messy world without needing perfect training data #pg2.

Conclusion: Rosa: So, to wrap up this whole thing about RoboAug, we're looking at how they took just one bounding box from a single image and turned it into hundreds of training scenes using this region-contrastive augmentation idea #pg2.

Dev: It really boils down to minimizing the need for those massive pretraining sets and perfect visual recognition because you don't have to nail every single scene perfectly anymore #pg2.

Taro: What they’ve done is focus the learning process on the parts of the world that actually matter for a specific task, rather than just throwing random data at it #pg3.

Rosa: They used this method to train robots like UR-5e and AgileX, and the results showed a big jump in success rates when dealing with messy environments #pg2 <ref:2602.14032#pg1>.

Dev: The numbers they shared are interesting because you saw success rates climb quite a bit on the AgileX robot, from about sixteen percent up to sixty percent #pg2.

Taro: And what’s really compelling is that this works well even when the lighting changes drastically or there are distractors in the background #pg2.

Rosa: So, the big question is whether this approach actually translates outside of a controlled lab setting, and how long these policies stay robust in those real-world scenarios #pg2.

Dev: That’s what we need to figure out next—does it hold up when the loop rate gets stressed or when unexpected failure modes pop up during actual operation #pg2.

Taro: We have to look at how this system handles things that aren't perfectly described in the training data, because that’s where real autonomy is tested #pg2.

Rosa: That leads us into the next part of our talk about how this framework actually performs when you push it past those initial success rates #pg2.

Beijing Innovation Center of Humanoid Robotics · Computation and Information Technology Department of Technical University of Munich · City University of Hong Kong · School of Mechanical Engineering and Automation at Beihang University · State Key Laboratory of Multimedia Information Processing, School of Computer Science at Peking University · School of Advanced Manufacturing and Robotics at Peking University

cs.RO

Submitted: 2026-02-15

Updated: 2026-10-08

Code: https://github.com/CVHub520/X-AnyLabeling

Project page: https://x-roboaug.github.io

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 90/100

The gist: The gist The proposed RoboAug framework, a region-contrastive data augmentation framework, significantly minimizes reliance on large-scale pretraining and perfect visual recognition by requiring only

Key concepts

Robust Task-Relevant Region Extraction
This phase uses annotations from just one image to create dense pixel-level masks for important objects across all trajectory images. It starts with a lightweight matching strategy on the anchor frame and then uses a tracking framework like SAM2 to turn these sparse bounding boxes into detailed masks, ensuring the model focuses only on what matters for the specific task.
Semantic Data Augmentation
This step generates diverse full-scene backgrounds using generative models guided by Large Language Models (LLMs). It creates 500 background templates categorized by material (like wood or stone) and seamlessly composites the foreground objects onto these new scenes. This expands the training data significantly, helping the model learn to ignore complex environmental variations.
Region-Contrastive Policy Learning
This involves optimizing a policy using a contrastive loss that directly modifies the visual encoder. The system isolates task-relevant objects and compares their features against full image features, enhanced by spatial self-attention. This forces the model to learn representations that are invariant to irrelevant background changes while remaining sensitive to the target object.
Generalization Error Bound Analysis
Theoretical analysis uses Rademacher complexity to prove why RoboAug works. It shows that semantic augmentation increases the effective sample size, while region-contrastive learning reduces the hypothesis space complexity by enforcing feature invariance. This dual mechanism tightens the generalization bound, leading to more robust performance on unseen scenes.

Terminology

Summary

The gist The proposed RoboAug framework, a region-contrastive data augmentation framework, significantly minimizes reliance on large-scale pretraining and perfect visual recognition by requiring only bounding box annotation of a single image during training <ref:2602.14032#pg8>.

How it works

RoboAug is designed to enhance policy generalization in real-world robotic manipulation by synergizing three key technical phases: robust task-relevant region extraction, semantic data augmentation, and region-contrastive policy learning <ref:2602.14032#pg2>. The framework addresses the challenge of environmental interferences such as complex background variations, drastic lighting changes, and distractors <ref:2602.14032#pg2>.

The first phase involves robust task-relevant region extraction which generates semantic masks across all trajectory images using annotations from only a single frame <ref:2602.14032#pg3>. This is achieved through a lightweight, two-step extraction pipeline that utilizes a one-shot region matching strategy to locate key elements in the anchor frame of every trajectory <ref:2602.14032#pg5>. Subsequently, these spatial annotations are propagated across subsequent frames using a tracking-and-segmentation framework like SAM2 to transform sparse bounding box priors into dense pixel-level masks Mtask <ref:2602.14032#pg5>.

The second phase is semantic data augmentation, which employs a generative model to synthesize diverse fullscene backgrounds and seamlessly composite the foreground regions onto them <ref:2602.14032#pg6>. This process involves leveraging a Large Language Model (ChatGPT) to generate a rich set of descriptive prompts, constructing 500 background description templates categorized into material types like wood (58%) and stone (35%) <ref:2602.14032#pg7>. The resulting augmented dataset Dfnl is then used for the final phase <ref:2602.14032#pg8>.

Region-Contrastive Policy Learning

The third phase introduces a region-contrastive policy learning objective that integrates a contrastive loss directly into the visual encoder without architectural modifications <ref:2602.14032#pg9>. During training, for each image sampled from Dfnl, masked images Iobj are generated by isolating task-relevant objects using the mask Mtask <ref:2602.14032#pg10>. These inputs are processed by a shared visual encoder E(·) to extract object feature embeddings zobj = E(Iobj) <ref:2602.14032#pg10>.

To enhance the signal, features from the full image z = E(I) are used to accentuate salient information in zobj through a spatial self-attention mechanism <ref:2602.14032#pg11>. The final attentive features are calculated as aatt = sigmoid(A(z) ⊙ z), zatt = aatt ⊙ zobj <ref:2602.14032#pg11>. The policy is then optimized using the Region-Contrastive Loss (LRC) which enforces representation invariance on manipulated objects while encouraging robustness to background variations <ref:2602.14032#pg11>.

Experimental Validation and Results

RoboAug was validated through extensive real-world experiments spanning over 35k rollouts on three robots: UR-5e, AgileX, and Tien Kung 2.0 <ref:2602.14032#pg2>. The evaluation rigorously decouples environmental variables, testing background shifts, lighting variations, and distractors both individually and largely in composition <ref:2602.14032#pg2>.

Empirical results demonstrate that RoboAug significantly outperforms state-of-the-art data augmentation baselines <ref:2602.14032#pg2>. Specifically, the success rates increased from 0.09 to 0.47 on UR-5e, from 0.16 to 0.60 on AgileX, and from 0.19 to 0.67 on Tien Kung 2.0 <ref:2602.14032#pg2>.

The method shows superior performance in challenging triple-factor variation settings, achieving average success rates of 0.67, 0.47, and 0.60 across the three robots, significantly outperforming the leading baseline <ref:2602.14032#pg2>. Furthermore, when evaluating generalization capabilities in unseen scenes featuring diverse combinations of backgrounds, distractors, and lighting conditions <ref:2602.14032#pg2>, RoboAug achieves substantial gains over the baseline without augmentation <ref:2602.14032#pg2>.

Theoretical Justification

The theoretical analysis provides a foundation for RoboAug by analyzing the generalization error bound using Rademacher complexity <ref:2602.14032#pg18>. The method improves generalization through two synergistic mechanisms: increasing the effective sample size via semantic augmentation and reducing the hypothesis space complexity via region-contrastive learning <ref:2602.14032#pg19>.

Semantic data augmentation expands the original expert dataset of size N to a significantly larger augmented dataset of size Ntotal = N + Naug by generating diverse xscen while preserving xtask <ref:2602.14032#pg19>. This expansion leads to a tighter generalization bound because the Rademacher complexity for neural networks typically scales with O(1/√N), and Ntotal ≫ N <ref:2602.14032#pg19>.

The Region-Contrastive Learning (RCL) objective further tightens the generalization bound by enforcing feature invariance with respect to task-irrelevant regions xscen, effectively regularizing the search space towards Hinv <ref:2602.14032#pg19>. This results in a tighter bound because RNtotal (Hinv) ≤ RNtotal (H) <ref:2602.14032#pg19>.

The final generalization bound is then expressed as R(πRCL) ≤ Rˆ(πRCL)+2LlRNtotal (Hinv)reduced complexity +c s log(1/δ) 2Ntotal <ref:2602.14032#pg19>. This demonstrates that RoboAug achieves robust generalization by simultaneously reducing the error bound from two directions: expanding the denominator of the complexity term via Semantic Augmentation (N → Ntotal) and reducing the Rademacher complexity via Region-Contrastive Learning (H → Hinv) <ref:2602.14032#pg19>.

Conclusion

RoboAug introduces a data augmentation framework designed to enhance robotic generalization across diverse and unseen scenes <ref:2602.14032#pg2>. By utilizing generative models for semantic augmentation and integrating a plug-and-play region-contrastive loss, RoboAug effectively guides the model to focus on task-relevant regions <ref:2602.14032#pg2>. Extensive real-world validation, comprising over 35k trials on UR-5e, AgileX, and Tien Kung 2.0 robots, demonstrates that RoboAug consistently outperforms state-of-the-art baselines <ref:2602.14032#pg2>. These results highlight the superior effectiveness and robustness of RoboAug in complex real-world manipulation tasks >.

--- Page 1 ---

The gist The proposed RoboAug framework, a region-contrastive data augmentation framework, significantly minimizes reliance on large-scale pretraining and perfect visual recognition by requiring only bounding box annotation of a single image during training <ref:2602.14032#pg2>.

Improvements for AI systems

  1. Robust Policy Generalization via Region-Contrastive Loss: The system can achieve robust generalization by reducing the hypothesis space complexity via region-contrastive learning, which enforces feature invariance with respect to task-irrelevant regions (Corollary 6.2.1). This allows the policy to maintain precision even when facing diverse visual perturbations.

  2. Data Augmentation for Semantic Diversity: The framework can synthesize diverse training data by leveraging generative models to create varied environments, as it effectively guides the model to focus on task-relevant regions (Abstract) and expands the dataset by synthesizing a complete, coherent background image rather than filling in missing regions (Section III.C).

  3. One-Shot Semantic Mask Extraction: The system can generate pixel-level masks for task-relevant regions using a one-shot region matching strategy combined with foundation models, ensuring that every frame is equipped with semantic masks Mtask corresponding to the task-relevant elements (Section III.B).

  4. Improved Object Detection Precision: The system can enhance object detection performance by outperforming baselines, as demonstrated by achieving a 34.6% improvement over GroundingDINO on the RoboAug-D dataset for specific categories like Bun, where baselines struggle to exceed 0.10 mAP (Section IV.A).

  5. Enhanced Robustness to Distractors and Illumination: The system exhibits superior resilience against environmental shifts, as evidenced by its ability to maintain high success rates even when tested under Dual-Factor Variation (unseen backgrounds and object clutter) and Single-Factor Generalization: Robustness to Illumination Variation (Section IV.D).

Sources

Related papers