Learning Where it Matters: Autonomous Data Collection and Offline Reinforcement Learning for Local Contact-Rich Manipulation

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: Hi, I'm Rosa.

Dev: And I'm Dev, here with Taro.

Rosa: We're looking at the paper: "Learning Where it Matters".

Dev: Here's the gist of the paper. The gist: Learned policies trained end-to-end on large datasets often remain brittle in high-precision tasks and struggle with generalization, which this work addresses by proposing an automated data-collection scheme in combination with offline deep reinforcement learning for the critical segment of contact-rich tasks Key Insight and Problem Statement The limitations of end-to-end robot learning stem from a lack of structure and focus in data collection The authors argue that at least part of this gap is not merely a matter of scale, but of what data is collected and how tasks are structured Specifically, they show that high-precision tasks usually fail at the contact-rich, critical segment—such as an insertion with tight clearance They propose to exploit this structure explicitly by densely collecting data where it matters—at the critical segment Autonomous Data Collection Scheme The proposed scheme involves combining dense data collection with offline deep reinforcement learning (DRL) for policy learning This scheme eliminates the constant dependency on a human teleoperator The data collection process begins by replaying a single demonstration until the critical segment of the task is reached The exploration policy used for dense data collection is defined as πexplore(st) = at = a ∼ U(A) with probability p a∗t with probability 1 − p The greedy action is defined as a∗t = v · (sg + ∆sgrasp) − st∥(sg + ∆sgrasp) − st∥2 The task reward rt is a sparse reward determined by the distance of state st to the final state of the hardcoded trajectory sg: rt = Rsuccess · 1(∥st − (sg + ∆sgrasp)∥2 ≤ ϵ Policy Learning and Deployment The learned policy πθ is obtained by combining data collection with offline DRL The learning algorithm builds on top of SAC with improvements for sample efficiency Interaction during critical interactions employs hybrid force-position control The central claim is that traditional planning combined with data collection and learning for the critical segment leads to high success rates and robust policies The method performs well in out-of-distribution scenarios where end-to-end approaches struggle A key finding is that the sequential pipeline achieves consistently ≥ 90% SR Ablation and Robustness The ablation study on data collection parameters shows that successful policies require dense data coverage of the critical segment Policies trained on little exploration (p <= 0.3) achieve low success rates Multi-modal sensing for high success rates shows that combining a wrist-mounted F/T sensor with a single wrist camera achieves 100 % success in simulation The method maintains high success rates even under OOD scenarios, unlike end-to-end methods By adopting a modular approach with targeted data collection, the paper concludes that focusing collection on tasks and task-segments that are otherwise difficult to solve is key to robust generalizable manipulation The paper demonstrates that targeted reliance on robot data and structured task decomposition are key enablers of robust generalizable manipulation Limitations The paper notes limitations regarding task diversity, stating that sequential boundaries are harder to define for tasks like t-shirt folding Hardware precision is also a limitation, as the method benefits from hardware that precisely reaches target poses for waypoint tracking and policy switching<ref:2607.

Paper summary: Rosa: So we just covered the main idea of this paper, which is that traditional end-to-end learning struggles with precision tasks because they lack focus in data collection. The authors argue the solution is to leverage dense data collection only for the critical segment of contact-rich tasks.

Dev: Exactly, and they show how combining that collected data with offline deep reinforcement learning creates a policy pi theta that performs well when sequentially chained with off-the-shelf pose estimation and motion planning.

Taro: I like the idea of replaying a single demonstration until the critical segment is reached before starting dense data collection using a greedy-random exploration policy. That seems like a structured way to focus the data gathering effort.

Rosa: It’s about eliminating that constant dependency on a human teleoperator's skill, and also removing the need for online policy updates during deployment, which simplifies things a lot for real-time control.

Dev: And they model contact-rich interaction as an MDP, but they focus their learning goal only on maximizing the expected discounted return J(pi theta) = E

P infinity t=zero gamma t r t: within that critical segment <ref:2607.15982#pg3,model contact-rich interaction as an MDP>.

Rosa: They’re not trying to solve the whole task with one massive model; they are exploiting the structure of contact-rich manipulation itself by learning only what is necessary for robustness.

Taro: So, if you look at how they set up the autonomous offline data collection, it seems like they first record a single demonstration using kinesthetic teaching before replaying it until that critical segment is hit.

Dev: Right, and then during that replay phase, they execute a mixed greedy-random policy pi explore to gather both successful and failed rollouts for the critical part of the task.

Rosa: And then they use all that collected data to obtain the learned policy pi theta through offline DRL training, which is much more sample efficient than trying to train it end-to-end from scratch.

Taro: The key seems to be that this sequential pipeline—data collection, learning the critical segment, and then chaining it with off-the-shelf planning—leads to consistently high success rates above ninety percent.

Dev: It changes how we think about these complex manipulation tasks; instead of a single monolithic policy, we can focus on targeted data collection for the hardest parts.

Conclusion: Rosa: So to wrap up this discussion on "Learning Where it Matters: Autonomous Data Collection and Offline Reinforcement Learning for Local Contact-Rich Manipulation," the main point is that focusing data collection only on the critical segment of contact tasks yields high success rates.

Dev: The paper by Hausdorfer et al. shows that this modular approach, combining autonomous data collection with offline deep reinforcement learning, is a viable way to achieve robust manipulation policies.

Taro: It suggests that for precision applications, we can move away from training massive end-to-end models and instead use targeted reliance on robot data and structured task decomposition.

Rosa: The implication for the world is that this method opens the door for targeted data collection in contact-rich tasks, leading to high success rates in precision applications across many different scenarios.

Dev: It means we can tackle these challenging physical interactions with policies that are more reliable and better at handling unexpected situations than those trained end-to-end.