PneuTac: Tactile Manipulation with Soft Pneumatic Robots via Unified MPM-Gaussian Splatting Simulation

arXiv:2609.38418 · cs.RO · Submitted 2026-09-29 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: Today's paper: "PneuTac: Tactile Manipulation with Soft Pneumatic Robots via Unified MPM-Gaussian Splatting Simulation".

Dev: Soft robots and tactile sensors have demonstrated great potential in delicate manipulation tasks,

Rosa: First, who's behind it and why it matters.

Paper summary: Rosa: So we're diving into PneuTac: Tactile Manipulation with Soft Pneumatic Robots via Unified MPM-Gaussian Splatting Simulation, which addresses a real headache in soft robotics—the lack of efficient simulation tools for tactile manipulation. What's the core idea here, Dev?

Dev: Well, Rosa, the paper proposes PneuTac as a unified framework that couples the Material Point Method or MPM for modeling dynamics with three dee Gaussian Splatting or three deeGS for rendering. The main claim is that this coupling allows them to do efficient real-to-sim modeling and then train policy networks using surrogate models derived from this simulation, which was previously very hard to achieve.

Taro: That sounds interesting because the abstract mentions that existing simulators often model soft robots and tactile sensors in isolation, leading to big calibration gaps that are tough to bridge. I'm curious how this unified approach actually tackles those gaps when dealing with compliant hardware.

Rosa: Exactly, Taro. The authors are aiming for a single framework that handles both the soft robot's dynamics and the deformable gel of the tactile sensor together using MPM. It seems like they believe this combined modeling capability is what unlocks more reliable policy learning for real-world contact-rich manipulation tasks.

Dev: From an engineering standpoint, the paper highlights a few key components that make it work, like using a simple vision-based procedure for calibrating both the robot and the sensor devices. They also employ action and perception networks to create these fast surrogate models for both robot actuation and tactile rendering.

Taro: The idea of using perception networks to map probe deformations to surface layer deformations seems crucial for making the tactile simulation fast enough, which is a big hurdle in running complex simulations. What happens when things go wrong during that mapping process?

Rosa: That's where Taro's point hits home. The paper discusses using a tactile-guided pipeline to collect demonstrations, which uses real-world observations as warm starts for a model-predictive controller. This suggests they are trying to use the simulation not just for training from scratch, but for augmenting demonstrations collected in the real world.

Dev: And that augmentation step is where I worry about loop rates and latency. The goal here is to encourage agreement with demonstrated contact-area trajectories during simulation, which hopefully supports policy training more effectively than just using real data alone. We need to make sure the simulation fidelity doesn't introduce unacceptable delays in the feedback loop.

Taro: If we look at the future potential, this unified modeling capability could mean that researchers don't have to spend as much time painstakingly calibrating individual models for every new soft robot or sensor setup. Imagine deploying tactile manipulation policies across a wider variety of compliant hardware with less initial setup effort.

Paper summary: Rosa: That's a big picture implication, Taro. If the simulation can handle the complexity of both the robot and the sensor at once, it opens up possibilities for more general applications in delicate interaction tasks that are currently too data-intensive to solve reliably.

Dev: I'm still thinking about deployment outside of a pristine lab setting. How robust is this MPM-three deeGS simulator when the soft robot or the sensor encounters unexpected external disturbances or material variations? The paper focuses heavily on calibration, but real-world robustness is always a concern.

Taro: The authors do mention that they are evaluating this framework on three contact-rich manipulation tasks: switch flipping, egg carton opening, and card pulling. Those tasks are fairly complex interactions, so I wonder how well the learned policies generalize to situations where the environment isn't perfectly controlled.

Rosa: That leads us nicely into the broader impact of PneuTac. The authors conclude that policies trained with this simulation-augmented demonstration pipeline outperform baselines trained only on real data for these contact-rich manipulation tasks. This suggests that we can use simulation to effectively expand our training data collection strategy, which is a significant step forward in practical robotics.

Dev: So, looking at the title and authors of this PneuTac paper, it seems the focus is on creating a practical framework for tactile manipulation on compliant hardware. It’s not just about a fancy simulation; it’s about making that simulation useful for actual robot control loops.

Taro: I think the real implication here is in the autonomy space, Dev. If we can reliably simulate complex tactile interactions using this method, it means we can test and refine autonomous agents in virtual environments before deploying them to physical systems where collecting enough real interaction data is time-consuming.

Rosa: I agree with Taro that the ability to augment demonstrations through simulation is valuable for improving policy success rates when we are limited by real-world data collection. It moves us closer to systems that can handle delicate physical interactions more reliably across different hardware setups.

Dev: From a control perspective, the speedup they report—achieving a "five times speedup in comparison" for simulating the soft finger compared to running the full MPM simulation —is something I'll be watching closely when we look at real-time performance requirements. That level of acceleration is critical for latency management.

Taro: If the authors can solve the calibration issue efficiently, that would be a huge step toward making these systems deployable outside controlled laboratory settings where every parameter needs to be painstakingly tuned from scratch.

Rosa: It sounds like the main thrust of PneuTac is providing a practical tool that bridges the gap between high-fidelity physical modeling and efficient policy learning for complex tactile tasks. We'll see how this impacts real-world deployment soon.

Conclusion: Rosa: So, we've been deep in the technical weeds of PneuTac, and now we’re wrapping up to talk about what this whole thing actually means for soft robotics and tactile feedback manipulation.

Dev: I think the core message is that they managed to unify the modeling of both the physical robot and its sensor using MPM and three deeGS, which really streamlines how we can test control systems.

Taro: From an autonomy standpoint, I see this as a massive step toward building more robust agents because they’ve addressed the simulation-to-real gap in a very concrete way.

Rosa: Exactly, Taro, it seems they're tackling the difficulty of creating reliable simulation data for tasks that involve delicate physical contact on compliant materials.

Dev: And I'm really focused on how this unified approach helps us manage those demanding loop rates; the speedup they achieved in simulating the soft finger is something that could actually make real-time control more feasible.

Taro: That speedup is important, but what about when things go wrong in a real-world scenario, like unexpected material shifts or sensor noise? How does this framework handle those kinds of misbehaves?

Rosa: Well, the paper shows they built in mechanisms for calibration that should help them adapt to variations in physical parameters without needing a completely new simulation setup every time.

Dev: If the calibration procedure is simple enough, it lowers the barrier for deploying these models outside of a perfectly controlled lab environment; we need to know how long this fidelity holds up before we consider field deployment.

Taro: The implication here is that we might see robots performing complex manipulation tasks in more unstructured settings because the policy learned in this high-fidelity simulation will have a better understanding of real-world contact dynamics.

Rosa: That’s what I'm excited about—moving beyond just successful lab demonstrations to actual practical application where the environment isn't perfectly pristine.

Dev: So, if we look at the authors, they’ve clearly put a lot of work into making this framework practical for tactile manipulation on compliant hardware.

Taro: And that practicality is what matters most; it shows that complex interaction modeling isn't just theoretical anymore, it’s becoming a tool we can actually use to improve autonomous behavior.

Shaohong Zhong, Marco Pontin, Joe Watson, Perla Maiolino, Ingmar Posner

Oxford Robotics Institute

cs.RO

Submitted: 2026-09-29

Updated: 2026-09-29

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 77/100

The gist: Soft robots and tactile sensors have demonstrated great potential in delicate manipulation tasks, but learning tactile manipulation with compliant robots has been challenging due to the lack of

Key concepts

Material Point Method (MPM): MPM is a hybrid simulation technique that models deformable bodies by tracking material particles within a background grid. This method is excellent for accurately simulating complex dynamics, such as the large deformations of soft pneumatic robots and the behavior of tactile sensors under contact, ensuring robust physical modeling.
3D Gaussian Splatting (3DGS): 3DGS is a rendering technique that represents scenes using anisotropic 3D Gaussians. In this framework, particles from the MPM model define the positions of these Gaussians. This coupling allows for fast and efficient rendering of complex dynamics, enabling real-to-sim modeling of both the robot's shape and tactile sensor interactions.
Real-to-Sim Modelling: This process involves using simple vision methods to calibrate a simulator against the real world. For example, material properties of the soft robot are identified by actuating it with fixed pressures and observing deformation via a camera. This ensures the simulation accurately reflects physical reality.
Surrogate Models: These are simplified models trained to replace computationally expensive full simulations. PneuTac uses action and perception networks to map complex MPM models into fast surrogate models for both robot actuation and tactile rendering, significantly speeding up the simulation process.

Terminology

Calibration — 26× in this paper · explained in 7 episodes
Gaussian Splatting — 12× in this paper · explained in 4 episodes
Material Point Method — 9× in this paper
Adam — 8× in this paper · explained in 2 episodes
3D Gaussian Splatting — 7× in this paper · explained in 2 episodes
Unified Framework — 7× in this paper · explained in 5 episodes
Fidelity — 6× in this paper · explained in 2 episodes
3D Gaussian Splatting (3DGS) — 4× in this paper · explained in 4 episodes
Fine-Tuning — 4× in this paper · explained in 5 episodes
Actor — 3× in this paper · explained in 3 episodes
Core — 3× in this paper · explained in 2 episodes
Action Space — 1× in this paper · explained in 1 episode
Digital Twin — 1× in this paper · explained in 4 episodes
Robustness — 1× in this paper · explained in 5 episodes
Sim-to-Real Gap — 1× in this paper · explained in 6 episodes
Sim-to-Real Transfer — 1× in this paper · explained in 2 episodes
System Identification — 1× in this paper · explained in 3 episodes

Summary

Soft robots and tactile sensors have demonstrated great potential in delicate manipulation tasks, but learning tactile manipulation with compliant robots has been challenging due to the lack of efficient simulation tools. This paper presents PneuTac, a unified framework for tactile feedback manipulation with soft pneumatic robots that leverages Material Point Method (MPM) and 3D Gaussian Splatting (3DGS) for accurate modeling and simulation, enabling efficient real-to-sim modeling and policy learning.

The gist

PneuTac is a unified framework for tactile feedback manipulation with soft pneumatic robots that leverages the material point method (MPM) for modelling dynamics and 3D Gaussian splatting (3DGS) for rendering to enable efficient real-to-sim modelling, surrogate models, and tactile-guided demonstration augmentation.

How it works

The PneuTac framework consists of four main components: an MPM-3DGS simulator, a simple vision-based procedure for calibration of both devices, action and perception networks for efficient surrogate modelling, and a tactile-guided pipeline for using the calibrated simulator to collect demonstrations from a small set of real-world demonstrations. The core simulation engine represents both the soft robot and the deformable gel of the vision-based tactile sensor using MPM. This hybrid particle–grid scheme tracks a deformable body as material particles while resolving its dynamics on a background grid, which handles large deformation and contact robustly.

For learning the appearance model, 3D Gaussian Splatting is coupled with the MPM model. 3DGS represents a scene as anisotropic 3D Gaussians, and sampled particles from the MPM model define their positions. The framework enables fast simulation of the complex dynamics while allowing efficient rendering and real-to-sim modelling of the tactile sensor and the soft robot with simple vision-based methods.

Real-to-Sim Modelling

The framework facilitates real-to-sim modelling through a simple vision-based procedure. For calibrating the soft robot model, material parameters such as Young’s modulus Es and Poisson’s ratio νs are identified by actuating the robot with fixed pressures and capturing deformation with a camera. This parameter identification is performed using a sampling-based method with an image-based loss, optimizing parameters to minimize the difference between simulated renderings and real-world observations.

For calibrating the tactile sensor reading, once the base 3DGS model is trained on a rest background image (Irest), its parameters are fixed. A single indentation image (Iindent) is then used to calibrate a photometric model for contact. This involves fitting a slope- and depth-dependent contact photometric model where coefficients like color increment are estimated by minimizing the loss between the rendered image and Iindent, specifically optimizing coefficients such as lx, ly,cd.

Surrogate Models with Action and Perception Networks

To increase simulation speed, PneuTac trains two separate networks to map MPM models to fast surrogate models. For soft robot actuation, a simplified skeleton model is used with a learned residual torque τr added to the base actuation torque τmpm to obtain the total applied torque τs = τmpm + τr. This allows for efficient simulation of the soft finger by achieving a 5x speedup in comparison compared to running the full MPM simulation.

For tactile rendering, a perception network maps probe deformations to surface layer deformations, learning a mapping fm: ∆probe → ∆mpm. This network is trained by minimizing MSE over the contact region with the Adam optimizer. This allows tactile images to be rendered from predicted indentations during simulation.

Tactile-guided Demonstration Collection and Manipulation

The framework utilizes a tactile-guided pipeline for demonstration augmentation. Real-world demonstrations are used as warm starts for a model-predictive controller with the cross-entropy method (CEM) in simulation, guided by real tactile observations. This allows for the collection of additional demonstrations in simulation with the same observation space and action space as the real world, encouraging agreement with the demonstrated contact-area trajectory to support policy training.

The framework is evaluated on three contact-rich manipulation tasks: switch flipping, egg carton opening, and card pulling. Policies trained using this simulation-augmented demonstration pipeline outperform baselines trained on the same amount of real data on these tasks. The results show that leveraging additional demonstrations collected from simulation substantially increases policy success rates in the real world compared to using only real-world demonstrations for training. PneuTac is presented as a practical framework for tactile manipulation on compliant hardware.

Conclusion

PneuTac proposes a unified MPM-3DGS simulator that jointly simulates soft pneumatic actuators and vision-based tactile sensors, enabling data-efficient real-to-sim modelling and sim-to-real policy learning. The framework successfully demonstrates the utility of simulation in augmenting demonstrations for tactile manipulation on compliant hardware. It achieves high fidelity in modeling both robot deformation and tactile sensor readings, leading to superior performance in contact-rich manipulation tasks when trained with simulation data.

Improvements for AI systems

Here are the specific improvements that can be made to existing AI systems, based on the PneuTac framework, and what these improved systems could achieve:

The PneuTac framework provides a unified, physics-informed simulation environment that bridges the gap between complex soft robot dynamics and high-resolution tactile sensing. Applying this methodology to current AI systems yields the following specific improvements:

Improve the training efficiency of Reinforcement Learning (RL) policies for compliant manipulation tasks by using a unified, high-fidelity simulator that accurately models both soft robot compliance (via MPM) and vision-based tactile feedback (via 3DGS).
Enable Sim-to-Real transfer for tactile manipulation policies with significantly reduced real-world data requirements by utilizing the framework's data-efficient real-to-sim modeling pipeline.
Enhance the perception capabilities of robotic systems by training specialized deep neural networks (like UNets) to accurately map raw tactile sensor probe deformations onto the internal state representation of a soft body model (MPM).
Improve the robustness and generalization of learned policies by augmenting simulation demonstrations with tactile-guided trajectory collection, ensuring policies learn contact behaviors directly from physical interaction signals rather than just end-effector trajectories.

These improved AI systems can achieve the following specific capabilities:

A robotic system could perform delicate tasks such as switching a toggle switch or opening an egg carton lid with high success rates (e.g., achieving 80%+ success on the Egg task in simulation, as shown in Table III), by directly controlling pneumatic pressures based on real-time tactile feedback during contact.
The system could operate effectively in environments where physical training is costly or dangerous, achieving near state-of-the-art performance by leveraging simulation demonstrations augmented with tactile guidance (Sim+Real policy), which substantially outperforms policies trained solely on sparse real data.
The AI can perform precise force/contact estimation during manipulation; specifically, the perception network could accurately map the observed deformation of a small probe sensor onto the internal deformation of a soft finger, allowing for highly accurate depth and contact area readings (achieving a mean absolute error of 0.077 mm for indentation depth).
The AI can learn complex, non-linear actuation strategies for compliant robots; specifically, it can predict the required residual joint torques needed to achieve desired bending shapes under varying internal pressures, resulting in a 5x speedup in simulation compared to full MPM models while maintaining high fidelity.