Dynamic robotic cloth folding with efficient Koopman operator-based model predictive control

summary

Video file (mp4)

The gist

Robotic cloth folding is addressed by integrating physics-based simulation with efficient, kernel-based Koopman operator regression within a model predictive control framework to generate fast,

In short

The method integrates physics simulation with data-driven modeling and Model Predictive Control (MPC) to fold cloth quickly and accurately on a robot. By using Koopman operator regression, nonlinear cloth dynamics are recast into a linear form, allowing an MPC strategy to generate fast trajectories that successfully transfer from simulation to real-world robotic execution.

Key concepts

Physics-based Cloth Simulator
This simulator models how cloth moves by solving differential equations that account for real physical properties like density, bending stiffness, and aerodynamic effects. It generates high-fidelity data used to train the subsequent machine learning model.
Koopman Operator Regression
This technique learns a linear representation of complex, nonlinear system dynamics (like cloth folding). It transforms the nonlinear equations into a simpler linear space that can be efficiently learned using data from the physics simulation.

Terminology used across episodes

This episode discusses

The paper

Dynamic robotic cloth folding with efficient Koopman operator-based model predictive control · Read on arXiv

Istituto Italiano di Tecnologia Genoa, Italy · Institut de Robotica i Informatica Industrial, CSIC–UPC Barcelona, Spain

Robotic cloth folding is a challenging task, particularly when considering dynamic folding tasks, which aim at folding cloth by fast motions that leverage its dynamics. When subject to such fast motions, the complexity of cloth dynamics hinders both system identification and planning of folding trajectories, resulting in a difficult simulation-to-reality transfer when using physical models of cloth. Compared to the dexterity that humans exhibit when performing folding tasks, robotic approaches usually employ small garments with quite rigid dynamics, and are either too slow, or fast but imprecise, requiring several attempts to achieve a reasonably good fold. In this paper, we tackle these challenges by generating fast folding trajectories with a novel model predictive controller, integrating physics-based simulation of cloth dynamics and efficient, kernel-based Koopman operator regression. Koopman operator regression, an increasingly popular machine learning technique for nonlinear system identification, is used to obtain a linear model for the cloth being folded. Such a surrogate model, trained with data from a high-fidelity, physics-based cloth simulator, can then be employed within a suitable model predictive control algorithm, in place of the costly, nonlinear one, to efficiently generate folding trajectories to be executed by a robotic manipulator. Both in simulated and real-robot experiments, we show how the linearization supplied by the Koopman operator-based model can be employed to efficiently generate fast folding trajectories to unseen poses, without sacrificing folding accuracy.

DOI: 10.1109/ICRA57385.2026.11696823

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: Today's paper: "Dynamic robotic cloth folding with efficient Koopman operator-based model predictive control".

Dev: Robotic cloth folding is addressed by integrating physics-based simulation with efficient, kernel-based Koopman operator regression within a model predictive control framework to generate fast, accurate trajectories for real robotic execution.

Rosa: First, who's behind it and why it matters.

Title and authors: Rosa: So we're looking at the title "Dynamic robotic cloth folding with efficient Koopman operator-based model predictive control," which really tells us a lot about what this work is trying to accomplish in terms of making manipulation easier. It suggests they are tackling the difficulty of moving deformable objects quickly, which is something traditional physics models struggle with during fast motions.

Dev: The authors are Caldarelli, Coltraro, Colom, Rosasco, and Torras; they're clearly folks deep into the area where control theory meets data-driven modeling for these kinds of problems <ref:2605.18373#pg0>.

Taro: I see this as them aiming at the common issue where physics models get too slow or too complicated when you need fast dynamic maneuvers, which is a big problem in autonomy research <ref:2605.18373#pg0>.

Rosa: They are proposing a new way to use Koopman operator regression to handle these nonlinear dynamics specifically for cloth folding, which seems like they are trying to bridge the gap between complex physics and practical control methods <ref:2605.18373#pg0>.

Dev: The implication here is that if you can linearize the cloth dynamics efficiently this way, it opens up the door for using model predictive control strategies that are usually too slow to run in real-time during dynamic tasks <ref:2605.18373#pg0>.

Taro: This suggests we can shift away from systems that rely only on strict models toward autonomous agents that can learn and adjust their dynamics as they interact with the environment, which is crucial when things don't behave exactly as expected <ref:2605.18373#pg0>.

Rosa: So, essentially, they’re suggesting that using data to linearize complex physics makes dynamic manipulation tasks much more feasible for robots <ref:2605.18373#pg0>.

Dev: And the critical part is making sure this learned model runs fast enough to fit into a practical control loop without introducing too much latency or causing unexpected failures, which is where I focus my attention <ref:2605.18373#pg0>.

The paper's summary: Rosa: Now let's look at the summary of "Dynamic robotic cloth folding with efficient Koopman operator-based model predictive control" to understand exactly how they put all these pieces together to achieve their goal. This section lays out the step-by-step process they use for this whole system.

Dev: They begin by using a physics-based simulator, which we call the SOM, to generate synthetic folding trajectories; these trajectories then serve as the training data for the Koopman operator regression model <ref:2605.18373#pg1>.

Taro: That means they are using that simulator not just to see what happens visually, but actively feeding it the ground truth dynamics needed to train their learning algorithm <ref:2605.18373#pg1>.

Rosa: After training, this data-driven model is integrated into a Model Predictive Control strategy, which generates trajectories that respect performance goals like speed and accuracy while also following constraints from the actual robot setup <ref:2605.18373#pg1>.

Dev: They detail a complex state transformation involving a canonical feature map to lift the nonlinear cloth state into an infinite-dimensional space, which is then defined using block-diagonal operators derived from the kernel matrices learned during regression <ref:2605.18373#pg1>.

Taro: That step of reconstructing the state back into a finite representation is where they manage the complexity of working in that infinite space so the controller can actually function practically <ref:2605.18373#pg1>.

Rosa: Finally, they emphasize that they embed constraints directly into the optimal control problem to make sure the trajectory generation is robust enough for real-world execution <ref:2605.18373#pg1>.

Dev: These constraints cover several things, including making sure the data-driven approximation of the Koopman operator is followed, limits on control actions based on how close the corner gets to the table, and requirements for smooth changes in control inputs <ref:2605.18373#pg1>.

Taro: It's clear they are not just handing us a model; they are providing a complete framework that manages nonlinear cloth dynamics while enforcing physical limits directly through these constraints <ref:2605.18373#pg2>.

Rosa: That’s a very thorough summary of their method for moving from simulation data to actual control, and it shows how they manage the inherent difficulties in cloth physics <ref:2605.18373#pg1>.

Dev: It's impressive how they manage that transformation into an infinite-dimensional space effectively enough for a practical implementation inside an MPC loop <ref:2605.18373#pg1>.

The paper's improvements: Rosa: Now let’s talk about the suggested improvements in "Dynamic robotic cloth folding with efficient Koopman operator-based model predictive control" to see where the authors think their approach can be taken even further. They aren't just describing what they did; they are pointing out how this system can be enhanced.

Dev: They suggest combining the physics-based SOM, or SOM from four, with the efficient Koopman operator regression algorithm studied in twelve to get a data-driven, linear COM for a piece of cloth <ref:2605.18373#pg0>.

Taro: That combination is important because it bridges the gap between high-fidelity simulation and the efficient control model, which seems like a significant technical win for making this method applicable in more general scenarios <ref:2605.18373#pg0>.

Rosa: They also propose using that data-driven COM within an LMPC strategy to generate constrained robot trajectories in closed-loop with the SOM, folding the cloth to an unseen target pose <ref:2605.18373#pg1>.

Dev: That points toward a system where the SOM gives accurate feedback in simulation while the COM handles generating the actual trajectory for execution on the real robot <ref:2605.18373#pg1>.

Taro: The aim of this improvement is to achieve zero-shot manipulation by relying on this generalized physics model and learned Koopman operator rather than having to retrain everything for every new material or shape <ref:2605.18373#pg0>.

Rosa: They are aiming for a system that can handle novel scenarios effectively, which speaks to the potential impact when we encounter unexpected objects in the real world <ref:2605.18373#pg0>.

Dev: From an engineering standpoint, they're trying to reduce the sim-to-real gap substantially by using this approach when testing against poses that haven't been seen before <ref:2605.18373#pg1>.

Taro: This improvement addresses the main hurdle of robust execution, allowing the system to perform complex tasks on new cloth items with high precision <ref:2605.18373#pg0>.

Conclusion: Rosa: So, we’ve covered a lot about "Dynamic robotic cloth folding with efficient Koopman operator-based model predictive control," and to wrap up, what are the main implications we should be focusing on as we finish this discussion?

Dev: The main implication is that combining physics-based simulation with Koopman operator regression and MPC allows for generating fast, accurate cloth folding trajectories for real robots with successful zero-shot sim-to-real transfer <ref:2605.18373#pg0>.

Taro: I see this as a strong foundation for future autonomous systems that need to handle dynamic objects in unpredictable environments without needing extensive retraining for every new object <ref:2605.18373#pg0>.

Rosa: It really is about making the manipulation of deformable objects more accessible by using learned dynamics to overcome the traditional limitations of rigid physical modeling <ref:2605.18373#pg0>.

Dev: For the control engineer, it means we can design robust, constrained trajectories in real-time that respect both physics and operational limits during execution <ref:2605.18373#pg1>.

Taro: Ultimately, this work shows how a system can generalize its capabilities across different cloth types effectively through the use of the learned generalized physics model <ref:2605.18373#pg0>.

Rosa: That’s all we have for this paper today; it’s been fascinating to see how they tackle nonlinear dynamics with these techniques <ref:2605.18373#pg0>.

Dev: I'm looking forward to seeing how they handle the real-time constraints and latency in future implementations of this work <ref:2605.18373#pg1>.

Taro: I’m excited to see what kind of autonomous applications we can build with this level of dynamic manipulation capability <ref:2605.18373#pg0>.

More episodes

← Home