Dynamic robotic cloth folding with efficient Koopman operator-based model predictive control

arXiv:2605.18373 · cs.RO, cs.LG, math.DS, math.OC · Submitted 2026-05-18 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: Today's paper: "Dynamic robotic cloth folding with efficient Koopman operator-based model predictive control".

Dev: Robotic cloth folding is addressed by integrating physics-based simulation with efficient, kernel-based Koopman operator regression within a model predictive control framework to generate fast, accurate trajectories for real robotic execution.

Rosa: First, who's behind it and why it matters.

Title and authors: Rosa: So we're looking at the title "Dynamic robotic cloth folding with efficient Koopman operator-based model predictive control," which really tells us a lot about what this work is trying to accomplish in terms of making manipulation easier. It suggests they are tackling the difficulty of moving deformable objects quickly, which is something traditional physics models struggle with during fast motions.

Dev: The authors are Caldarelli, Coltraro, Colom, Rosasco, and Torras; they're clearly folks deep into the area where control theory meets data-driven modeling for these kinds of problems <ref:2605.18373#pg0>.

Taro: I see this as them aiming at the common issue where physics models get too slow or too complicated when you need fast dynamic maneuvers, which is a big problem in autonomy research <ref:2605.18373#pg0>.

Rosa: They are proposing a new way to use Koopman operator regression to handle these nonlinear dynamics specifically for cloth folding, which seems like they are trying to bridge the gap between complex physics and practical control methods <ref:2605.18373#pg0>.

Dev: The implication here is that if you can linearize the cloth dynamics efficiently this way, it opens up the door for using model predictive control strategies that are usually too slow to run in real-time during dynamic tasks <ref:2605.18373#pg0>.

Taro: This suggests we can shift away from systems that rely only on strict models toward autonomous agents that can learn and adjust their dynamics as they interact with the environment, which is crucial when things don't behave exactly as expected <ref:2605.18373#pg0>.

Rosa: So, essentially, they’re suggesting that using data to linearize complex physics makes dynamic manipulation tasks much more feasible for robots <ref:2605.18373#pg0>.

Dev: And the critical part is making sure this learned model runs fast enough to fit into a practical control loop without introducing too much latency or causing unexpected failures, which is where I focus my attention <ref:2605.18373#pg0>.

The paper's summary: Rosa: Now let's look at the summary of "Dynamic robotic cloth folding with efficient Koopman operator-based model predictive control" to understand exactly how they put all these pieces together to achieve their goal. This section lays out the step-by-step process they use for this whole system.

Dev: They begin by using a physics-based simulator, which we call the SOM, to generate synthetic folding trajectories; these trajectories then serve as the training data for the Koopman operator regression model <ref:2605.18373#pg1>.

Taro: That means they are using that simulator not just to see what happens visually, but actively feeding it the ground truth dynamics needed to train their learning algorithm <ref:2605.18373#pg1>.

Rosa: After training, this data-driven model is integrated into a Model Predictive Control strategy, which generates trajectories that respect performance goals like speed and accuracy while also following constraints from the actual robot setup <ref:2605.18373#pg1>.

Dev: They detail a complex state transformation involving a canonical feature map to lift the nonlinear cloth state into an infinite-dimensional space, which is then defined using block-diagonal operators derived from the kernel matrices learned during regression <ref:2605.18373#pg1>.

Taro: That step of reconstructing the state back into a finite representation is where they manage the complexity of working in that infinite space so the controller can actually function practically <ref:2605.18373#pg1>.

Rosa: Finally, they emphasize that they embed constraints directly into the optimal control problem to make sure the trajectory generation is robust enough for real-world execution <ref:2605.18373#pg1>.

Dev: These constraints cover several things, including making sure the data-driven approximation of the Koopman operator is followed, limits on control actions based on how close the corner gets to the table, and requirements for smooth changes in control inputs <ref:2605.18373#pg1>.

Taro: It's clear they are not just handing us a model; they are providing a complete framework that manages nonlinear cloth dynamics while enforcing physical limits directly through these constraints <ref:2605.18373#pg2>.

Rosa: That’s a very thorough summary of their method for moving from simulation data to actual control, and it shows how they manage the inherent difficulties in cloth physics <ref:2605.18373#pg1>.

Dev: It's impressive how they manage that transformation into an infinite-dimensional space effectively enough for a practical implementation inside an MPC loop <ref:2605.18373#pg1>.

The paper's improvements: Rosa: Now let’s talk about the suggested improvements in "Dynamic robotic cloth folding with efficient Koopman operator-based model predictive control" to see where the authors think their approach can be taken even further. They aren't just describing what they did; they are pointing out how this system can be enhanced.

Dev: They suggest combining the physics-based SOM, or SOM from four, with the efficient Koopman operator regression algorithm studied in twelve to get a data-driven, linear COM for a piece of cloth <ref:2605.18373#pg0>.

Taro: That combination is important because it bridges the gap between high-fidelity simulation and the efficient control model, which seems like a significant technical win for making this method applicable in more general scenarios <ref:2605.18373#pg0>.

Rosa: They also propose using that data-driven COM within an LMPC strategy to generate constrained robot trajectories in closed-loop with the SOM, folding the cloth to an unseen target pose <ref:2605.18373#pg1>.

Dev: That points toward a system where the SOM gives accurate feedback in simulation while the COM handles generating the actual trajectory for execution on the real robot <ref:2605.18373#pg1>.

Taro: The aim of this improvement is to achieve zero-shot manipulation by relying on this generalized physics model and learned Koopman operator rather than having to retrain everything for every new material or shape <ref:2605.18373#pg0>.

Rosa: They are aiming for a system that can handle novel scenarios effectively, which speaks to the potential impact when we encounter unexpected objects in the real world <ref:2605.18373#pg0>.

Dev: From an engineering standpoint, they're trying to reduce the sim-to-real gap substantially by using this approach when testing against poses that haven't been seen before <ref:2605.18373#pg1>.

Taro: This improvement addresses the main hurdle of robust execution, allowing the system to perform complex tasks on new cloth items with high precision <ref:2605.18373#pg0>.

Conclusion: Rosa: So, we’ve covered a lot about "Dynamic robotic cloth folding with efficient Koopman operator-based model predictive control," and to wrap up, what are the main implications we should be focusing on as we finish this discussion?

Dev: The main implication is that combining physics-based simulation with Koopman operator regression and MPC allows for generating fast, accurate cloth folding trajectories for real robots with successful zero-shot sim-to-real transfer <ref:2605.18373#pg0>.

Taro: I see this as a strong foundation for future autonomous systems that need to handle dynamic objects in unpredictable environments without needing extensive retraining for every new object <ref:2605.18373#pg0>.

Rosa: It really is about making the manipulation of deformable objects more accessible by using learned dynamics to overcome the traditional limitations of rigid physical modeling <ref:2605.18373#pg0>.

Dev: For the control engineer, it means we can design robust, constrained trajectories in real-time that respect both physics and operational limits during execution <ref:2605.18373#pg1>.

Taro: Ultimately, this work shows how a system can generalize its capabilities across different cloth types effectively through the use of the learned generalized physics model <ref:2605.18373#pg0>.

Rosa: That’s all we have for this paper today; it’s been fascinating to see how they tackle nonlinear dynamics with these techniques <ref:2605.18373#pg0>.

Dev: I'm looking forward to seeing how they handle the real-time constraints and latency in future implementations of this work <ref:2605.18373#pg1>.

Taro: I’m excited to see what kind of autonomous applications we can build with this level of dynamic manipulation capability <ref:2605.18373#pg0>.

Istituto Italiano di Tecnologia Genoa, Italy · Institut de Robotica i Informatica Industrial, CSIC–UPC Barcelona, Spain

cs.RO, cs.LG, math.DS, math.OC

Submitted: 2026-05-18

Updated: 2026-05-18

Comments: Accepted for presentation at the 2026 IEEE International Conference on Robotics and Automation (ICRA)

DOI: 10.1109/ICRA57385.2026.11696823

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 88/100

The gist: Robotic cloth folding is addressed by integrating physics-based simulation with efficient, kernel-based Koopman operator regression within a model predictive control framework to generate fast,

Key concepts

Physics-based Cloth Simulator: This simulator models how cloth moves by solving differential equations that account for real physical properties like density, bending stiffness, and aerodynamic effects. It generates high-fidelity data used to train the subsequent machine learning model.
Koopman Operator Regression: This technique learns a linear representation of complex, nonlinear system dynamics (like cloth folding). It transforms the nonlinear equations into a simpler linear space that can be efficiently learned using data from the physics simulation.

Terminology

Model Predictive Control — 17× in this paper
System Identification — 8× in this paper · explained in 3 episodes
Accuracy — 6× in this paper · explained in 1 episode
Fidelity — 6× in this paper · explained in 2 episodes
Sim-to-Real Gap — 6× in this paper · explained in 6 episodes
Sim-to-Real Transfer — 5× in this paper · explained in 2 episodes
Model Predictive Control (MPC) — 2× in this paper · explained in 5 episodes
Ordinary Differential Equations (ODEs) — 2× in this paper · explained in 2 episodes
Benchmarking — 1× in this paper · explained in 3 episodes
Core — 1× in this paper · explained in 2 episodes
Elias — 1× in this paper · explained in 2 episodes
Generalization — 1× in this paper · explained in 13 episodes
Hilbert Space — 1× in this paper · explained in 2 episodes
Motion Primitives — 1× in this paper · explained in 2 episodes

Summary

Robotic cloth folding is addressed by integrating physics-based simulation with efficient, kernel-based Koopman operator regression within a model predictive control framework to generate fast, accurate trajectories for real robotic execution.

The core methodology combines a physics-based simulator with data-driven modeling and control.

The approach begins by generating high-fidelity folding data using a physics-based cloth simulator [4] which models the cloth dynamics via ordinary differential equations (ODEs) that account for density, bending stiffness, Rayleigh damping, and aerodynamic effects. This simulation is used to train a non-parametric machine learning approximation obtained through Koopman operator regression [13], [14], [15]. The goal of this step is to recast the nonlinear dynamics of cloth to linear ones (albeit in an infinite dimensional vector space), that can be learned efficiently for use as a surrogate model.

The surrogate model is integrated into a Model Predictive Control (MPC) strategy.

The Koopman operator regression, approximated via the Nystrom method, yields a data-driven linear model for the cloth dynamics. This linear approximation is then embedded within a linear MPC pipeline to generate trajectories that account for performance criteria and user-defined constraints (as opposed to the training data). The MPC solves an optimal control problem (OCP) over a finite horizon, using the surrogate model's dynamics to forecast future states and minimize a cost function that balances tracking the target pose with adherence to control constraints.

The control architecture involves state transformation, reconstruction, and constraint enforcement.

To utilize the linear system in MPC, the nonlinear cloth state is lifted into an infinite-dimensional space via a canonical feature map. A data-driven system for t ≥ 0 is then defined using block-diagonal operators derived from the kernel matrices learned during regression. This surrogate state is then reconstructed back into a finite-dimensional representation of the cloth's pose using a reconstruction matrix, allowing the MPC to operate effectively in this transformed space.

Constraints are embedded directly into the OCP for robust trajectory generation.

The Model Predictive Controller (MPC) incorporates several types of constraints crucial for successful real-world execution. These include:

Constraints related to the lifting of the state and dynamics, which enforce adherence to the data-driven approximation of the Koopman operator.
Constraints on control actions, such as preventing the controlled corner from getting too close to (or below) the table, defined by bounds like uκ−1 + ∆u0 ≥[-∞, −ymin, hmin]T.
Constraints enforcing Smoothness of the generated control trajectories, formulated as bounds on successive control increments: -s ≤∆u0 − (uκ−1 − uκ−2) ≤ s and similar constraints for subsequent steps.
Constraints ensuring that the position of the two controlled points in the mesh vary by the same displacement, to emulate the end-effector of the cloth moving with constant orientation.

The proposed pipeline demonstrates successful sim-to-real transfer.

The entire pipeline is executed on a real robotic platform (e.g., UR5 arm) to test its effectiveness against unseen target poses. Experiments show that this methodology achieves a successful zero-shot sim-to-real transfer on fast folding actions (less than 1.5 s), effectively counteracting the inertial effects of cloth at high speeds and reducing the sim-to-real gap. Performance is evaluated using metrics like the relative folding error (Efold), which compares real robot results against target poses generated by the simulator, showing satisfactory performance for different garment materials like wool and denim.

Contributions include:

Combining a physics-based, collision-aware SOM from [4] with the efficient Koopman operator regression algorithm studied in [12], obtaining a data-driven, linear COM for a piece of cloth.
Using such a COM in an LMPC control strategy to generate constrained robot trajectories in closed-loop with the SOM, folding the cloth to an unseen target pose.
Executing and evaluating these generated trajectories on a real robotic platform, showcasing how the methodology reduces the sim-to-real gap.

The gist: The paper demonstrates that combining physics-based simulation with Koopman operator regression and Model Predictive Control allows for generating fast, accurate cloth folding trajectories for real robots with successful zero-shot sim-to-real transfer.

Improvements for AI systems

As a fastidious researcher, I have analyzed this paper and identified several high-impact areas where integrating its core components—physics-based modeling, Koopman operator regression, and Model Predictive Control (MPC)—can lead to significant improvements in robotic manipulation AI systems.

Here are the specific improvements and what the resulting AI system can achieve:

Incorporate a Physics-Informed Surrogate Model for High-Speed Trajectory Generation:
Enable Derivative-Free, Real-Time Trajectory Optimization for Deformable Objects:
Achieve Robust Sim-to-Real Transfer via Data-Driven Linearization of Nonlinear Dynamics:
Implement Constraint Handling in the Control Space for Physical Feasibility and Safety:

The resulting improved AI system will possess the following capabilities:

An AI agent capable of autonomously planning and executing high-speed, dynamic folding motions for cloth objects (e.g., clothing, banners) with human-level dexterity, significantly reducing the sim-to-real gap.
A control system that generates optimal trajectories in real-time (within the MPC loop) that simultaneously account for complex physical constraints like cloth inextensibility and collision avoidance (self-collision and table contact).
A model capable of learning a simplified, linear representation of highly nonlinear cloth dynamics from high-fidelity physics simulations, allowing the controller to solve computationally expensive optimal control problems efficiently using Linear Quadratic Regulators (LQR) techniques.
An AI that can perform zero-shot manipulation—successfully executing a complex folding task on unseen cloth materials or shapes without requiring extensive retraining on those specific instances, relying instead on the generalized physics model and learned Koopman operator.

Abstract

Robotic cloth folding is a challenging task, particularly when considering dynamic folding tasks, which aim at folding cloth by fast motions that leverage its dynamics. When subject to such fast motions, the complexity of cloth dynamics hinders both system identification and planning of folding trajectories, resulting in a difficult simulation-to-reality transfer when using physical models of cloth. Compared to the dexterity that humans exhibit when performing folding tasks, robotic approaches usually employ small garments with quite rigid dynamics, and are either too slow, or fast but imprecise, requiring several attempts to achieve a reasonably good fold. In this paper, we tackle these challenges by generating fast folding trajectories with a novel model predictive controller, integrating physics-based simulation of cloth dynamics and efficient, kernel-based Koopman operator regression. Koopman operator regression, an increasingly popular machine learning technique for nonlinear system identification, is used to obtain a linear model for the cloth being folded. Such a surrogate model, trained with data from a high-fidelity, physics-based cloth simulator, can then be employed within a suitable model predictive control algorithm, in place of the costly, nonlinear one, to efficiently generate folding trajectories to be executed by a robotic manipulator. Both in simulated and real-robot experiments, we show how the linearization supplied by the Koopman operator-based model can be employed to efficiently generate fast folding trajectories to unseen poses, without sacrificing folding accuracy.