TouchTherm: Building Multimodal Digital Twins of Objects for Tactile and Thermal Rendering

arXiv:2610.01943 · cs.RO · Submitted 2026-10-01 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: Today's paper: "TouchTherm: Building Multimodal Digital Twins of Objects for Tactile and Thermal Rendering".

Dev: TouchTherm introduces a framework for constructing simulation-ready multimodal digital twins of real-world objects by integrating visual geometry, contact-aligned tactile microgeometry, and observation-driven dynamic thermal fields.

Rosa: First, who's behind it and why it matters.

Title and authors: Rosa: To recap, the core of this paper is introducing the TouchTherm framework which creates multimodal object assets by combining visual geometry, registered micro-height fields for touch sensing, and a dynamic thermal field reconstructed from infrared videos.

Dev: Specifically, it introduces a pipeline where you start with structured-light scanning and photometric stereo to get initial geometry and normal maps. Then, these are processed to derive contact-conditioned height fields that encode the high-frequency surface relief needed for tactile rendering on top of the coarse collision mesh.

Taro: And then, in parallel, they take multiview infrared videos and use a physics-regularized dynamic thermal reconstruction technique to build a time-varying thermal field that supports queries about surface temperature over time.

Rosa: That’s the summary; it moves beyond just static geometry by explicitly modeling the microscale surface structure for touch and incorporating transient temperature dynamics for interaction simulation.

Dev: It essentially decouples the coarse mesh, which handles collision detection, from the registered microgeometry, which is dedicated to preserving contact-scale relief without increasing dense collision geometry complexity.

Taro: This decoupling is important because it suggests we can get high-fidelity tactile cues without having to build an incredibly complex three dee model that would be computationally prohibitive for many simulations.

Rosa: And the thermal field reconstruction is handled by fusing observations from different cameras, correcting for temporal drift and viewing-angle bias to get observed temperatures at each point and time.

Dev: The methodology then models heat spreading along the surface using Gaussian weights, normalized by local weight sums, which approximates the negative surface Laplacian operator to simulate how temperature evolves based on neighbors.

Taro: That physics-regularized approach for modeling diffusion is what really gives the thermal field its physical grounding; it’s not just a black box prediction but something that follows heat transfer laws.

Rosa: And they use a multilayer perceptron to represent the final temperature field, Tbi(t), using spatial features derived from the eigenvectors of the diffusion operator L, which is then optimized jointly with parameters like surface diffusion coefficient alpha and ambient-relaxation coefficient h.

Dev: So, they are not just predicting a static image; they are identifying physical constants within that reconstruction process through data loss against observations and a physics loss term based on the thermal dynamics.

Taro: This joint optimization step is where the framework gets its strength; it tries to find a temperature field that looks right observationally while also adhering to physical heat transfer constraints.

Rosa: Ultimately, the summary is that they provide these integrated assets: coarse collision mesh, micro-height fields, and dynamic thermal fields for simulation readiness.

Dev: It’s about creating a holistic digital twin that addresses the limitations of existing datasets by providing those spatially varying tactile and thermal information necessary for high-fidelity haptic rendering.

The paper's summary: Rosa: Moving on to the specific improvements this paper suggests, they highlight the creation of these multimodal object twins as a significant step forward in representing real-world objects for robotic simulation.

Dev: The key improvement they detail is the development of a pipeline for constructing simulation-ready visuo-tactile-thermal object assets by integrating visual geometry, tactile micro-height fields, and dynamic thermal fields into one cohesive asset.

Taro: I think the most impactful part is the specific method for tactile rendering: developing a contact-conditioned pipeline to render optical tactile observations from those reconstructed micro-height fields.

Rosa: That pipeline starts by defining a local tangent frame at a contact point, sampling a metric surface patch, and then bilinearly sampling and transforming the normal map into that local frame to derive local height gradients.

Dev: Those height gradients are then recovered using Poisson reconstruction to obtain the optimal least-squares height field, which is what allows for high-frequency relief encoding without needing dense collision geometry.

Taro: That sounds like a method that effectively extracts the critical contact-scale information from the photometric stereo data and translates it directly into a usable height map for haptic rendering.

Rosa: The second major improvement lies in their dynamic thermal field rendering, where they reconstructed the field from multiview infrared cooling videos using that physics-regularized dynamic thermal reconstruction approach.

Dev: That reconstruction method involves fusing measurements from different cameras onto a common surface, correcting for temporal drift and viewing-angle bias to get the observed temperature at each point and time, Tobs i(t).

Taro: The way they model surface diffusion is by connecting neighboring points with Gaussian weights (wij) normalized by local weight sums to reduce density influence before applying the graph diffusion operator L.

Rosa: Then, they use a multilayer perceptron to represent the temperature field using spatial features derived from the eigenvectors of L, which are then jointly optimized against observations and a physics loss that penalizing residuals based on thermal dynamics.

Dev: The final improvement is that their validation showed that this approach significantly outperformed Coarse Geometry and Image-space Height baselines in terms of both tactile appearance fidelity and thermal prediction accuracy.

Taro: I think the biggest implication for future work is leveraging these assets to build more robust models, perhaps physics-informed neural networks or PINNs, by using the spatially varying, time-dependent dynamic thermal field as a crucial input constraint.

Rosa: So, they are showing how you can leverage this reconstruction not just for visualization but to feed into deeper AI systems that require that level of physical realism in their simulations.

The paper's improvements: Dev: So, wrapping up the discussion on TouchTherm: they successfully created a framework for building simulation-ready multimodal digital twins by integrating visual geometry, tactile micro-height fields, and dynamic thermal fields.

Rosa: They showed that combining these elements results in assets capable of supporting high-fidelity haptic rendering and temperature-aware interaction in robotic simulations.

Taro: The implications are substantial because it means AI agents can move beyond simple geometric recognition to perform more nuanced manipulation based on surface texture and temperature cues.

Dev: If this framework moves into the real world, we’re talking about better synthetic-to-real transfer, where an agent trained in simulation could reliably handle objects with subtle surface features that are critical for manipulation.

Rosa: And for HRI and VR training, the ability to provide spatially and temporally varying thermal feedback means human operators can train more effectively in realistic scenarios involving temperature-sensitive materials.

Taro: From my side, the potential lies in using that dynamic thermal field as a crucial input constraint within PINNs to simulate complex material behavior under dynamic thermal loads, which is something we need to explore.

Dev: I think the framework is sound technically, but my main practical question remains about how long this setup can reliably operate outside of a controlled lab environment before we hit significant failure modes due to sensor noise or environmental interference.

Rosa: That’s a fair point; the current setup uses microgeometry only for tactile rendering and doesn't model its effects on contact mechanics, and their dynamic thermal field focuses on natural cooling and doesn't capture bidirectional heat transfer during human or robotic contact.

Taro: So, while it’s not perfect yet, the next step is definitely testing how this framework performs when the world misbehaves and if those learned representations hold up under unexpected conditions.

Dev: Agreed; we need to see more stability in the loop rate and latency before we can really talk about deploying this kind of high-fidelity sensing into a working robot.

Rosa: So, that’s our summary of TouchTherm, showing how multimodal data construction can significantly enhance simulation fidelity for tactile and thermal interaction.

Conclusion: Rosa: So we've talked about how TouchTherm builds these multimodal digital twins by combining visual geometry, tactile micro-height fields, and dynamic thermal fields for simulation readiness.

Dev: Yeah, and that pipeline is quite clever for separating the coarse collision mesh from the high-frequency tactile relief without ballooning our computational load.

Taro: I'm still really thinking about what happens when the world isn't behaving nicely; how does this system handle unexpected contact or sudden changes in surface properties?

Rosa: That’s a valid concern, Taro, but for now, the paper shows it works well across twenty objects and four perspectives, validating both the tactile appearance fidelity and the thermal prediction accuracy against baselines.

Dev: The thermal prediction results are solid too; those held-out surface-temperature MAEs of zero point four six five degrees Celsius at thirty seconds really prove that the physics-regularized dynamic reconstruction is doing its job.

Taro: But I'm still pushing on the robustness; if the input infrared videos are noisy or have significant temporal drift, does the joint field reconstruction still hold up, or does it start drifting off?

Rosa: The authors did address that by optimizing parameters like surface diffusion and ambient relaxation against observation residuals, which suggests some resilience against noise, but they're admitting a limitation there.

Dev: Exactly; they flag that the thermal field reconstruction is currently focused on natural cooling and doesn't model bidirectional heat transfer during actual contact, which means it’s not ready for direct use in high-speed robotic grasping yet.

Taro: That lack of bidirectional modeling is a big gap; if we want true haptic realism, we need that two-way thermal feedback when a robot actually touches something.

Rosa: It's definitely where future work needs to focus, but for now, the proof on synthetic-to-real tactile object recognition increasing Top-one accuracy by fourteen percentage points is compelling evidence of its immediate utility.

Dev: That recognition boost is pretty impressive if it holds up when we push the loop rate higher; I’m still waiting on more data demonstrating how stable those local tangent frame calculations are under high-speed motion.

Taro: I hope they look into incorporating contact mechanics into the microgeometry modeling next, because that’s what gets us closer to truly autonomous interaction in dynamic environments.

Rosa: Well, that wraps up our discussion on TouchTherm: it provides a powerful way to inject physical realism into our simulations through these integrated assets.

Dev: It's a solid piece of work, though we need more rigorous testing on latency and failure modes before we can rely on it for mission-critical control loops.

Taro: I'm still keen to see how they expand the thermal modeling to handle those complex, dynamic heat exchange scenarios that are essential for real-world robotics.

Rosa: We’ll keep an eye on their next steps toward incorporating those contact mechanics you mentioned and see how it evolves from this initial framework.

ShanghaiTech University

cs.RO

Submitted: 2026-10-01

Updated: 2026-10-02

Comments: 8 pages, 8 figures. Project webpage: https://anonymous-research1.github.io/

Project page: https://anonymous-research1.github.io

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 82/100

The gist: TouchTherm introduces a framework for constructing simulation-ready multimodal digital twins of real-world objects by integrating visual geometry, contact-aligned tactile microgeometry, and

Key concepts

Coarse Collision Mesh
This is the basic 3D shape of an object used to define its overall physical boundaries. It acts as the foundation for visual rendering and collision detection in simulations, providing a simple structure that doesn't require extremely high-detail geometry.
Local Micro-height Fields
These fields encode high-frequency surface details, such as small bumps and grooves, specifically for tactile rendering. Instead of using a dense mesh for these fine details, this method uses local tangent frames to reconstruct the height variations at contact points efficiently.
Physics-Regularized Dynamic Thermal Field
This technique reconstructs how heat moves across an object over time using infrared videos. It models heat spreading via diffusion and uses a neural network to predict the temperature field, ensuring the simulation follows realistic physical laws like cooling and warming.

Terminology

Summary

TouchTherm introduces a framework for constructing simulation-ready multimodal digital twins of real-world objects by integrating visual geometry, contact-aligned tactile microgeometry, and observation-driven dynamic thermal fields. This approach addresses the limitations of existing 3D datasets by providing spatially registered representations of contact-scale surface structure and transient temperature dynamics necessary for high-fidelity haptic rendering and temperature-aware interaction in robotic simulation.

The gist

TouchTherm reconstructs spatially varying tactile fields and dynamic thermal fields at the object level, combining a coarse collision mesh with local tangent-space micro-height fields for tactile rendering, and reconstructing a physicsregularized dynamic thermal field from multiview infrared cooling videos for temperature-field simulation.

Pipeline Overview

The pipeline converts a real object into a simulator-ready multimodal asset comprising three core components:

  1. A coarse collision mesh, which serves as the foundation for visual rendering and physical collision detection.

  2. Registered local micro-height fields, which encode high-frequency surface relief exclusively for tactile rendering, preserving contact-scale relief without increasing dense collision geometry complexity.

  3. A dynamic thermal field, reconstructed from multiview infrared videos to support time-varying surface-temperature queries for robotic and virtual interactions.

Tactile Reconstruction

For tactile rendering, the framework decouples contact geometry from the tactile appearance on the scale of tactile sensor observations. The process involves:

  1. Normal Estimation and Registration: Using a high-fidelity handheld structured-light scanner for geometry and multiview normal maps obtained from photometric stereo (via a pretrained SDM-UniPS model), these are registered to the scanned geometry, and an object-to-camera pose is estimated using PnP.

  2. Contact-conditioned Tactile Rendering: At a contact point, a local tangent frame is defined, and a metric surface patch is sampled. The normal map is bilinearly sampled and transformed into the local tangent frame to derive local height gradients, which are then recovered using Poisson reconstruction to obtain the optimal leastsquares height field.

  3. Contact-conditioned Composition: The final tactile asset combines the texture with the coarse geometry via the equation: Hrender(u, v) = B(H0, M) + γM(u, v)hµ(u, v).

Dynamic Thermal Field Reconstruction

The thermal field is reconstructed from synchronized multiview infrared videos following a physicsregularized dynamic thermal reconstruction approach. This involves:

  1. Fusing thermal observations: Measurements from different cameras are mapped onto a common object surface, corrected for temporal drift and viewing-angle bias, and averaged to obtain the observed temperature at each point and time, Tobsi(t).

  2. Modeling surface diffusion: Heat spreading along the surface is modeled by connecting neighboring points with Gaussian weights (wij), which are normalized by local weight sums to reduce density influence. The graph diffusion operator L approximates the negative surface Laplacian, where the term −αLT cools or warms a point based on its neighbors' temperatures.

  3. Joint field reconstruction and parameter estimation: A multilayer perceptron (MLP) is used to represent the temperature field, Tbi(t) = Nθ(Φ(xi), t), where Φ(xi) are spatial features derived from the eigenvectors of L. The network parameters θ, surface diffusion coefficient α, and ambient-relaxation coefficient h are jointly optimized using data loss against observations and a physics loss that penalizes residuals based on the thermal dynamics: ri(t) = ∂Tbi/∂t + α(LTb)i + h(Tbi − T∞).

  4. Runtime rollout: Once parameters are identified, the heattransfer model advances the temperature field directly using explicit Euler integration: Tn+1 = Tn - ∆t[αLTn + h(Tn − T∞1)].

Validation and Application

The framework was validated on 20 objects across four perspectives. Tactile appearance fidelity was evaluated using G-SSIM for gradient structure and HF-NCC for high-frequency texture correlation, where TouchTherm outperformed Coarse Geometry and Image-space Height baselines. In thermal prediction, the full model (α + h) achieved heldout surface-temperature MAEs of 0.465 ◦C at 30 s and 0.592 ◦C at 45 s. Furthermore, synthetic-to-real tactile object recognition demonstrated that TouchTherm increased Top-1 accuracy by 14.0 percentage points compared to Coarse Geometry alone, indicating the registered tactile microgeometry provides object-dependent surface cues that improve recognition over coarse collision geometry. Finally, a wearable glove system demonstrated spatially and temporally varying thermal feedback in VR with high participant ratings for realism and comfort.

Limitations and Future Work

The current framework uses microgeometry only for tactile rendering and does not model its effects on contact mechanics. Its dynamic thermal field focuses on natural cooling and does not capture bidirectional heat transfer during human or robotic contact.

Improvements for AI systems

Here are specific improvements to AI systems based on the TouchTherm framework, detailing what those improved systems can achieve:

The TouchTherm framework enables several high-impact advancements in AI systems, primarily by injecting multimodal physical realism into simulation and perception models.

  1. Improved Robotic Tactile Perception and Manipulation (Synthetic-to-Real Transfer):

  2. Enhanced Object Recognition from Haptic/Tactile Data:

  3. More Realistic Human-Robot Interaction (HRI) and VR Training:

  4. Foundation for Physics-Informed Neural Networks (PINNs) in Complex Material Simulation:

  5. Improved Robotic Tactile Perception and Manipulation (Synthetic-to-Real Transfer):

The improved system can perform high-fidelity tactile sensing directly on simulated objects, drastically reducing the reality gap between simulation and the physical world.

Specifically:

  • An AI agent trained on TouchTherm assets can perform precise manipulation tasks (e.g., grasping, insertion, fine motor control) in a simulator where the tactile sensor feedback is derived from reconstructed micro-height fields (Sec. III-C).

  • The system can achieve superior synthetic-to-real object recognition accuracy (TouchTherm achieves 34.0% Top-1 accuracy vs. Coarse Geometry's 20.0%). This means an AI deployed in the real world can reliably identify and classify objects based on subtle surface textures (embossing, grooves) that are crucial for manipulation, even when training data is synthetic.

  1. Enhanced Object Recognition from Haptic/Tactile Data:

The system can move beyond simple geometric recognition to tactile-aware object classification.

Specifically:

  • An AI model trained on TouchTherm data can classify objects based on their reconstructed micro-height fields (e.g., distinguishing between a smooth cylinder and a textured, ribbed surface). This is critical for tasks requiring fine discrimination, such as quality control in manufacturing or automated sorting of complex parts where visual appearance alone is ambiguous.
  1. More Realistic Human-Robot Interaction (HRI) and VR Training:

The system can enable the creation of highly immersive and physically grounded training environments.

Specifically:

  • VR/AR systems can provide spatially and temporally varying thermal feedback (Sec. IV-D), allowing human operators to train for tasks involving temperature-sensitive materials or processes (e.g., handling hot/cold components). This improves the realism and efficacy of VR simulations used for remote teleoperation or complex assembly training, as participants receive intuitive, consistent thermal cues that match physical reality.
  1. Foundation for Physics-Informed Neural Networks (PINNs) in Complex Material Simulation:

The system can be leveraged to build more robust physics models by incorporating dynamic material properties derived from thermal reconstruction.

Specifically:

  • The framework provides a spatially varying, time-dependent dynamic thermal field (Sec. III-D). This field can be used as a crucial input or constraint within PINNs to simulate complex heat transfer phenomena, such as transient thermal stress during contact or heating/cooling cycles in robotic grippers. This allows AI models to predict material behavior under dynamic thermal loads with higher fidelity than models relying on static material labels.

Sources

Related papers