VT-MUSE: Multimodal Unified Sequential Visuotactile Representation Learning for Manipulation

arXiv:2608.21290 · cs.RO, cs.CV · Submitted 2026-08-21 · Read on arXiv

cs.RO, cs.CV

Submitted: 2026-08-21

Updated: 2026-10-06

Terminology

• Transformer — 18× in this paper · explained in 2 episodes
• Representation Learning — 13× in this paper · explained in 2 episodes
• Alignment — 11× in this paper · explained in 1 episode
• Benchmark — 7× in this paper · explained in 2 episodes
• Consistency — 7× in this paper · explained in 3 episodes
• Ablation — 6× in this paper · explained in 1 episode
• Latent Representation — 6× in this paper · explained in 2 episodes
• Benchmarking — 3× in this paper · explained in 3 episodes
• Contrastive Learning — 3× in this paper · explained in 5 episodes
• Generalization — 3× in this paper · explained in 13 episodes
• Insert Tube — 3× in this paper
• Press Toaster — 3× in this paper
• Wipe Board — 3× in this paper
• World Model — 3× in this paper · explained in 7 episodes
• Lift Bottle — 2× in this paper
• Action Chunking — 1× in this paper · explained in 1 episode
• Action-Conditioned World Model — 1× in this paper · explained in 1 episode
• Aggregation — 1× in this paper · explained in 1 episode
• Embeddings — 1× in this paper · explained in 2 episodes
• Feature Extraction — 1× in this paper · explained in 2 episodes
• Fidelity — 1× in this paper · explained in 2 episodes
• Pull-out Drawer — 1× in this paper
• Pull-out Key — 1× in this paper
• Self-Supervised Learning — 1× in this paper · explained in 7 episodes
• Trace — 1× in this paper · explained in 2 episodes
• Transferability — 1× in this paper · explained in 4 episodes
• Unified Framework — 1× in this paper · explained in 5 episodes
• VLA Models — 1× in this paper · explained in 3 episodes
• Vision Transformer (ViT) — 1× in this paper · explained in 2 episodes

Sources

• FAWAM: Force-Aware World Action Models for Closed-Loop Contact-Rich Manipulation
• Sparsh: Self-supervised touch representations for vision-based tactile sensing
• Transferable Tactile Transformers for Representation Learning Across Diverse Sensors and Tasks
• Visuo-Tactile Transformers for Manipulation
• ViTaMIn: Learning Contact-Rich Tasks Through Robot-Free Visuo-Tactile Manipulation Interface
• ViTacFormer: Learning Cross-Modal Representation for Visuo-Tactile Dexterous Manipulation
• OmniVTA: Visuo-Tactile World Modeling for Contact-Rich Robotic Manipulation
• ManiSkill-ViTac 2025: Challenge on Manipulation Skill Learning With Vision and Tactile Sensing
• ManiFeel: Benchmarking and Understanding Visuotactile Manipulation Policy Learning
• UniVTAC: A Unified Simulation Platform for Visuo-Tactile Manipulation Data Generation, Learning, and Benchmarking
• TacO: Benchmarking Tactile Sensors for Object Manipulation
• Tabero: Learning Gentle Manipulation with Closed-Loop Force Feedback from Vision, Touch, and Language
• TactiDex: A Real-World Tactile-Guided Benchmark for Human-Like Dexterous Manipulation
• $N_0$-VTLA: Scaling Vision-Tactile-Language-Action Model with Latent Tactile Tokens
• UniTac: A Unified Multimodal Model for Cross-Sensor Tactile Understanding and Generation
• FreeTacMan: Robot-free Visuo-Tactile Data Collection System for Contact-rich Manipulation
• Touch begins where vision ends: Generalizable policies for contact-rich manipulation
• Dream-Tac: A Unified Tactile World Action Model for Contact-Rich Robot Manipulation
• Tactile-WAM: Touch-Aware World Action Model with Tactile Asymmetric Attention
• TacForeSight: Force-Guided Tactile World Model for Contact-Rich Manipulation

Related papers

• FMT x: An Efficient and Asymptotically Optimal Extension of the Fast Marching Tree for Dynamic Replanning
• MPCFormer: A physics-informed data-driven approach for explainable socially-aware autonomous driving
• RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies
• HRDexDB: A 4D Dexterous Grasping Dataset Across Human and Multiple Robot Embodiments
• APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies
• Fine-tuning is Not Enough: A Parallel Framework for Collaborative Imitation and Reinforcement Learning in End-to-end Autonomous Driving