Towards the Vision-Sound-Language-Action Paradigm: The HEAR Framework for Sound-Centric Manipulation

arXiv:2603.16086 · cs.RO, cs.AI, cs.CV, cs.SD · Submitted 2026-03-17 · Read on arXiv

cs.RO, cs.AI, cs.CV, cs.SD

Submitted: 2026-03-17

Updated: 2026-09-18

Comments: Accepted by The International Journal of Robotics Research (IJRR 2026). Project page: https://hear.irmv.top

DOI: 10.1177/02783649261477778

Project page: https://hear.irmv.top

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Transformer — 24× in this paper · explained in 2 episodes
• Benchmark — 21× in this paper · explained in 2 episodes
• Grounding — 16× in this paper · explained in 3 episodes
• Blind Execution Interval — 14× in this paper
• Foundation Model — 14× in this paper · explained in 3 episodes
• Foundation Models — 12× in this paper · explained in 7 episodes
• Action Chunking — 9× in this paper · explained in 1 episode
• World Model — 8× in this paper · explained in 7 episodes
• Core — 7× in this paper · explained in 2 episodes
• Ablation — 6× in this paper · explained in 1 episode
• Fine-Tuning — 6× in this paper · explained in 5 episodes
• VLA Models — 6× in this paper · explained in 3 episodes
• Representation Learning — 5× in this paper · explained in 2 episodes
• Failure Modes — 4× in this paper · explained in 3 episodes
• Robustness — 4× in this paper · explained in 5 episodes
• Streaming Stateful Transformer — 4× in this paper
• Action Space — 3× in this paper · explained in 1 episode
• Flow Matching — 3× in this paper · explained in 7 episodes
• Generalization — 3× in this paper · explained in 13 episodes
• Verification — 3× in this paper · explained in 2 episodes
• Vision-Language-Action (VLA) Models — 3× in this paper · explained in 2 episodes
• Accuracy — 2× in this paper · explained in 1 episode
• Affordance — 2× in this paper · explained in 1 episode
• Alignment — 2× in this paper · explained in 1 episode
• Decoupling — 2× in this paper · explained in 2 episodes
• Generative Model — 2× in this paper · explained in 2 episodes
• Latent Representation — 2× in this paper · explained in 2 episodes
• Multimodal Inputs — 2× in this paper · explained in 2 episodes
• Reinforcement Learning — 2× in this paper · explained in 4 episodes
• 3D Gaussian Splatting — 1× in this paper · explained in 2 episodes
• AdamW Optimizer — 1× in this paper · explained in 1 episode
• Automatic Speech Recognition (ASR) — 1× in this paper · explained in 2 episodes
• Current Vision-Language-Action — 1× in this paper
• Diffusion Model — 1× in this paper · explained in 8 episodes
• Diffusion Models — 1× in this paper · explained in 6 episodes
• Domain Shift — 1× in this paper · explained in 4 episodes
• Embeddings — 1× in this paper · explained in 2 episodes
• Gaussian Splatting — 1× in this paper · explained in 4 episodes
• Generative Modeling — 1× in this paper · explained in 3 episodes
• Generative Models — 1× in this paper · explained in 2 episodes
• KV cache — 1× in this paper · explained in 3 episodes
• Latent Diffusion Model — 1× in this paper · explained in 2 episodes
• Latent Diffusion Models — 1× in this paper · explained in 2 episodes
• Motion Primitives — 1× in this paper · explained in 2 episodes
• Multi-modal Inputs — 1× in this paper · explained in 2 episodes
• Multimodal Learning — 1× in this paper · explained in 2 episodes
• Multimodal Reasoning — 1× in this paper · explained in 5 episodes
• Rectified Flow — 1× in this paper · explained in 3 episodes
• Self-Supervised Learning — 1× in this paper · explained in 7 episodes
• Two-stage training — 1× in this paper · explained in 2 episodes

Sources

• RoboTwin 2.0: A Scalable Data Generator and Benchmark with Strong Domain Randomization for Robust Bimanual Robotic Manipulation
• Moshi: a speech-text foundation model for real-time dialogue
• Compact 3D Gaussian Splatting For Dense Visual SLAM
• What Is The Best 3D Scene Representation for Robotics? From Geometric to Foundation Models
• $\pi_{0.5}$: a Vision-Language-Action Model with Open-World Generalization
• Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperation
• Cosmos Policy: Fine-Tuning Video Models for Visuomotor Control and Planning
• OmniVLA: Physically-Grounded Multimodal VLA with Unified Multi-Sensor Perception for Robotic Manipulation
• Memory-based control with recurrent neural networks
• Tactile-VLA: Unlocking Vision-Language-Action Model's Physical Knowledge for Tactile Generalization
• MLA: A Multisensory Language-Action Model for Multimodal Understanding and Forecasting in Robotic Manipulation
• VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learning
• The Sound of Simulation: Learning Multimodal Sim-to-Real Robot Policies with Generative Audio
• RoboOmni: Proactive Robot Manipulation in Omni-modal Context
• Audio-VLA: Adding Contact Audio Perception to Vision-Language-Action Model for Robotic Manipulation
• ERMV: Editing 4D Robotic Multi-view images to enhance embodied agents
• Qwen3-Omni Technical Report
• Qwen3 Technical Report
• VibeCheck: Using Active Acoustic Tactile Sensing for Contact-Rich Manipulation
• VLAS: Vision-Language-Action Model With Speech Instructions For Customized Robot Manipulation

Related papers

• FMT x: An Efficient and Asymptotically Optimal Extension of the Fast Marching Tree for Dynamic Replanning
• MPCFormer: A physics-informed data-driven approach for explainable socially-aware autonomous driving
• RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies
• HRDexDB: A 4D Dexterous Grasping Dataset Across Human and Multiple Robot Embodiments
• APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies
• Fine-tuning is Not Enough: A Parallel Framework for Collaborative Imitation and Reinforcement Learning in End-to-end Autonomous Driving