HERMES: A Holistic End-to-End Risk-Aware Multimodal Embodied System with Vision-Language Models for Long-Tail Autonomous Driving

arXiv:2602.00993 · cs.RO, cs.AI · Submitted 2026-02-01 · Read on arXiv

cs.RO, cs.AI

Submitted: 2026-02-01

Updated: 2026-09-24

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Foundation Model — 11× in this paper · explained in 3 episodes
• Embeddings — 10× in this paper · explained in 2 episodes
• Transformer — 10× in this paper · explained in 2 episodes
• Ablation — 8× in this paper · explained in 1 episode
• Benchmark — 8× in this paper · explained in 2 episodes
• Adaptation — 7× in this paper · explained in 1 episode
• Foundation Models — 7× in this paper · explained in 7 episodes
• Robustness — 7× in this paper · explained in 5 episodes
• Semantic Understanding — 7× in this paper · explained in 2 episodes
• Ablation Studies — 3× in this paper · explained in 1 episode
• Alignment — 3× in this paper · explained in 1 episode
• Conference on Robot Learning — 3× in this paper
• Fine-Tuning — 3× in this paper · explained in 5 episodes
• Latent Space — 3× in this paper · explained in 7 episodes
• Multimodal Fusion — 3× in this paper · explained in 3 episodes
• Reinforcement Learning — 3× in this paper · explained in 4 episodes
• Steering — 3× in this paper · explained in 2 episodes
• World Model — 3× in this paper · explained in 7 episodes
• Ablation study — 2× in this paper · explained in 1 episode
• Accuracy — 2× in this paper · explained in 1 episode
• Conference on Computer Vision and Pattern Recognition — 2× in this paper
• Consistency — 2× in this paper · explained in 3 episodes
• Core — 2× in this paper · explained in 2 episodes
• Multimodal Inputs — 2× in this paper · explained in 2 episodes
• AND F OREIGN O BJECT D EBRIS — 1× in this paper
• Abstraction — 1× in this paper · explained in 1 episode
• Adam — 1× in this paper · explained in 2 episodes
• Adaptive Multimodal Fusion — 1× in this paper · explained in 1 episode
• Adaptive Reasoning — 1× in this paper · explained in 2 episodes
• Affordance — 1× in this paper · explained in 1 episode
• Aggregation — 1× in this paper · explained in 1 episode
• Artificial Intelligence (AI) — 1× in this paper · explained in 2 episodes
• Attention Mechanism — 1× in this paper · explained in 2 episodes
• Closed-Loop Control — 1× in this paper · explained in 2 episodes
• Cross-Attention Mechanism — 1× in this paper · explained in 2 episodes
• Data Augmentation — 1× in this paper · explained in 7 episodes
• Diffusion Model — 1× in this paper · explained in 8 episodes
• Distillation — 1× in this paper · explained in 2 episodes
• Distribution Shift — 1× in this paper · explained in 5 episodes
• Final Displacement Error — 1× in this paper
• Generalization — 1× in this paper · explained in 13 episodes
• Gradient Flow — 1× in this paper · explained in 2 episodes
• Human-in-the-Loop — 1× in this paper · explained in 3 episodes
• IEEE International Conference on Computer Vision — 1× in this paper
• Interpretability — 1× in this paper · explained in 15 episodes
• Knowledge Distillation — 1× in this paper · explained in 11 episodes
• Large Language Models (LLMs) — 1× in this paper · explained in 6 episodes
• Modular Architecture — 1× in this paper · explained in 4 episodes
• Multi-Agent System — 1× in this paper · explained in 6 episodes
• Multimodal Learning — 1× in this paper · explained in 2 episodes
• Rater Feedback Score — 1× in this paper
• Reasoning Traces — 1× in this paper · explained in 3 episodes
• Semantic Alignment — 1× in this paper · explained in 3 episodes
• Structured Reasoning — 1× in this paper · explained in 3 episodes
• Trace — 1× in this paper · explained in 2 episodes
• Unified Framework — 1× in this paper · explained in 5 episodes
• VLA Models — 1× in this paper · explained in 3 episodes
• Vector Representations — 1× in this paper · explained in 2 episodes
• Verification — 1× in this paper · explained in 2 episodes
• Vision Transformer (ViT) — 1× in this paper · explained in 2 episodes
• alpha — 1× in this paper · explained in 1 episode

Sources

• End to End Learning for Self-Driving Cars
• DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model
• EMMA: End-to-End Multimodal Model for Autonomous Driving
• DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models
• WOD-E2E: Waymo Open Dataset for End-to-End Driving in Challenging Long-tail Scenarios
• Planning Safety Trajectories with Dual-Phase, Physics-Informed, and Transportation Knowledge-Driven Large Language Models
• SEAL: Vision-Language Model-Based Safe End-to-End Cooperative Autonomous Driving with Adaptive Long-Tail Modeling
• LightEMMA: A Longitudinal Evaluation of Vision-Language Models for Autonomous Driving
• OpenDriveVLA: Towards End-to-end Autonomous Driving with Large Vision Language Action Model
• AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuning
• GAIA-1: A Generative World Model for Autonomous Driving
• DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generation
• A Survey of World Models for Autonomous Driving
• Alpamayo-R1: Bridging Reasoning and Action Prediction for Generalizable Autonomous Driving in the Long Tail
• M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation
• Qwen3-VL Technical Report

Related papers

• FMT x: An Efficient and Asymptotically Optimal Extension of the Fast Marching Tree for Dynamic Replanning
• MPCFormer: A physics-informed data-driven approach for explainable socially-aware autonomous driving
• RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies
• HRDexDB: A 4D Dexterous Grasping Dataset Across Human and Multiple Robot Embodiments
• APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies
• Fine-tuning is Not Enough: A Parallel Framework for Collaborative Imitation and Reinforcement Learning in End-to-end Autonomous Driving