Generative AI for Autonomous Driving: Frontiers and Opportunities

arXiv:2505.08854 · cs.CV, cs.AI, cs.RO · Submitted 2025-05-13 · Read on arXiv

cs.CV, cs.AI, cs.RO

Submitted: 2025-05-13

Updated: 2026-10-06

Code: https://github.com/taco-group/GenAI4AD

Terminology

• Generative AI — 300× in this paper · explained in 2 episodes
• Generative Model — 114× in this paper · explained in 2 episodes
• Transformer — 111× in this paper · explained in 2 episodes
• Generative Models — 94× in this paper · explained in 2 episodes
• Diffusion Model — 88× in this paper · explained in 8 episodes
• Large Language Model — 77× in this paper
• Conference on Computer Vision and Pattern Recognition — 63× in this paper
• Diffusion Models — 63× in this paper · explained in 6 episodes
• Fidelity — 48× in this paper · explained in 2 episodes
• Benchmark — 40× in this paper · explained in 2 episodes
• Foundation Model — 40× in this paper · explained in 3 episodes
• Digital Twin — 39× in this paper · explained in 4 episodes
• Generalization — 33× in this paper · explained in 13 episodes
• Consistency — 30× in this paper · explained in 3 episodes
• Foundation Models — 29× in this paper · explained in 7 episodes
• World Model — 29× in this paper · explained in 7 episodes
• Intelligent Transportation Systems — 28× in this paper
• Reinforcement Learning — 28× in this paper · explained in 4 episodes
• Alignment — 24× in this paper · explained in 1 episode
• Gaussian Splatting — 23× in this paper · explained in 4 episodes
• Robustness — 23× in this paper · explained in 5 episodes
• Fine-Tuning — 21× in this paper · explained in 5 episodes
• Point Clouds — 21× in this paper · explained in 2 episodes
• Representation Learning — 21× in this paper · explained in 2 episodes
• Embodied AI — 20× in this paper · explained in 2 episodes
• Generative Modeling — 20× in this paper · explained in 3 episodes
• Generative Adversarial Network — 17× in this paper
• Latent Space — 16× in this paper · explained in 7 episodes
• Adaptation — 15× in this paper · explained in 1 episode
• Core — 14× in this paper · explained in 2 episodes
• Embeddings — 14× in this paper · explained in 2 episodes
• In European Conference on Computer Vision — 14× in this paper
• Adaptability — 13× in this paper · explained in 1 episode
• Federated Learning — 13× in this paper · explained in 14 episodes
• International Conference on Robotics and Automation — 12× in this paper
• 3D Gaussian Splatting — 11× in this paper · explained in 2 episodes
• Accuracy — 11× in this paper · explained in 1 episode
• Temporal consistency — 11× in this paper · explained in 2 episodes
• Visual Question Answering — 10× in this paper
• Accountability — 9× in this paper · explained in 2 episodes
• Hallucination — 9× in this paper · explained in 3 episodes
• Intelligent Vehicles Symposium — 9× in this paper
• Interpretability — 9× in this paper · explained in 15 episodes
• Multimodal Large Language Models — 9× in this paper
• Personalization — 9× in this paper · explained in 2 episodes
• Adam — 8× in this paper · explained in 2 episodes
• Latent Diffusion Model — 8× in this paper · explained in 2 episodes
• Synthetic Data Generation — 8× in this paper · explained in 5 episodes
• Actor — 7× in this paper · explained in 3 episodes
• Cooperation — 7× in this paper · explained in 2 episodes
• Distillation — 7× in this paper · explained in 2 episodes
• Grounding — 7× in this paper · explained in 3 episodes
• Large Language Models (LLMs) — 7× in this paper · explained in 6 episodes
• Multimodal Inputs — 7× in this paper · explained in 2 episodes
• Quantization — 7× in this paper · explained in 12 episodes
• 3D Gaussian Splatting (3DGS) — 6× in this paper · explained in 4 episodes
• Agent Interactions — 6× in this paper · explained in 1 episode
• Data Augmentation — 6× in this paper · explained in 7 episodes
• Latent Representation — 6× in this paper · explained in 2 episodes
• Regularization — 6× in this paper · explained in 3 episodes
• Unified Framework — 6× in this paper · explained in 5 episodes
• Adversarial Training — 5× in this paper · explained in 1 episode
• Contrastive Learning — 5× in this paper · explained in 5 episodes
• RAG — 5× in this paper · explained in 3 episodes
• Transformer Architecture — 5× in this paper · explained in 2 episodes
• Uncertainty Quantification — 5× in this paper · explained in 8 episodes
• Verification — 5× in this paper · explained in 2 episodes
• Attention Mechanism — 4× in this paper · explained in 2 episodes
• Deep Reinforcement Learning — 4× in this paper · explained in 2 episodes
• Generalizability — 4× in this paper · explained in 6 episodes
• Loss Function — 4× in this paper · explained in 2 episodes
• Mode Collapse — 4× in this paper · explained in 2 episodes
• Self-Supervised Learning — 4× in this paper · explained in 7 episodes
• Transfer Learning — 4× in this paper · explained in 10 episodes
• Transferability — 4× in this paper · explained in 4 episodes
• Variational Autoencoder (VAE) — 4× in this paper · explained in 3 episodes
• Winter Conference on Applications of Computer Vision — 4× in this paper
• Benchmarking — 3× in this paper · explained in 3 episodes
• Chain-of-Thought Prompting — 3× in this paper · explained in 2 episodes
• Domain Adaptation — 3× in this paper · explained in 5 episodes
• Explainability — 3× in this paper · explained in 3 episodes
• Failure Modes — 3× in this paper · explained in 3 episodes
• Human-in-the-Loop — 3× in this paper · explained in 3 episodes
• In International Conference on Learning Representations — 3× in this paper
• Instruction Tuning — 3× in this paper · explained in 2 episodes
• LLM-as-a-Judge — 3× in this paper · explained in 3 episodes
• Latent Diffusion Models — 3× in this paper · explained in 2 episodes
• Multi-Agent System — 3× in this paper · explained in 5 episodes
• Multimodal Dataset — 3× in this paper · explained in 2 episodes
• Multimodal Learning — 3× in this paper · explained in 2 episodes
• Multimodal Reasoning — 3× in this paper · explained in 5 episodes
• Sim-to-Real Transfer — 3× in this paper · explained in 2 episodes
• Sparsity — 3× in this paper · explained in 4 episodes
• Steering — 3× in this paper · explained in 2 episodes
• AI Alignment — 2× in this paper · explained in 2 episodes
• Action Translation — 2× in this paper · explained in 1 episode
• Adaptive Signal Control — 2× in this paper · explained in 1 episode
• Affordance — 2× in this paper · explained in 1 episode
• Analyzer — 2× in this paper · explained in 1 episode
• Calibration — 2× in this paper · explained in 7 episodes

Sources

• nuScenes: A multimodal dataset for autonomous driving
• Specification and Validation of Autonomous Driving Systems: A Multilevel Semantic Framework
• Automatic Generation of Scenarios for System-level Simulation-based Verification of Autonomous Driving Systems
• Moving Forward: A Review of Autonomous Driving Software and Hardware Systems
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• EMMA: End-to-End Multimodal Model for Autonomous Driving
• YouTube-8M: A Large-Scale Video Classification Benchmark
• Mcity Data Collection for Automated Vehicles Study
• A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPT
• LLaMA: Open and Efficient Foundation Language Models
• Code Llama: Open Foundation Models for Code
• Scenario Dreamer: Vectorized Latent Diffusion for Generating Driving Simulation Environments
• DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models
• A Survey of World Models for Autonomous Driving
• Exploring the Interplay Between Video Generation and World Models in Autonomous Driving: A Survey
• From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge
• Generative AI in Transportation Planning: A Survey
• Vision-Language-Action (VLA) Models: Concepts, Progress, Applications and Challenges
• A Survey for Foundation Models in Autonomous Driving
• LLM4Drive: A Survey of Large Language Models for Autonomous Driving

Related papers

• Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
• AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
• Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
• MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
• TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
• A Survey on Efficient Vision-Language-Action Models