Less Language, More Latents: Annotation-Efficient VLAs for Driving

arXiv:2609.27747 · cs.RO, cs.LG · Submitted 2026-09-23 · Read on arXiv

cs.RO, cs.LG

Submitted: 2026-09-23

Updated: 2026-09-23

Terminology

• Transformer — 11× in this paper · explained in 2 episodes
• Accuracy — 8× in this paper · explained in 1 episode
• Action Space — 7× in this paper · explained in 1 episode
• Benchmark — 5× in this paper · explained in 2 episodes
• Commentary — 4× in this paper · explained in 2 episodes
• Embeddings — 4× in this paper · explained in 2 episodes
• World Model — 4× in this paper · explained in 7 episodes
• Ablation — 3× in this paper · explained in 1 episode
• Chain-of-Thought (CoT) — 3× in this paper · explained in 4 episodes
• Vision-language models — 3× in this paper · explained in 2 episodes
• Abstraction — 2× in this paper · explained in 1 episode
• Alignment — 2× in this paper · explained in 1 episode
• Benchmarking — 2× in this paper · explained in 3 episodes
• Clustering — 2× in this paper · explained in 2 episodes
• Decode — 2× in this paper · explained in 2 episodes
• Fine-Tuning — 2× in this paper · explained in 5 episodes
• Foundation Model — 2× in this paper · explained in 3 episodes
• Foundation Models — 2× in this paper · explained in 7 episodes
• Grounding — 2× in this paper · explained in 3 episodes
• Interpretability — 2× in this paper · explained in 15 episodes
• Latent Action Driving Annotations — 2× in this paper
• Latent Space — 2× in this paper · explained in 7 episodes
• Pseudo-labels — 2× in this paper · explained in 2 episodes
• Computer Vision and Pattern Recognition Conference — 1× in this paper
• Consistency — 1× in this paper · explained in 3 episodes
• Core — 1× in this paper · explained in 2 episodes
• Diffusion Model — 1× in this paper · explained in 8 episodes
• Domain Shift — 1× in this paper · explained in 4 episodes
• EMMA — 1× in this paper · explained in 2 episodes
• Explainability — 1× in this paper · explained in 3 episodes
• Quantization — 1× in this paper · explained in 12 episodes
• Robustness — 1× in this paper · explained in 5 episodes
• Vision-Language Models (VLMs) — 1× in this paper · explained in 6 episodes

Sources

• TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Driving
• Safety-Enhanced Autonomous Driving Using Interpretable Sensor Fusion Transformer
• Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baseline
• Planning-oriented Autonomous Driving
• VAD: Vectorized Scene Representation for Efficient Autonomous Driving
• Bench2Drive: Towards Multi-Ability Benchmarking of Closed-Loop End-To-End Autonomous Driving
• SimLingo: Vision-Only Closed-Loop Autonomous Driving with Language-Action Alignment
• ORION: A Holistic End-to-End Autonomous Driving Framework by Vision-Language Instructed Action Generation
• OmniDrive: A Holistic Vision-Language Dataset for Autonomous Driving with Counterfactual Reasoning
• Imitating Latent Policies from Observation
• Learning to Act without Actions
• Genie: Generative Interactive Environments
• Think2Drive: Efficient Reinforcement Learning by Thinking in Latent World Model for Quasi-Realistic Autonomous Driving (in CARLA-v2)
• RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
• OpenVLA: An Open-Source Vision-Language-Action Model
• Octo: An Open-Source Generalist Robot Policy
• GPT-Driver: Learning to Drive with GPT
• ADriver-I: A General World Model for Autonomous Driving
• DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models
• DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model

Related papers

• FMT x: An Efficient and Asymptotically Optimal Extension of the Fast Marching Tree for Dynamic Replanning
• MPCFormer: A physics-informed data-driven approach for explainable socially-aware autonomous driving
• RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies
• HRDexDB: A 4D Dexterous Grasping Dataset Across Human and Multiple Robot Embodiments
• APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies
• Fine-tuning is Not Enough: A Parallel Framework for Collaborative Imitation and Reinforcement Learning in End-to-end Autonomous Driving