Vision-Language-Action Autonomous Driving Agent with Language-based Memory

arXiv:2609.38641 · cs.CV, cs.RO · Submitted 2026-09-29 · Read on arXiv

cs.CV, cs.RO

Submitted: 2026-09-29

Updated: 2026-09-29

Project page: https://kaiyan289.github.io/projects/ad-memo/ABSTRACT

Terminology

• Accuracy — 21× in this paper · explained in 1 episode
• Actor — 21× in this paper · explained in 3 episodes
• Decode — 14× in this paper · explained in 2 episodes
• Ablation — 12× in this paper · explained in 1 episode
• GRPO — 12× in this paper · explained in 6 episodes
• Benchmark — 10× in this paper · explained in 2 episodes
• Reinforcement Learning — 7× in this paper · explained in 4 episodes
• Vulnerable Road Users — 7× in this paper
• Fine-Tuning — 6× in this paper · explained in 5 episodes
• Adam — 5× in this paper · explained in 2 episodes
• Credit assignment — 5× in this paper · explained in 3 episodes
• Exposure Bias — 5× in this paper · explained in 2 episodes
• Large Language Model — 4× in this paper
• Supervised Fine-Tuning — 4× in this paper
• Visual Question Answering — 4× in this paper
• Foundation Model — 3× in this paper · explained in 3 episodes
• In-Context Learning — 3× in this paper · explained in 4 episodes
• Multiple-Choice Question — 3× in this paper
• Supervised Fine-Tuning (SFT) — 3× in this paper · explained in 8 episodes
• Two-stage training — 3× in this paper · explained in 2 episodes
• Adaptive Reasoning — 2× in this paper · explained in 2 episodes
• Agentic Framework — 2× in this paper · explained in 6 episodes
• Alignment — 2× in this paper · explained in 1 episode
• Chain-of-Thought (CoT) — 2× in this paper · explained in 4 episodes
• Exponential Moving Average — 2× in this paper
• Exponential Moving Average (EMA) — 2× in this paper · explained in 2 episodes
• Grounding — 2× in this paper · explained in 3 episodes
• Reinforcement Learning (RL) — 2× in this paper · explained in 25 episodes
• Vision-Language Models (VLMs) — 2× in this paper · explained in 6 episodes
• Vision-Language-Action (VLA) Models — 2× in this paper · explained in 2 episodes
• Active Perception — 1× in this paper · explained in 2 episodes
• Adaptive Planning — 1× in this paper · explained in 1 episode
• Chain-of-Thought Prompting — 1× in this paper · explained in 2 episodes
• Core — 1× in this paper · explained in 2 episodes
• Diffusion Model — 1× in this paper · explained in 8 episodes
• Driving Vision-Language-Action — 1× in this paper
• Foundation Models — 1× in this paper · explained in 7 episodes
• Generalizability — 1× in this paper · explained in 6 episodes
• Group Relative Policy Optimization — 1× in this paper
• Group Relative Policy Optimization (GRPO) — 1× in this paper · explained in 5 episodes
• Knowledge Graph — 1× in this paper · explained in 7 episodes
• Knowledge graphs — 1× in this paper · explained in 2 episodes
• Large Language Model (LLM) — 1× in this paper · explained in 3 episodes
• Minimum Average Displacement Error — 1× in this paper
• Minimum Final Displacement Error — 1× in this paper
• Online Reinforcement Learning — 1× in this paper · explained in 2 episodes
• RAG — 1× in this paper · explained in 3 episodes
• Retrieval-Augmented Generation (RAG) — 1× in this paper · explained in 16 episodes
• Robustness — 1× in this paper · explained in 5 episodes
• Transformer — 1× in this paper · explained in 2 episodes
• VLA Models — 1× in this paper · explained in 3 episodes
• Verifiability — 1× in this paper · explained in 2 episodes
• Verifiable Rewards — 1× in this paper · explained in 2 episodes
• Verification — 1× in this paper · explained in 2 episodes
• Verifier — 1× in this paper · explained in 2 episodes
• World Model — 1× in this paper · explained in 7 episodes

Sources

• Qwen3-VL Technical Report
• Instant NuRec: Feed-Forward 3D Gaussian Reconstruction for Driving Scene Simulation
• MindVLA-U1: VLA Beats VA with Unified Streaming Architecture for Autonomous Driving
• Think as Needed: Geometry-Driven Adaptive Perception for Autonomous Driving
• Occluded nuScenes: A Multi-Sensor Dataset for Evaluating Perception Robustness in Automated Driving
• VLN-AVP: Zero-Shot Vision-Language Navigation with Hybrid Long-Short-Term Memory for Autonomous Valet Parking
• UniDriveVLA: Unifying Understanding, Perception, and Action Planning for Autonomous Driving
• Understanding Time Variations of DNN Inference in Autonomous Driving
• Goal2Skill: Long-Horizon Manipulation with Adaptive Planning and Reflection
• DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
• CLEAR: Closed-Loop Reinforcement Learning at Scale for End-to-End Autonomous Driving
• OpenAI GPT-5 System Card
• MEM: Multi-Scale Embodied Memory for Vision Language Action Models
• Alpamayo-R1: Bridging Reasoning and Action Prediction for Generalizable Autonomous Driving in the Long Tail
• LiloDriver: A Lifelong Learning Framework for Closed-loop Motion Planning in Long-tail Autonomous Driving Scenarios
• WaymoQA: A Multi-View Visual Question Answering Dataset for Safety-Critical Reasoning in Autonomous Driving
• PRAM-R: A Perception-Reasoning-Action-Memory Framework with LLM-Guided Modality Routing for Adaptive Autonomous Driving

Related papers

• Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
• AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
• Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
• MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
• TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
• A Survey on Efficient Vision-Language-Action Models