doPlan: A Variable-Horizon Dataset for Multi-Stage Language-Conditioned Planning in Autonomous Driving

arXiv:2609.38028 · cs.RO, cs.AI, cs.CV, cs.LG · Submitted 2026-09-29 · Read on arXiv

cs.RO, cs.AI, cs.CV, cs.LG

Submitted: 2026-09-29

Updated: 2026-09-29

Code: https://github.com/NVlabs/alpamayo2

Terminology

• Alignment — 8× in this paper · explained in 1 episode
• Grounding — 8× in this paper · explained in 3 episodes
• Benchmark — 6× in this paper · explained in 2 episodes
• Consistency — 2× in this paper · explained in 3 episodes
• AVERAGE DISPLACEMENT ERROR — 1× in this paper
• Ablation — 1× in this paper · explained in 1 episode
• Actionability — 1× in this paper · explained in 3 episodes
• Adaptive Reasoning — 1× in this paper · explained in 2 episodes
• Affordance — 1× in this paper · explained in 1 episode
• Deferral — 1× in this paper · explained in 2 episodes
• Fine-Tuning — 1× in this paper · explained in 5 episodes
• Foundation Model — 1× in this paper · explained in 3 episodes
• Human-in-the-Loop — 1× in this paper · explained in 3 episodes
• International Conference on Automation Science and Engineering — 1× in this paper
• Multimodal Dataset — 1× in this paper · explained in 2 episodes

Sources

• Do As I Can, Not As I Say: Grounding Language in Robotic Affordances
• Inner Monologue: Embodied Reasoning through Planning with Language Models
• PaLM-E: An Embodied Multimodal Language Model
• RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
• Textual Explanations for Self-Driving Vehicles
• Grounding Human-to-Vehicle Advice for Self-driving Vehicles
• DRAMA: Joint Risk Localization and Captioning in Driving
• LingoQA: Visual Question Answering for Autonomous Driving
• CoVLA: Comprehensive Vision-Language-Action Dataset for Autonomous Driving
• Natural Language Instructions for Scene-Responsive Human-in-the-Loop Motion Planning in Autonomous Driving using Vision-Language-Action Models
• NuPlan: A closed-loop ML-based planning benchmark for autonomous vehicles
• OpenEMMA: Open-Source Multimodal Model for End-to-End Autonomous Driving
• Alpamayo-R1: Bridging Reasoning and Action Prediction for Generalizable Autonomous Driving in the Long Tail
• LMDrive: Closed-Loop End-to-End Driving with Large Language Models
• LaMPilot: An Open Benchmark Dataset for Autonomous Driving with Language Model Programs
• SimLingo: Vision-Only Closed-Loop Autonomous Driving with Language-Action Alignment
• Vega: Learning to Drive with Natural Language Instructions
• Grounding Complex Natural Language Commands for Temporal Tasks in Unseen Environments
• Learning Transferable Visual Models From Natural Language Supervision
• Automated Data Curation Using GPS & NLP to Generate Instruction-Action Pairs for Autonomous Vehicle Vision-Language Navigation Datasets

Related papers

• FMT x: An Efficient and Asymptotically Optimal Extension of the Fast Marching Tree for Dynamic Replanning
• MPCFormer: A physics-informed data-driven approach for explainable socially-aware autonomous driving
• RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies
• HRDexDB: A 4D Dexterous Grasping Dataset Across Human and Multiple Robot Embodiments
• APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies
• Fine-tuning is Not Enough: A Parallel Framework for Collaborative Imitation and Reinforcement Learning in End-to-end Autonomous Driving