ChatBEV: Empowering Traffic Scene Understanding and Simulation via Vision-Language Model
cs.CV, cs.AI
Submitted: 2025-03-18
Updated: 2026-09-05
Code: https://github.com/xuqingyao/ChatBEV
License: http://creativecommons.org/licenses/by/4.0/
The gist: Comprehensive traffic scene understanding is a foundational capability for Intelligent Transportation Systems (ITS) underpinning applications such as traffic simulation.
Terminology
Abstract
Comprehensive traffic scene understanding is a foundational capability for Intelligent Transportation Systems (ITS) underpinning applications such as traffic simulation. While VisionLanguage Models (VLMs) have demonstrated strong reasoning potential, their application to Bird's-Eye View (BEV) maps in traffic contexts remains limited by narrow task definitions and scarce annotated data. We introduce ChatBEV-QA, a large-scale BEV VQA benchmark of 137K+ QA pairs, designed to evaluate global scene understanding, vehicle-lane interactions, and vehiclevehicle interactions within complex traffic environments. Building on this, we fine-tune ChatBEV, a specialized VLM that accurately interprets diverse scene understanding queries from BEV maps. To demonstrate downstream utility in ITS applications, we integrate ChatBEV into a language-guided traffic simulation framework. Its global understanding and navigation reasoning provide crucial context-aware guidance, reducing trajectory displacement error by up to 20.9% and scenario collision rates by up to 37.9% over text-only baselines.
Sources
- GPT-4 Technical Report
- NuPlan: A closed-loop ML-based planning benchmark for autonomous vehicles
- Talk2BEV: Language-enhanced Bird's-eye View Maps for Autonomous Driving
- Talk2Car: Taking Control of Your Self-Driving Car
- InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model
- BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
- Language Conditioned Traffic Generation
- Gemini: A Family of Highly Capable Multimodal Models
- Language Prompt for Autonomous Driving
- Language-Driven Interactive Traffic Trajectory Generation
- Language-Guided Traffic Simulation via Scene-Level Diffusion
Related papers
- Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
- AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
- Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
- MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
- TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
- A Survey on Efficient Vision-Language-Action Models