Learning Social Navigation from Internet Videos in the Policy State Space

arXiv:2609.37476 · cs.RO, cs.CV, cs.LG · Submitted 2026-09-26 · Read on arXiv

cs.RO, cs.CV, cs.LG

Submitted: 2026-09-26

Updated: 2026-10-01

Code: https://github.com/ultralytics/ultralytics

Project page: https://jiaming.im/VideoSocNav

Terminology

• Reinforcement Learning — 9× in this paper · explained in 4 episodes
• Benchmark — 7× in this paper · explained in 2 episodes
• Ablation — 6× in this paper · explained in 1 episode
• Fine-Tuning — 6× in this paper · explained in 5 episodes
• Transformer — 5× in this paper · explained in 2 episodes
• Deep Reinforcement Learning — 4× in this paper · explained in 2 episodes
• Actor — 3× in this paper · explained in 3 episodes
• Abstraction — 2× in this paper · explained in 1 episode
• Generalization — 2× in this paper · explained in 13 episodes
• 3D Gaussian Splatting — 1× in this paper · explained in 2 episodes
• Benchmarking — 1× in this paper · explained in 3 episodes
• Computer Vision and Pattern Recognition — 1× in this paper
• Gaussian Splatting — 1× in this paper · explained in 4 episodes
• Generative Model — 1× in this paper · explained in 2 episodes
• Generative Models — 1× in this paper · explained in 2 episodes
• Imitation Learning (IL) — 1× in this paper · explained in 2 episodes

Sources

• Image2Sim: Scaling Embodied Navigation via Generative Neural Simulator
• VEGA: Learning Navigation VLAs from In-the-Wild Egocentric Video with Geometric Trajectory Supervision
• BoT-SORT: Robust Associations Multi-Pedestrian Tracking
• Proximal Policy Optimization Algorithms
• On Evaluation of Embodied Navigation Agents
• Change-Robust Online Topological Memory for Long-Term Relocalization and Semantic Navigation

Related papers

• FMT x: An Efficient and Asymptotically Optimal Extension of the Fast Marching Tree for Dynamic Replanning
• MPCFormer: A physics-informed data-driven approach for explainable socially-aware autonomous driving
• RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies
• HRDexDB: A 4D Dexterous Grasping Dataset Across Human and Multiple Robot Embodiments
• APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies
• Fine-tuning is Not Enough: A Parallel Framework for Collaborative Imitation and Reinforcement Learning in End-to-end Autonomous Driving