Workspace Models: Lightweight Robotic Memory via Saliency-Driven Supervision
cs.RO, cs.AI
Submitted: 2026-09-17
Updated: 2026-09-30
Comments: 26 pages; CoRL 2026; 11 figures
License: http://creativecommons.org/licenses/by/4.0/
The gist: Complex robotic manipulation tasks frequently require a long-term memory of past events and actions.
Terminology
Abstract
Complex robotic manipulation tasks frequently require a long-term memory of past events and actions. As conditioning on full histories renders policies prone to spurious correlations and degrades performance, many approaches to policy memory involve compressing historical information through expensive VLM queries in-the-loop to process only task-salient information. In this paper, we propose an alternative approach in which computationally intensive VLM queries are made during train-time to learn a lightweight latent memory that can be efficiently queried at deployment time. Our representation, which we call the workspace token, is trained by (1) using a VLM to identify current and historical information necessary for completing a task, then (2) distilling these into the workspace token using a set-reconstruction decoder loss. In both simulation and hardware, we show that the workspace token can be used as a drop-in replacement for observations during deployment, enabling policies to solve memory-intensive tasks without the need for VLM reasoning in-the-loop. Interestingly, we found that workspace tokens are not only more lightweight but also lead to better policy performance.
Sources
- Qwen3-VL Technical Report
- GABRIL: Gaze-Based Regularization for Mitigating Causal Confusion in Imitation Learning
- The Consciousness Prior
- $\pi_0$: A Vision-Language-Action Flow Model for General Robot Control
- Scaling Robot Policy Learning via Zero-Shot Labeling with Foundation Models
- End-to-End Object Detection with Transformers
- ShapeNet: An Information-Rich 3D Model Repository
- Diffusion Policy: Visuomotor Policy Learning via Action Diffusion
- MolmoPoint: Better Pointing for VLMs with Grounding Tokens
- Manipulate-Anything: Automating Real-World Robots using Vision-Language Models
- AutoFocus-IL: VLM-based Saliency Maps for Data-Efficient Visual Imitation Learning without Extra Human Annotations
- $\pi_{0.5}$: a Vision-Language-Action Model with Open-World Generalization
- OpenVLA: An Open-Source Vision-Language-Action Model
- ViSaRL: Visual Reinforcement Learning Guided by Human Saliency
- BPP: Long-Context Robot Imitation Learning by Focusing on Key History Frames
- Recurrent Models of Visual Attention
- FiLM: Visual Reasoning with a General Conditioning Layer
- U-Net: Convolutional Networks for Biomedical Image Segmentation
- Efficiently Guiding Imitation Learning Agents with Human Gaze
- The Pitfalls of Imitation Learning when Actions are Continuous
Related papers
- FMT x: An Efficient and Asymptotically Optimal Extension of the Fast Marching Tree for Dynamic Replanning
- MPCFormer: A physics-informed data-driven approach for explainable socially-aware autonomous driving
- RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies
- HRDexDB: A 4D Dexterous Grasping Dataset Across Human and Multiple Robot Embodiments
- APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies
- Fine-tuning is Not Enough: A Parallel Framework for Collaborative Imitation and Reinforcement Learning in End-to-end Autonomous Driving