Selective Agent Guidance via Entropy: Learning Autonomous Policies from Imperfect VLM Teachers
cs.AI, cs.CL, cs.LG
Submitted: 2026-09-01
Updated: 2026-09-02
Comments: 9 pages, 3 figures, 4 tables in the main text, 27 pages, 4 figures, 9 tables including Appendix
Journal ref: EMNLP 2026 Findings
Code: https://github.com/giobin/SAGE
License: http://creativecommons.org/licenses/by/4.0/
The gist: Vision-Language Models (VLMs) provide useful priors for interactive decision-making, but using them directly as policies is expensive and brittle: they must be queried at every step, do not improve
Terminology
Abstract
Vision-Language Models (VLMs) provide useful priors for interactive decision-making, but using them directly as policies is expensive and brittle: they must be queried at every step, do not improve from environment interaction, and can repeat systematic errors. We study how to learn a cheap autonomous policy from an online, expensive, and imperfect but informative VLM teacher. We propose SAGE (Selective Agent Guidance via Entropy), a framework that queries a VLM only when the learner is uncertain, executes the suggested action during training, and distills guidance into a lightweight Reinforcement Learning (RL) policy. Because VLM advice is not always reliable, SAGE can weight teacher-action distillation using environment-derived advantages rather than treating all suggestions as equally useful. Across sparse-reward visual reasoning and navigation tasks, SAGE learns policies that act without VLM guidance at evaluation time and improves over unguided RL in several environments, including settings where the learned policy exceeds its VLM teacher. The results show that selective guidance is most beneficial when the VLM can help the agent discover high-reward trajectories, and less useful when unguided exploration already succeeds or teacher actions do not lead to informative experience. SAGE also reduces VLM usage by prompting the teacher only on a fraction of training steps and requiring no VLM calls at deployment. Overall, our results suggest that VLMs don't need to be used as fixed policies to be useful; they can instead act as temporary, imperfect sources of guidance whose value is tested and internalized through interaction.
Sources
- RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
- Gemma 3 Technical Report
- OpenVLA: An Open-Source Vision-Language-Action Model
- Eureka: Human-Level Reward Design via Coding Large Language Models
- VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Training
- AWAC: Accelerating Online Reinforcement Learning with Offline Datasets
- Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning
- Policy Distillation
- Proximal Policy Optimization Algorithms
- Gymnasium: A Standard Interface for Reinforcement Learning Environments
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems
- VSAL: A Vision Solver with Adaptive Layouts for Graph Property Detection