VIA: Visual Interface Agent for Robot Control

arXiv:2607.11119 · cs.RO, cs.AI · Submitted 2026-07-13 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: I'm Rosa, and with me are Dev and Taro, guest researcher.

Dev: Today's paper: "VIA: Visual Interface Agent for Robot Control".

Rosa: Robot manipulation requires complex skills like visual understanding, physical reasoning, and planning, which have been enhanced by foundation models (FMs).

Dev: First, who's behind it and why it matters.

Paper summary: Rosa: To recap what we've covered, "VIA: Visual Interface Agent for Robot Control" proposes recasting robot control as an agentic task where an off-the-shelf foundation model drives a manipulator using a browser-based three dee interface <ref:2607.11119#pg0,VIA: Visual Interface Agent for Robot Control>. The authors argue that instead of converting foundation models into vision-language-action models by fine-tuning them on specific robot data, which often results in smaller models, VIA tests whether the general competence of these large FMs is sufficient to control a robot through this visual interface.

Dev: It claims that by treating control as a visual tool-use task for agents, VIA avoids the limitations imposed by fine-tuning on action data and instead leverages the FM's existing vision and reasoning capabilities, which are already quite capable. The framework uses calibrated third-person RGB-D cameras to build a three dee point cloud scene that acts as the main workspace for the agent <ref:2607.11119#pg0>.

Taro: So, to summarize the core thesis, VIA isn't about teaching the model *how* to move a gripper directly; it’s about giving it a visual workspace and a set of general computer use tools so it can learn to orchestrate those actions through observation and intuitive commands. That seems like a significant departure from how we've approached robotics for some time.

Rosa: Right, that’s the gist of the paper—it tests if competence in operating software via visual interfaces is enough to achieve robot control without needing dedicated robot fine-tuning. The framework outlines an agentic loop where it observes, reasons, acts through tools like screenshot or gripper teleport via click, and repeats this until the task ends.

Dev: I think the paper highlights the structure of these Model Context Protocol tools as key; they are designed to be minimalist and ergonomic, wrapping human operations in little abstraction while still providing direct alternatives to continuous actions. That seems crucial for managing the loop rate effectively during execution.

Taro: When we look at the stated results, it’s quite compelling because it shows that this approach can solve diverse manipulation tasks zero-shot just from a minimal prompt stating only the goal. That speaks volumes about the generality of what these foundation models are already capable of handling.

Rosa: It does suggest that the capability to operate software through visual interfaces is a powerful skill, and when paired with the right setup, it can be directly applied to controlling a robot without requiring specialized training data for every single task.

Dev: I think this framework suggests that we might not always need massive amounts of specific robot interaction data to get good performance; instead, providing the right visual context and a set of well-designed tools could be the deciding factor in successful control.

Taro: So, the main takeaway is that robot control is being reframed as a visual tool-use task for agents, and we are using existing foundation models' general reasoning abilities as the engine to drive that task.

Rosa: This really sets up the next part of our discussion—how this concept translates into practical implications for how we build and deploy robotic systems in real environments.

Conclusion: Dev: Thinking about "VIA: Visual Interface Agent for Robot Control," the authors are essentially proposing a new way to connect foundation models to robotics by casting robot control as a visual tool-use task for agents, avoiding the need to fine-tune them into specific vision-language-action models.

Rosa: I think the real implication here is that we can start seeing robot control become another economically valuable agentic task that benefits directly from modern foundation models and agents at scale, provided we design a suitable interface for them to operate through.

Taro: If this holds up, it could mean that the future development path for robotics isn't always about creating bespoke policies for every single physical interaction, but more about building flexible agents capable of using common visual interfaces effectively across many different robot setups.

Dev: From an engineering viewpoint, it means we can shift our focus from painstakingly gathering massive amounts of robot-specific action data to designing robust MCP tools and interfaces that enable these general agents to function reliably in a closed-loop manner.

Rosa: And I’m curious about the long-term outlook: will this approach scale up effectively once we move beyond tabletop tasks and into more complex, unstructured environments where the agent has to handle things that are completely novel?

Taro: That’s where the challenge lies; we need to ensure that when the world misbehaves in a real environment, this agentic loop is resilient enough to re-plan effectively without getting stuck or making catastrophic errors.

Dev: The paper mentions a limitation regarding performance scaling: it notes that reliance on "the best frontier models" due to the novel interface demanding high general capability can result in slow inference times during operation.

Rosa: So, while the potential for leveraging general model scaling is there, we still face practical hurdles related to inference speed and ensuring that the agent's performance holds up when faced with completely novel real-world scenarios.

Taro: The framework also supports future improvements like Automatic Tool Improvement and Learning via Reflection, which suggests that the system itself can evolve its own way of interacting with the environment over time based on feedback it receives.

Dev: If we can solve the inference speed issue and make those reflection mechanisms work smoothly, then this concept could genuinely unlock a new era where general agents can tackle a wider variety of physical manipulation tasks without needing massive, task-specific training regimes.

Rosa: It’s an exciting direction because it suggests that robot control is poised to harvest general gains from foundation model scaling, where each new generation of general capability transfers to the robot for free, with no fine-tuning required.

Stanford University

cs.RO, cs.AI

Submitted: 2026-07-13

Updated: 2026-10-07

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 83/100

The gist: Robot manipulation requires complex skills like visual understanding, physical reasoning, and planning, which have been enhanced by foundation models (FMs).

Key concepts

Agentic Task: This reframes robot control as a series of steps an intelligent agent must perform autonomously. The agent observes the environment, reasons about its current state, selects an action using tools, and repeats this cycle until the goal is achieved. It emphasizes planning and closed-loop error recovery.
Model Context Protocol (MCP): These are a small set of general tools that allow the AI agent to interact with the 3D robot interface. They include observation tools like taking screenshots and pose readers, and action tools like clicking or dragging. These tools wrap human operations in a minimal way, giving the agent direct control.
Zero-Shot Control: This means an AI agent can successfully perform a complex robot manipulation task immediately after being given only the goal as a simple prompt. VIA demonstrated that frontier models can achieve this without any prior training on that specific robot, relying instead on their general visual and reasoning skills.
Closed-Loop Control: This describes the continuous feedback mechanism in VIA where the agent observes an action's result (via a new screenshot), reasons about whether it succeeded, and adjusts its next command. This loop allows the agent to correct errors in real-time during manipulation.

Terminology

Foundation Model — 19× in this paper · explained in 3 episodes
Fine-Tuning — 16× in this paper · explained in 5 episodes
Foundation Models — 14× in this paper · explained in 7 episodes
Agentic Task — 10× in this paper · explained in 1 episode
Claude Code — 9× in this paper
Visual Interface Agent — 9× in this paper
Abstraction — 6× in this paper · explained in 1 episode
Closed-Loop Control — 6× in this paper · explained in 2 episodes
Foundation Models (FMs) — 6× in this paper · explained in 2 episodes
Agentic Loop — 5× in this paper · explained in 1 episode
Model Context Protocol — 5× in this paper
Benchmark — 4× in this paper · explained in 2 episodes
Benchmarking — 3× in this paper · explained in 3 episodes
Core — 3× in this paper · explained in 2 episodes
Generalization — 3× in this paper · explained in 13 episodes
Model Context Protocol (MCP) — 3× in this paper · explained in 8 episodes
Vision-Language-Action (VLA) Models — 3× in this paper · explained in 2 episodes
Adaptation — 2× in this paper · explained in 1 episode
In Conference on Robot Learning — 2× in this paper
International Conference on Robotics and Automation — 2× in this paper
Physical Constraints — 2× in this paper · explained in 7 episodes
Reinforcement Learning — 2× in this paper · explained in 4 episodes
Task Decomposition — 2× in this paper · explained in 2 episodes
Verification — 2× in this paper · explained in 2 episodes
Action Primitives — 1× in this paper · explained in 1 episode
Agentic Framework — 1× in this paper · explained in 6 episodes
Alignment — 1× in this paper · explained in 1 episode
Catastrophic Forgetting — 1× in this paper · explained in 5 episodes
Failure Modes — 1× in this paper · explained in 3 episodes
Generative AI — 1× in this paper · explained in 2 episodes
In International Conference on Learning Representations — 1× in this paper
International Conference on Machine Learning — 1× in this paper
Prompt Engineering — 1× in this paper · explained in 4 episodes
Scaffolding — 1× in this paper · explained in 3 episodes

Summary

Robot manipulation requires complex skills like visual understanding, physical reasoning, and planning, which have been enhanced by foundation models (FMs). This paper introduces VIA (Visual Interface Agent for robot control), a framework that recasts robot control as an agentic task where an off-the-shelf FM-powered agent drives a manipulator through a browser-based 3D interface using general computer use skills.

The gist: VIA is a framework that recasts robot control as an agentic task: an off-the-shelf FM-powered agent drives a manipulator through a browser-based 3D interface by taking screenshots, issuing intuitive commands, observing the outcome, and adjusting.

Core Concept and Motivation

The paper addresses the gap between powerful foundation models (FMs) and robot control policies. Current methods often involve converting FMs into vision-language-action (VLA) models through fine-tuning on robot data; however, VLAs are often orders of magnitude smaller than frontier FMs given the limited data and compute available for fine-tuning. VIA proposes a different paradigm: leveraging the growing ability of FMs to operate software through visual interfaces. The central question is whether this competence suffices to control a robot. VIA inherits the agent’s general reasoning, closed-loop error recovery, and ability to plan and re-plan from what it observes, without requiring robot-specific finetuning.

The VIA Framework Architecture

VIA recasts robot control as an agentic task where an agent operates a browser-based 3D robot-control UI. The interface is built using calibrated third-person RGB-D cameras to construct a 3D point cloud of the scene, which serves as the main workspace. The agent perceives the scene solely by taking screenshots, with no access to privileged state. Control is achieved through a small set of general tools called Model Context Protocol (MCP) tools, such as:

Observation tools: including screenshot, hover(u, v), and pose readers like gripper get pose.
Action tools: including gripper teleport via click(u, v), gripper drag(u1, v1, u2, v2, [constraint], [steps]), and execution tools like execute waypoint and end episode.

Agentic Loop and Tool Design

VIA operates through a closed-loop structure: the agent performs an observe-act loop by taking a screenshot, reasoning about what it sees, issuing an intuitive command via an MCP tool, observing the outcome (which returns an updated screenshot), and adjusting. The tools are designed around two principles: minimalism and agent ergonomics. For minimalism, each tool wraps a human operation with little or no extra abstraction, allowing agents to operate with the same freedom as a human. For ergonomics, they provide direct alternatives to continuous actions, such as using gripper rotate instead of continuous dragging. The system is guided by a short system prompt covering interface basics and guidance on task decomposition and planning, emphasizing closed-loop control over open-loop.

Evaluation and Results

VIA was evaluated with two popular agents, Claude Code (CC) and Codex, on a suite of six tabletop manipulation tasks. The results demonstrate that VIA solves diverse manipulation tasks zero-shot from a minimal prompt stating only the goal. With the strongest model (Fable 5 for CC), it achieved 96.7% success on three LIBERO-Goal tasks and 100% on a long-horizon rainbow assembly task. Performance improves with the scale and strength of the underlying model, suggesting that frontier agents already possess skills that transfer directly to robot control given the right interface. The paper also shows that performance can be significantly improved by equipping weaker models with detailed prompts containing example waypoint lists, demonstrating the potential for learning via reflection.

Implications and Future Directions

The findings suggest that robot control can become another economically valuable agentic task benefiting directly from modern models. VIA demonstrates that with the right interface and MCP tools, existing general agents can control a robot without adaptation. Limitations include reliance on the best frontier models due to the novel interface demanding high general capability, leading to slow inference. Future work includes extending VIA to real robots and developing a more sophisticated controller for precision tasks like T-block. The framework also supports Automatic Tool Improvement and Learning via Reflection, where the agent can suggest tool modifications or improve its own task prompts based on feedback. The paper concludes that robot control is poised to harvest general gains from foundation model scaling, as each new generation of general capability transfers to the robot for free, with no fine-tuning required.

Category Tools Summary

The MCP tools are categorized into Observation and Action tools. Observation tools like screenshot and hover(u, v) gather information about the environment.

Improvements for AI systems

Here are the specific improvements that can be made to AI systems, based on the VIA (Visual Interface Agent for Robot Control) framework:

Improve General-Purpose Foundation Models (FMs) for Robotics by using a Computer-Use Agent paradigm instead of direct Vision-Language-Action (VLA) fine-tuning.
Enable Zero-Shot, Domain-Agnostic Robot Control by leveraging the general reasoning and tool use capabilities of frontier models (like Claude Code or Codex) through a visual interface, requiring no robot-specific finetuning.
Develop Highly Scalable Robot Policies by scaling with the strength of frontier computer-use agents, allowing for rapid transfer of general skills to physical manipulation tasks.
Enhance Long-Horizon Planning and Execution in Robotics by recasting control as an agentic task (Observe-Act loop) where the agent inherits its existing planning and re-planning abilities from software operation agents.
Create Cost-Effective Robot Demonstration Pipelines by enabling the creation of high-quality, text/screenshot-based demonstrations for robotic policies using the agent's ability to generate structured waypoints (e.g., using detailed prompts).
Improve Agent Robustness and Self-Correction through Learning via Reflection by allowing agents to use textual feedback or multimodal reflection on their actions without requiring weight updates, leading to improved prompt engineering for task success.
Increase Tool Reliability and Ergonomics by designing a minimalist set of Model Context Protocol (MCP) tools that map complex robot operations (like 6-DoF manipulation) into intuitive, human-legible agent commands, addressing the difficulty agents have with continuous animation.
Develop Adaptive Tool Improvement Capabilities where the controlling agent can autonomously read tool documentation, test new tool functionality in the environment, and suggest or implement modifications to improve its own control toolkit.

The improved AI system (VIA) can perform:

Control physical robot manipulators for a diverse suite of tabletop manipulation tasks (pick-and-place, assembly, precise execution) with high success rates (up to 100% on specific tasks like the rainbow assembly).
Solve complex spatial reasoning and long-horizon planning problems zero-shot by interpreting a browser-based 3D interface rendered from RGB-D cameras.
Operate real robots without requiring any prior robot-specific fine-tuning, leveraging only the general perception, reasoning, and tool use skills of state-of-the-art foundation models.
Execute complex manipulation sequences by iteratively observing the outcome of actions (closed-loop control), diagnosing failures based on visual feedback from screenshots and point clouds, and dynamically re-planning a sequence of waypoints to achieve the final goal.
Generate high-fidelity, structured textual demonstrations (waypoint lists) for robot tasks, which can then be used to guide other agents or improve their own performance through reflection.

Abstract

Robot manipulation is a complex task that requires visual perception, physical reasoning, planning, and closed-loop control. General-purpose foundation models (FMs) have grown remarkably capable of some of these, especially perception and reasoning. Inspired by the growing ability of FM-powered agents to operate software through visual interfaces, we ask whether that same competence suffices to control a robot directly given the right interface. We present VIA (Visual Interface Agent), a framework that recasts robot control as an agentic task where an off-the-shelf agent drives a manipulator directly through an interface. The interface is a virtual workspace of interactable visual components where the agent can probe pixels of interest with mouse-like tools and command a virtual end-effector to set new target poses with a small set of general tools. We show that VIA enables agents of varying capabilities to perform diverse tabletop manipulation tasks in simulation. We also use VIA to drive an off-the-shelf mobile manipulation robot in the real world to complete tasks that require both navigation and manipulation. Both settings are zero-shot, requiring no robot training or additional action primitives. Performance scales well with the size and strength of the underlying FMs. These results suggest that frontier agents already possess skills that transfer directly to robot control given the right interface: your coding or computer-use agent is, in a sense, secretly a robot-control agent.