Probe to Act: Elevating Browser-Use Agent via Active Visual Probing

arXiv:2609.33646 · cs.AI, cs.CL, cs.CV · Submitted 2026-09-27 · Read on arXiv

cs.AI, cs.CL, cs.CV

Submitted: 2026-09-27

Updated: 2026-09-27

Code: https://github.com/OSU-NLP-Group/Online-Mind2Web

Terminology

• Alignment — 15× in this paper · explained in 1 episode
• Grounding — 13× in this paper · explained in 3 episodes
• Benchmark — 11× in this paper · explained in 2 episodes
• Action Space — 9× in this paper · explained in 1 episode
• Verification — 8× in this paper · explained in 2 episodes
• Ablation — 7× in this paper · explained in 1 episode
• Accuracy — 6× in this paper · explained in 1 episode
• Robustness — 5× in this paper · explained in 5 episodes
• Distillation — 4× in this paper · explained in 2 episodes
• Reinforcement Learning — 4× in this paper · explained in 4 episodes
• Visual Grounding — 4× in this paper · explained in 2 episodes
• Adam — 3× in this paper · explained in 2 episodes
• Trace — 3× in this paper · explained in 2 episodes
• Ablation study — 2× in this paper · explained in 1 episode
• Active Perception — 2× in this paper · explained in 2 episodes
• Adaptation — 2× in this paper · explained in 1 episode
• Core — 2× in this paper · explained in 2 episodes
• Fine-Tuning — 2× in this paper · explained in 5 episodes
• Pruning — 2× in this paper · explained in 4 episodes
• AI Agents — 1× in this paper · explained in 2 episodes
• AdamW Optimizer — 1× in this paper · explained in 1 episode
• Adaptive Probing — 1× in this paper · explained in 1 episode
• Agentic System — 1× in this paper · explained in 2 episodes
• Embeddings — 1× in this paper · explained in 2 episodes
• Fidelity — 1× in this paper · explained in 2 episodes
• Human-in-the-Loop — 1× in this paper · explained in 3 episodes
• LLM judge — 1× in this paper · explained in 2 episodes
• Reasoning Traces — 1× in this paper · explained in 3 episodes
• Transformer — 1× in this paper · explained in 2 episodes
• Verifier — 1× in this paper · explained in 2 episodes
• Vision Transformer (ViT) — 1× in this paper · explained in 2 episodes
• Vision-language models — 1× in this paper · explained in 2 episodes
• World Model — 1× in this paper · explained in 7 episodes

Sources

• Agent-E: From Autonomous Web Navigation to Foundational Design Principles in Agentic Systems
• Let's Think in Two Steps: Mitigating Agreement Bias in MLLMs with Self-Grounded Verification
• Qwen3-VL Technical Report
• GUI-Eyes: Tool-Augmented Perception for Visual Grounding in GUI Agents
• Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling
• Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
• Is Your LLM Secretly a World Model of the Internet? Model-Based Planning for Web Agents
• UI-Venus Technical Report: Building High-performance UI Agents with RFT
• GPT-4o System Card
• Zoom in, Click out: Unlocking and Evaluating the Potential of Zooming for GUI Grounding
• WebCoach: Self-Evolving Web Agents with Cross-Session Memory Guidance
• ScaleCUA: Scaling Open-Source Computer Use Agents with Cross-Platform Data
• FocusUI: Efficient UI Grounding via Position-Preserving Visual Token Selection
• WALT: Web Agents that Learn Tools
• WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learning
• UI-TARS: Pioneering Automated GUI Interaction with Native Agents
• Compress to Focus: Efficient Coordinate Compression for Policy Optimization in Multi-Turn GUI Agents
• Scaling Long-Horizon LLM Agent via Context-Folding
• Auto-scaling Continuous Memory for GUI Agent
• Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V

Related papers

• MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
• Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
• The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
• MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
• Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems
• VSAL: A Vision Solver with Adaptive Layouts for Graph Property Detection