Do Multimodal Agents Really Benefit from Tool Use? A Systematic Study of Capability Gains

arXiv:2606.02357 · cs.CV, cs.AI · Submitted 2026-06-01 · Read on arXiv

cs.CV, cs.AI

Submitted: 2026-06-01

Updated: 2026-08-27

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Benchmark — 82× in this paper · explained in 2 episodes
• Ablation — 10× in this paper · explained in 1 episode
• Accuracy — 8× in this paper · explained in 1 episode
• Trace — 7× in this paper · explained in 2 episodes
• Reinforcement Learning — 5× in this paper · explained in 4 episodes
• Information Gain — 4× in this paper · explained in 2 episodes
• Adam — 3× in this paper · explained in 2 episodes
• Multimodal Reasoning — 3× in this paper · explained in 5 episodes
• Benchmarking — 2× in this paper · explained in 3 episodes
• Reasoning Traces — 2× in this paper · explained in 3 episodes
• Robustness — 2× in this paper · explained in 5 episodes
• Core — 1× in this paper · explained in 2 episodes
• Distribution Shift — 1× in this paper · explained in 5 episodes
• Fine-Tuning — 1× in this paper · explained in 5 episodes
• Foundation Model — 1× in this paper · explained in 3 episodes
• Foundation Models — 1× in this paper · explained in 7 episodes
• GRPO — 1× in this paper · explained in 6 episodes
• Training-Free Framework — 1× in this paper · explained in 2 episodes

Sources

• MATRIX: Multimodal Agent Tuning for Robust Tool-Use Reasoning
• Qwen2.5-VL Technical Report
• VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Models
• GeoBrowse: A Geolocation Benchmark for Agentic Tool Use with Expert-Annotated Reasoning Traces
• VisualToolAgent (VisTA): A Reinforcement Learning Framework for Visual Tool Selection
• Qwen3-VL Technical Report
• Faithful-First Reasoning, Planning, and Acting for Multimodal LLMs
• TIR-Bench: A Comprehensive Benchmark for Agentic Thinking-with-Images Reasoning
• DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
• AgentVista: Evaluating Multimodal Agents in Ultra-Challenging Realistic Visual Scenarios
• MM-Eureka: Exploring the Frontiers of Multimodal Reasoning with Rule-based Reinforcement Learning
• GPT-4o System Card
• MuSEAgent: A Multimodal Reasoning Agent with Stateful Experiences
• DeepSeek-OCR: Contexts Optical Compression
• Qwen-Image Technical Report
• LogicVista: Multimodal LLM Logical Reasoning Benchmark in Visual Contexts
• Qwen3 Technical Report
• Walk the Talk: Bridging the Reasoning-Action Gap for Thinking with Images via Multimodal Agentic Policy Optimization
• Are Tools All We Need? Unveiling the Tool-Use Tax in LLM Agents
• Adaptive Chain-of-Focus Reasoning via Dynamic Visual Search and Zooming for Efficient VLMs

Related papers

• Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
• AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
• Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
• MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
• TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
• A Survey on Efficient Vision-Language-Action Models