SwarmBench: Can Large Language Models Act as Agent Swarm Orchestrators?

arXiv:2608.30661 · cs.CL · Submitted 2026-08-31 · Read on arXiv

cs.CL

Submitted: 2026-08-31

Updated: 2026-08-31

Code: https://github.com/ying1973/SwarmBench

Terminology

• Aggregation — 41× in this paper · explained in 1 episode
• Accuracy — 27× in this paper · explained in 1 episode
• Benchmark — 24× in this paper · explained in 2 episodes
• Task Decomposition — 13× in this paper · explained in 2 episodes
• Multi-Agent System — 10× in this paper · explained in 6 episodes
• Batch Download — 8× in this paper
• Long-Text Generation — 7× in this paper
• Trace — 7× in this paper · explained in 2 episodes
• Wide Search — 7× in this paper
• Root Cause Analysis — 6× in this paper
• Multi-Text Understanding — 5× in this paper
• Benchmarking — 4× in this paper · explained in 4 episodes
• Failure Modes — 4× in this paper · explained in 3 episodes
• Ablation — 3× in this paper · explained in 1 episode
• Consistency — 3× in this paper · explained in 3 episodes
• Core — 3× in this paper · explained in 2 episodes
• Task Planning — 3× in this paper · explained in 2 episodes
• Ablation study — 2× in this paper · explained in 1 episode
• LLM judge — 2× in this paper · explained in 2 episodes
• Action Space — 1× in this paper · explained in 1 episode
• Agentic Evaluation — 1× in this paper · explained in 1 episode
• Agentic System — 1× in this paper · explained in 2 episodes
• Continual Learning — 1× in this paper · explained in 2 episodes
• Cross-Model Transfer — 1× in this paper · explained in 2 episodes
• Paper Review — 1× in this paper · explained in 3 episodes
• Reinforcement Learning — 1× in this paper · explained in 4 episodes
• Transferability — 1× in this paper · explained in 4 episodes
• Treasure Hunt Game — 1× in this paper
• Verification — 1× in this paper · explained in 2 episodes

Sources

• EvoSkill: Automated Skill Discovery for Multi-Agent Systems
• $\tau^2$-Bench: Evaluating Conversational Agents in a Dual-Control Environment
• MATA: A Trainable Hierarchical Automaton System for Multi-Agent Visual Reasoning
• MARG: Multi-Agent Review Generation for Scientific Papers
• Large Language Model based Multi-Agents: A Survey of Progress and Challenges
• Empowering GUI Agents via Autonomous Experience Exploration and Hindsight Experience Utilization for Task Planning
• LLM Multi-Agent Systems: Challenges and Open Problems
• Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces
• Pushing the Limits of LLM Tool Calling via Experiential Knowledge Integration and Activation
• XSkill: Continual Learning from Experience and Skills in Multimodal Agents
• A Comprehensive Survey on Multi-Agent Cooperative Decision-Making: Scenarios, Approaches, Challenges and Perspectives
• MAS-Orchestra: Understanding and Improving Multi-Agent Reasoning Through Holistic Orchestration and Controlled Benchmarks
• Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Application
• In-the-Flow Agentic System Optimization for Effective Planning and Tool Use
• Beyond Individual Intelligence: Surveying Collaboration, Failure Attribution, and Self-Evolution in LLM-based Multi-Agent Systems
• AOrchestra: Automating Sub-Agent Creation for Agentic Orchestration
• Benchmarking LLMs' Swarm intelligence
• From Context to Skills: Can Language Models Learn from Context Skillfully?
• Kimi K2.5: Visual Agentic Intelligence
• Agent Workflow Memory

Related papers

• Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
• Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
• Subliminal Steering: Stronger Encoding of Hidden Signals
• MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• Untangling the Mechanisms of Misleading Context in Medical Question Answering