ARGOS: Who, Where, and When in Agentic Multi-Camera Person Search

ARGOS is a novel benchmark and framework designed to address the limitations of traditional person re-identification by reformulating multi-camera person search as an interactive reasoning problem. This system moves beyond relying solely on visual appearance descriptions, instead equipping an agent with the ability to actively plan questions or leverage spatial and temporal cues—information that witnesses routinely provide in real-world scenarios. The benchmark is significant because it evaluates a complex, multi-turn dialogue process grounded in physical constraints, providing a rigorous testbed for multimodal spatial intelligence.

The ARGOS Framework Architecture

The core of the system is the ARGOS Agent, which operates as a tool-augmented LLM agent processing each turn through four distinct modules: Analyst, Planner, Interviewer, and Interpreter. This architecture forms an observe-think-act loop over the evaluation environment. The agent leverages eight specialized tools—such as gallery queries (T1–T2) and temporal feasibility checking via the STTG (T5)—to execute actions based on strategic decisions made by the Planner.

The Spatio-Temporal Topology Graph

The structural backbone of ARGOS is a Spatio-Temporal Topology Graph (STTG), which serves as a directed weighted graph encoding camera connectivity and empirically validated transition times. The STTG captures how people move between cameras, classifying transitions into three types: OVERLAP (shared field of view), SOFT ADJ (physically adjacent), or TRAVEL. This graph is used both to derive ground-truth tasks and to provide the agent with a grounding tool for spatio-temporal reasoning, ensuring that candidate movements are physically feasible.

The Multi-Track Benchmark Design

ARGOS comprises 2,691 tasks across 14 real-world scenarios, organized into three progressive tracks:

• Track 1 (Who): Tests semantic perception by extracting attributes from a completed multi-turn dialogue.

• Track 2 (Where): Tests spatial reasoning, requiring the agent to resolve specific sub-areas through spatial questions and attribute queries.

• Track 3 (When): Tests temporal reasoning, where the agent eliminates candidates whose transitions violate STTG constraints across different sightings.

The benchmark utilizes a deterministic witness simulator that returns ground-truth values for three observable attributes (gender, upper clothing color, and lower clothing color), while responding with uncertainty for other attributes to ensure reproducibility.

Key Contributions and Results

The authors define interactive multi-camera person search as a task at the intersection of multi-turn witness interaction and spatio-temporal reasoning over camera networks. They constructed the ARGOS benchmark using Turn-Weighted Success (TWS) as the primary metric, jointly measuring correctness and turn efficiency. The results show that direct LLM inference is fundamentally insufficient; tool-augmented interaction substantially outperforms baseline methods. Ablation studies confirm that removing domain-specific tools is highly detrimental, with accuracy dropping by up to 49.6 percentage points in certain tracks.

As a diligent AI researcher, I have analyzed this paper not merely as a benchmark, but as a blueprint for an entirely new class of Agentic Spatial-Temporal Reasoning Systems. The fundamental flaw in current search systems—the assumption that static visual data is sufficient—is the critical vulnerability we must exploit.

Here are the specific improvements to AI systems, derived from the principles of ARGOS, and what the improved system can do:

Improvement: The core search process must be redesigned as a multi-turn, tool-augmented dialogue loop rather than a single query. We replace the standard search to return result paradigm with the ARGOS Agent Architecture:

• Analyst Module: Before querying, this module must calculate the Information Gain (IG) of each potential question against current candidate sets. It determines which attribute has the highest discriminative power, ensuring queries are optimally designed to reduce ambiguity.

• Planner Module: This module decides if the next action is a semantic query (Attribute), a spatial query (Zone/Sub-area), or a temporal check (STTG Feasibility). It must weigh these options based on which tool provides the highest expected reduction in candidate pool size.

• Interviewer & Interpreter Modules: These modules handle the real-world interaction, parse ambiguous natural language responses, and apply filtering logic, ensuring that errors are not compounded by misinterpretation.

What the Improved System Can Do: It can solve complex cases where a single search query would fail. Instead of simply reporting no match, it will actively engage with a witness/sensor data to narrow down the target based on dynamic, real-time evidence gathered through its planning and questioning process.

Improvement: We must move beyond simple bounding box detection and integrate a Spatio-Temporal Topology Graph (STTG) as the agent's grounding tool. The STTG must encode not just camera connectivity, but also empirically validated transition time statistics (t, t med, t).

• Mandatory Temporal Feasibility Check: For any two observed sightings of a target, the agent must invoke the STTG to calculate if the travel time is physically possible.

• System Constraint Enforcement: The system will automatically eliminate candidates whose trajectories violate these empirically measured constraints (e.g., eliminating a candidate who was seen in Camera A and then B, but travel between 7.6s and 20.7s is physically impossible).

Improvement: The Planner must use the Greedy Information Gain (IG) algorithm for query selection, rather than heuristic or random selection.

IG(a) = H value(a C t) times (1 - alpha times p uncertain(a))

This mandates that the agent asks the question whose answer will most effectively partition the current set of candidates (C t), weighted by how often that attribute is uncertain.

Improvement: We must develop a Robust Natural Language (NL) Parsing Layer capable of handling vague or ambiguous witness responses, moving beyond simple keyword matching.

• Semantic Normalization: The parser must normalize varied natural language phrases (a black jacket, the dark blue top) into canonical attribute values (e.g., color: Blue) and recognize hedging phrases (Pretty sure it was...) as potential uncertainty flags.

• Contextual Interpretation: The system must be designed to understand that if an attribute is non-observable (e.g., leg visibility vs upper color), the response should be flagged as 'Uncertain' rather than being ignored, allowing the agent to strategically pivot its next query.

Existing person search methods assume access to complete visual queries or exhaustive tracking, yet real-world witness accounts are vague, partial, and spread across cameras and time. We introduce ARGOS (Agentic Retrieval with Grounded Observational Search), a benchmark and agent framework that recasts multi-camera person search from one-shot retrieval on a complete query into interactive reasoning from partial clues. To our knowledge, ARGOS is the first interactive benchmark to couple witness dialogue with camera-network topology, requiring an agent to plan, question, and eliminate under information asymmetry. An ARGOS agent receives a vague witness statement and must decide what to ask, when to invoke spatial or temporal tools, and how to interpret ambiguous natural-language responses, all within a limited turn budget. To ground reasoning in physical constraints, the agent accesses a Spatio-Temporal Topology Graph (STTG) encoding camera connectivity and empirically validated transition times. The benchmark comprises 2, 691 tasks across 14 real-world scenarios in three progressive tracks: semantic perception (Who, 989 tasks), spatial reasoning (Where, 550 tasks), and temporal reasoning (When, 1, 152 tasks). We propose Turn-Weighted Success (TWS) as the primary metric, jointly measuring correctness and turn efficiency. Experiments with four LLM backbones show the benchmark is far from solved: the best agent achieves TWS of 0.383 (Track 2) and 0.590 (Track 3). Ablations confirm each component is essential: removing domain-specific tools drops Top-1 accuracy by up to 49.6 percentage points, and removing strategic reasoning halves TWS while barely affecting Top-1.