CTBench: Evaluating Troubleshooting Capabilities of AI Agents in Realistic Telecom Network Operations

CTBench: Evaluating Troubleshooting Capabilities of AI Agents in Realistic Telecom Network Operations

Summary

This paper introduces CTBench, a public benchmark designed to evaluate the troubleshooting capabilities of AI agents in realistic telecom network operations and maintenance (NetO&M). The benchmark focuses on two operational task families: root cause analysis (RCA) and path restoration. The authors argue that existing telecom benchmarks fail to model real network characteristics, such as partial observability, multi-vendor environments, and heterogeneous devices, and often evaluate only final answers rather than the diagnostic process.

The benchmark contains 234 expert-curated tasks: 126 for RCA and 108 for path restoration. Each task is constructed by 15 senior telecom experts with an average of 20 years of experience and validated by an independent expert who solves tasks without access to ground truths. A key feature is the annotation of each task with expert-validated golden evidence steps, which capture the observations and diagnostic actions required to solve the task. Tasks are also annotated with rich metadata, including evidence observability, vendor/device heterogeneity, protocol complexity, root-cause count, restored path count, fault-propagation chains, and golden solution length.

For RCA tasks, the target answer is a set of normalized root-cause triples: (affected node, affected object, normalized root-cause label). For path-restoration tasks, the target answer is an ordered sequence of path elements reconstructing the forwarding path. The evaluation uses expert-grounded metrics that assess not only final answers but also diagnostic evidence. Specifically, CTBench defines three complementary capability metrics for each task type: localization (identifying where the problem occurs), identification/restoration (identifying the correct root-cause label or reconstructing the path), and evidence (whether the agent collects relevant key diagnostic evidence compared to golden steps, measured via F1 score).

The paper evaluates five representative agent-model combinations: Codex+GPT-5.5, ClaudeCode+Qwen3.7-Plus, HermesAgent+DeepSeek-V4-Pro, HermesAgent+Qwen3.7-Max, and HermesAgent+TelecomGPT-R1. Results show that state-of-the-art agents perform very well at identifying endpoints in path-restoration tasks but underperform in root cause analysis. Codex+GPT-5.5 achieves the strongest overall results, with 47.62% RCA accuracy and 87.96% path restoration accuracy. However, even this agent reaches only 47.84% path-restoration evidence and 15.80% RCA evidence, showing that producing correct final answers does not imply producing the evidence expected from a telecom troubleshooting engineer.

The paper reports several key findings. First, agents are better at RCA identification than RCA localization (e.g., 66.83% vs. 52.90% for Codex+GPT-5.5), indicating they sometimes infer the right fault type but fail to identify the precise faulty node or object. Second, performance varies substantially across fault categories: C2 (Security/NAT/access) faults are the most reliably diagnosed, while C1 (Interface/Link) faults are uniformly hard, with accuracy between 0.00% and 10.71%. Third, partial observability degrades performance: for ClaudeCode+Qwen3.7-Plus, RCA accuracy drops from 20.56% on fully observable tasks to 15.79% on partially observable tasks, and localization drops from 37.02% to 17.14%. Fourth, network heterogeneity significantly impacts path restoration: for ClaudeCode+Qwen3.7-Plus, path accuracy decreases from 29.31% in low heterogeneity settings to 4.00% in high heterogeneity settings.

The paper also analyzes cost and efficiency, reporting interaction rounds, latency, and token consumption. Results show that higher computational cost does not necessarily translate into stronger diagnostic capability. For instance, ClaudeCode+Qwen3.7-Plus consumes substantially more rounds and tokens than Codex+GPT-5.5 but obtains lower overall capability scores. The paper notes that human experts achieved 92.6% and 56.4% accuracy on path-restoration and RCA tasks respectively, taking 40-60 minutes per task.

The supplementary material provides additional analysis, including a trajectory-level RCA failure analysis. The main failure pattern is not failed tool use but rather failures in mapping evidence to final answers. The failure taxonomy includes: causal coverage deficits (missing independent root causes), causal minimality violations (including redundant causes), mechanism discrimination errors (confusing redundancy, routing, policy, or forwarding-layer faults), fault-object grounding errors (wrong device, interface, or role), and answer realization failures (unparseable outputs). The paper concludes that current agents remain far from reliable telecom troubleshooting operators, especially in evidence-grounded RCA and partially observable settings, and hopes CTBench will support future research on trustworthy, domain-grounded, and operationally safe AI agents for communication networks.

Improvements to AI Systems Based on CTBench

1. Evidence-Grounded Diagnostic Training: Train agents to explicitly generate and justify intermediate diagnostic evidence (e.g., check interface X for CRC errors) before producing final root-cause labels. Use the golden evidence steps in CTBench as supervision for a two-stage model: first predict evidence steps, then predict root causes conditioned on those steps. This addresses the observed gap where agents achieve high identification accuracy but low evidence F1 (e.g., 15.80% for RCA evidence vs. 47.62% accuracy).

2. Localization-First Reasoning Module: Add a dedicated submodule that forces the agent to output a candidate faulty node and object before classifying the fault type. This mitigates the systematic weakness where agents infer correct fault categories (66.83%) but fail on precise localization (52.90%). Implement a hard constraint: the final answer must include a valid (node, object) pair, and train with a loss that weights localization errors more heavily.

3. Partial-Observability-Aware Action Selection: Modify the agent's planning policy to explicitly track which network elements are unobservable and prioritize actions that reduce uncertainty (e.g., querying neighbor devices, requesting vendor-specific logs). Use CTBench's evidence observability annotations to train a reinforcement learning reward that penalizes actions on fully observable tasks less than on partially observable ones, directly addressing the observed performance drop (e.g., RCA accuracy from 20.56% to 15.79%).

4. Heterogeneity-Adaptive Path Reconstruction: Introduce a context encoder that ingests device vendor, protocol version, and interface type metadata before generating path restoration sequences. Train the agent to switch between vendor-specific forwarding rules (e.g., Cisco vs. Juniper) using CTBench's heterogeneity labels. This targets the severe degradation seen in high-heterogeneity settings (path accuracy from 29.31% to 4.00%).

5. Causal Coverage and Minimality Checker: Add a post-processing verification layer that (a) checks whether all independent root causes are present (causal coverage) and (b) removes redundant causes (causal minimality). Use CTBench's multi-root-cause annotations to train a classifier that flags incomplete or over-complete answer sets, reducing the dominant failure modes of causal coverage deficits and minimality violations.

6. Mechanism Discrimination Training: Create a contrastive learning objective that forces the agent to distinguish between similar fault mechanisms (e.g., redundancy vs. routing vs. policy vs. forwarding-layer faults). Use CTBench's fault-propagation chains to generate hard negative examples where the same evidence leads to different root causes, improving the agent's ability to discriminate between mechanisms.

7. Fault-Object Grounding Refinement: Implement a named-entity recognition (NER) module specialized for telecom objects (devices, interfaces, roles) that is trained on CTBench's normalized root-cause triples. This module pre-processes raw observations to extract candidate objects, then feeds them to the reasoning engine, reducing grounding errors where agents identify the wrong device or interface.

8. Answer-Realization Robustness: Add a structured output decoder with grammar constraints that forces the agent to produce parseable answers in the exact format of (node, object, label) triples or ordered path elements. Use CTBench's answer realization failure annotations to train a rejection sampler that retries or rephrases outputs when parsing fails, eliminating unparseable outputs.

9. Cost-Aware Capability Optimization: Train a meta-controller that dynamically allocates interaction rounds and token budgets based on task difficulty (e.g., using CTBench's metadata like fault-propagation chain length and golden solution length). This prevents wasteful computation on simple tasks while ensuring sufficient exploration on complex ones, addressing the finding that higher cost does not correlate with better capability.

10. Human-Expert Trajectory Imitation: Fine-tune agents on the validated expert solution trajectories (from the independent expert who solved tasks without ground truth) using behavioral cloning. This teaches efficient diagnostic sequences and evidence collection patterns that match human experts, who achieve 92.6% path restoration and 56.4% RCA accuracy, providing a strong upper bound for agent performance.

What the Improved AI System Can Do:

• Diagnose telecom faults with 2–3× higher evidence alignment, matching expert-grounded golden steps rather than just producing correct final labels.

• Localize faults precisely (correct node and object) even when the fault type is ambiguous, reducing misattribution errors.

• Maintain performance under partial observability by actively seeking missing information instead of guessing, with less than 10% accuracy drop in partially observable scenarios.

• Reconstruct forwarding paths across multi-vendor, heterogeneous networks with minimal degradation, adapting to vendor-specific behaviors on the fly.

• Generate complete and minimal root-cause sets for multi-fault scenarios, avoiding both missing causes and redundant inclusions.

• Distinguish between subtle fault mechanisms (e.g., policy vs. routing) that currently confuse state-of-the-art agents.

• Produce structured, parseable outputs 100% of the time, eliminating answer realization failures.

• Operate cost-efficiently, using fewer tokens and rounds on simple tasks while allocating more resources to complex ones, achieving higher capability per unit cost.

• Mimic expert troubleshooting workflows, including evidence collection order and diagnostic reasoning, making the agent's process auditable and trustworthy for telecom operators.

Agents are increasingly considered for automating network operations and maintenance, where engineers must diagnose network faults, optimize configurations to enhance services, and reduce operational costs while acting under strict constraints. However, existing evaluations fail to accurately model real network characteristics or assess agents under partially observable telecom environments with diverse vendors, devices, protocols, and interfaces. In this paper, we introduce CTBench, a public benchmark for assessing whether an agent behaves like a competent telecom troubleshooting engineer. CTBench focuses on root cause analysis and path restoration. Each task is constructed by experts and annotated with rich task metadata, including golden evidence steps. CTBench uses expert-grounded metrics that evaluate both final answers and the diagnostic evidence. Experiments with representative harness-model combinations show that state-of-the-art agents perform very well at identifying endpoints in path-restoration tasks but, more generally, underperform in root cause analysis. In particular, agents struggle with interface state, link-layer, service-management, and other operational faults. Most importantly, even when agents produce plausible or correct final answers, they often fail to provide the evidence-grounded diagnoses required in operational practice. Our results further show that path restoration is generally more resource expensive, yet larger resource usage does not necessarily translate into better diagnosis.