Conversational versus Dashboard Explainable AI for UAV Intrusion Detection: An Empirical Study of Operator Trust and Reliance

This paper proposes a Conversational XAI interface powered by Large Language Models (LLM) to address interpretability challenges in machine learning-based Intrusion Detection Systems (IDS) for Unmanned Aerial Vehicle (UAV) networks. The authors note that the 'black-box' nature of these models, combined with the high dimensionality of multimodal cyber-physical data, poses significant interpretability challenges, and that Static visualization dashboards may struggle to present complex relationships among multimodal cyber-physical features in a form that is easy for operators to inspect and interpret.

The proposed framework integrates four XAI methods: Global Explanation (PDP) to address how feature values affect attack probabilities, Local Feature Attribution (SHAP) to answer why a specific alert was classified as malicious or benign, Contrastive Counterfactual Explanation (MACE) to answer how would this alert need to differ to be classified differently?, and Interactive Simulation (What-If) to support hypothesis testing. The conversational interface is powered by llama3.1:70b-instruct-q4 K M and operates in three steps: Intent Recognition, Tool Execution, and Response Synthesis. A traditional XAI Dashboard covering the same underlying methods serves as a comparative baseline.

In a controlled experiment with 57 participants evenly distributed across three conditions (no-explanation Control, Dashboard XAI, and Conversational XAI), each operator audited 20 alerts where the AI was correct on 14 alerts and wrong on six. The results show that the Conversational interface was perceived as more useful than the dashboard, potentially because it helped participants access and synthesize relevant information more easily. Specifically, In Explanation Utility, the Conversational interface scored 4.10 against the Dashboard’s 3.90, and Operator Trust was descriptively and only marginally higher for the Conversational interface (4.23 versus 4.20). Operator Understanding was also slightly higher in the Conversational condition (3.78 versus 3.68).

However, the behavioral results revealed a potential risk: Relative to the no-explanation Control, both interfaces sharply raised reliance, with the agreement fraction climbing from 0.74 to 0.86 (Dashboard) and 0.89 (Conversational). Yet appropriate self-reliance (RSR) decreased substantially from 0.57 to 0.29 and 0.11, reaching its lowest value under the Conversational interface. Kruskal–Wallis tests indicated statistically significant differences across conditions for agreement fraction (H = 33.66, p <.001), switch fraction (H = 19.14, p <.001), RAIR (H = 11.01, p =.004), RSR (H = 38.26, p <.001), and decision accuracy (H = 9.09, p =.011). Critically, the Conversational interface, despite eliciting the highest agreement and the lowest self-reliance, did not achieve higher accuracy than the Dashboard; descriptively, its accuracy was slightly lower (0.71 versus 0.74).

The authors interpret these findings as evidence of a trade-off: One possible interpretation is that the natural-language responses made the AI advice easier to accept, which may have reduced participants’ tendency to verify the underlying evidence when the IDS was incorrect. They note that the Llama-powered agent presents anomalies as coherent narratives, which may make uncertainty less salient to users unless uncertainty cues are explicitly shown. The paper concludes that smooth interaction may be insufficient for high-stakes UAV defense; future designs may benefit from Cognitive Forcing Functions that encourage active verification, transforming the operator from a passive consumer into an active auditor.

Improvements to AI Systems:

1. Integrate Cognitive Forcing Functions into Conversational XAI: Modify the LLM-powered interface to periodically require the operator to articulate their own assessment of an alert before revealing the AI’s explanation. For example, after presenting a SHAP-based local explanation, the system prompts: Based on the features shown, do you predict this alert is malicious or benign? Explain your reasoning in one sentence. This forces active verification and reduces passive over-reliance.

2. Inject Explicit Uncertainty Cues into Natural Language Responses: Enhance the response synthesis step so that every explanation includes a quantified confidence interval and a verbal hedge when the model’s prediction is uncertain. For instance, instead of This alert is malicious because feature X is high, the system would say: This alert is likely malicious (confidence: 72%, margin of error ±5%). The key feature X is elevated, but note that feature Y is borderline, which increases uncertainty. This makes the probabilistic nature of the IDS salient, countering the narrative coherence of LLM outputs.

3. Implement Adaptive Explanation Depth Based on Operator Verification Behavior: Track the operator’s agreement/switch decisions and self-reliance score in real time. If the system detects a pattern of high agreement with low verification (e.g., RSR dropping below a threshold), it automatically escalates the interaction mode—switching from conversational summaries to step-by-step evidence breakdowns, requiring the operator to click through each feature’s contribution before accepting the AI’s verdict.

4. Add a Devil’s Advocate Mode for Counterfactual Challenges: Extend the MACE counterfactual tool to proactively generate alternative explanations that contradict the AI’s current classification. For each alert, the system offers: Here is a plausible counterfactual where this alert would be classified as benign. Do you see why the model chose malicious instead? What evidence would change your mind? This encourages operators to test the model’s reasoning rather than passively accept it.

5. Calibrate Trust via Post-Hoc Accuracy Feedback: After each batch of 20 alerts, the system provides a personalized trust calibration report, showing the operator their agreement fraction, self-reliance, and accuracy compared to the AI’s true performance. It then highlights specific instances where the operator over-relied on incorrect AI advice, with a brief narrative: On alert #7, you agreed with the AI, but it was wrong. Here’s what the evidence actually showed. This trains operators to better discriminate between reliable and unreliable AI outputs over time.

What the Improved AI System Can Do:

• Actively audit high-stakes alerts by forcing operators to reason before accepting AI explanations, thereby increasing self-reliance without sacrificing the conversational interface’s usability.

• Communicate uncertainty transparently in natural language, preventing the LLM’s coherent narratives from masking model confidence, which directly addresses the observed drop in RSR (0.57 → 0.11).

• Dynamically adjust its interaction style based on operator behavior, intervening when over-reliance is detected, and thus improving decision accuracy beyond the dashboard baseline (0.74) rather than falling to 0.71.

• Provide counterfactual challenges that help operators understand the model’s decision boundaries, reducing blind agreement (from 0.89 back toward a more calibrated level) while maintaining high explanation utility.

• Foster long-term trust calibration by giving operators concrete feedback on their own performance, turning the system from a passive advisor into an active training tool for UAV defense operators.

Machine learning-based Intrusion Detection Systems (IDS) have demonstrated superior performance in securing Unmanned Aerial Vehicle (UAV) networks. However, the 'black-box' nature of these models, combined with the high dimensionality of multimodal cyber-physical data, poses significant interpretability challenges. Static visualization dashboards may struggle to present complex relationships among multimodal cyber-physical features in a form that is easy for operators to inspect and interpret. To address this, we propose a Conversational XAI interface powered by Large Language Models (LLM) to facilitate on-demand investigation. In a controlled experiment with participants, we systematically evaluated the impact of this conversational interface versus a traditional XAI Dashboard on operator understanding, trust, and reliance during post-incident auditing tasks. Our results suggest that the conversational interface was perceived as more useful than the dashboard, potentially because it helped participants access and synthesize relevant information more easily. However, this benefit was accompanied by a lower level of appropriate self-reliance, indicating a potential risk of over-reliance. One possible interpretation is that the natural-language responses made the AI advice easier to accept, which may have reduced participants' tendency to verify the underlying evidence when the IDS was incorrect. These findings point to a potential trade-off in human-AI collaboration for UAV intrusion auditing: interaction mechanisms that improve perceived usability may also increase the risk of inappropriate reliance. We conclude by discussing design implications for future XAI systems that balance seamless interaction with cognitive forcing functions to foster appropriate reliance.