Critical or Compliant? The Double-Edged Sword of Reasoning in Chain-of-Thought Explanations

Explanations, particularly Chain-of-Thought (CoT) reasoning in vision language models, present a double-edged sword because they can simultaneously clarify complex judgments and foster dangerous blind trust. This study systematically investigates how users perceive reasoning fidelity when explanations are intentionally flawed or stylistically manipulated, revealing that delivery tones can override the actual correctness of the reasoning chain.

The Gist

Users often equate trust with outcome agreement, sustaining reliance even when reasoning is flawed, and the confident tone suppresses error detection while maintaining reliance, showing that delivery styles can override correctness.

Experimental Design and Measures

The researchers developed an experimental framework to measure trust calibration in reasoning chains by systematically perturbing reasoning correctness and manipulating confidence tones. The study utilized a 2x4 between-subjects design involving 800 experimental stimuli drawn from the MORALISE benchmark, which consists of image-text pairs annotated with a binary moral ground-truth label. The manipulation involved orthogonally varying two factors: reasoning correctness (Clean vs. Omission, Contradiction, Hallucination) and confidence tone (Confident, Hedged, Neutral). To capture user response comprehensively, three complementary measures of trust were employed:

1. Error Detection: A binary question asking if the reasoning contains a factual or logical error.

2. Agreement with Judgment: A 7-pt Likert scale measuring how much participants agree with the model’s final judgment.

3. Self-Reported Trust: A 7-pt Likert scale assessing willingness to delegate to the AI system without human review.

Key Findings on Reasoning Correctness and Reliance

The analysis revealed that users’ agreement with the model’s output is by far the most dominant factor influencing trust, even when the reasoning is flawed. Specifically, Table 2 showed strong correlations between agreement and trust (r= +0.82), and both were inversely related to error detection. For instance, when examining INCORRECT cases, higher agreement and trust coincide with lower error detection. Furthermore, Omissions were identified as being the most insidious, as they are shown to be the most susceptible to confidence-driven overtrust, often going unnoticed in CORRECT cases while still sustaining high agreement and trust. Contradictions and Hallucinations led to the highest rates of error detection (82% for contradictions and 78% for hallucinations, respectively), paired with low agreement.

Impact of Delivery Tone

The study demonstrated that delivery styles significantly modulate user sensitivity to reasoning correctness. Across both CORRECT and INCORRECT final judgments, confident tones consistently suppress error detection—participants are less likely to notice reasoning flaws—while leaving agreement and trust largely unchanged. This effect is most pronounced for omission errors, where omission errors, which dominate in practice, are also the least detectable by humans. Conversely, hedged tones showed little effect on error detection but were found to reduce agreement on incorrect outputs. The discussion concluded that confident delivery re6[s]presents a powerful determinant of reliable use, suggesting that tone shapes how users evaluate the process of reasoning.

Model-Side Profiling and Real-World Risks

The research extended its findings by profiling six widely used VLMs in the wild to quantify error prevalence and epistemic markers. The analysis revealed a prevalence–detectability gap: omission errors are the most frequent failure mode, particularly in proprietary models, yet they are the least detectable by humans. Additionally, models exhibited a systematic bias toward confident reasoning styles, as boosters (certainty markers) appeared much more frequently than hedges (uncertainty markers). This combination of frequent omissions paired with confident delivery suggests that reasoning chains can be prone to fostering blind trust and amplifying the miscalibrated trust of model explainability. The results caution against assuming that transparency through reasoning chains naturally promotes calibrated trust.

Conclusion and Implications

The study successfully introduced a framework for controlled experiments to examine how flawed reasoning chains and confidence tones shape user trust in VLMs by triangulating trust through error detection, agreement, and self-reported reliance. The key takeaway is that explanation style is not a neutral feature, but a powerful determinant of reliable use. This work calls for explanation design that incorporates hedging or uncertainty markers to encourage critical evaluation rather than blind compliance. Future work should focus on testing these patterns across other high-stakes domains to determine if the observed mechanisms of overtrust are universal. The findings underscore the need for explanation design that promotes critical evaluation rather than blind confidence in model reasoning.

Limitations

Limitations include relying on systematically constructed perturbations instead of purely in-the-wild chains, focusing on a specific domain (MORALISE), and using controlled lexical substitutions for tone rather than capturing full pragmatic richness like prosody. Furthermore, the measures of trust are not exhaustive as they do not measure downstream reliance in consequential tasks. The study acknowledges the tradeoff between experiment control and ecological validity.

Based on the scientific paper, here are specific, actionable improvements for AI systems, categorized by the insights derived from their study:

)1. Implement Calibrated Trust Mechanisms (Addressing Q1):

The core finding is that users equate trust with outcome agreement, sustaining reliance even when reasoning is flawed.

• Improvement: Integrate a Reasoning Scrutiny Layer into the VLM output pipeline. This layer should be trained to generate meta-commentary that explicitly flags potential reasoning errors (omissions, contradictions) using uncertainty markers (hedges) before presenting the final judgment.

• Improved AI Capability: The system will proactively provide critical checkpoints in its Chain-of-Thought, forcing the user's attention onto logical steps rather than just the final conclusion.

)2. Develop Adaptive Confidence Calibration (Addressing Q2):

The finding that confident tones suppress error detection, especially for omission errors, necessitates a tone control mechanism.

• Improvement: Implement a dynamic tone adjustment module based on the detected error type and outcome correctness. If the system detects an omission or contradiction during reasoning, it must automatically default to a hedged or neutral epistemic marker (e.g., inserting it might be that, or this step suggests) regardless of the model's internal confidence score.

• Improved AI Capability: The system will learn to modulate its linguistic style in real-time; it will sound more cautious and less certain when its internal reasoning process hits known failure modes, preventing confident hallucinations from leading to blind trust.

)3. Prioritize Detection Over Agreement (Addressing Q1/Q2):

The study shows that Agreement is the most dominant factor in trust, while Error Detection is the most critical for calibration.

• Improvement: Re-weight the training objectives for VLMs to prioritize internal consistency and logical fidelity over generating high-agreement outputs. Use Reinforcement Learning from Human Feedback (RLHF) specifically to penalize chains that are confident but logically flawed or incomplete, even if they lead to a socially acceptable outcome.

• Improved AI Capability: The system will shift from being an outcome predictor to a reasoning verifier. It will be optimized not just for getting the right answer, but for generating reasoning paths that are robust and transparent enough to be critically evaluated by a human user.

)4. Mitigate the Omission-Detectability Gap (Addressing Section 5):

The finding that omission errors are frequent but hardest to detect is a major risk factor.

• Improvement: Develop an automated Completeness Checker module that cross-references visual inputs with textual reasoning steps to ensure all necessary premises and inferential links have been explicitly mentioned.

• Improved AI Capability: The system will perform self-auditing on its CoT, flagging instances where critical contextual details (especially from the image) are omitted, thereby reducing the prevalence of plausible but incomplete reasoning that users blindly trust.

)5. Enhance Model Profiling for Safety (Addressing Section 5):

The model-side analysis shows that closed-source models lean toward omissions and open-source models lean toward hallucinations/contradictions.

• Improvement: Implement a rigorous, standardized Faithfulness Metric pipeline during model fine-tuning that explicitly targets the error types identified in the study (omission, contradiction, hallucination) with specific loss functions.

• Improved AI Capability: The system will be hardened against its own known failure modes. For instance, if it is an open-source model, it will be specifically trained to reduce hallucinations; if it is a closed-source model, it will be incentivized to ensure comprehensive step coverage and minimal omissions in its reasoning structure.

Explanations are often promoted as tools for transparency, but they can also foster confirmation bias; users may assume reasoning is correct whenever outputs appear acceptable. We study this double-edged role of Chain-of-Thought (CoT) explanations in multimodal moral scenarios by systematically perturbing reasoning chains and manipulating delivery tones. Specifically, we analyze reasoning errors in vision language models (VLMs) and how they impact user trust and the ability to detect errors. Our findings reveal two key effects: (1) users often equate trust with outcome agreement, sustaining reliance even when reasoning is flawed, and (2) the confident tone suppresses error detection while maintaining reliance, showing that delivery styles can override correctness. These results highlight how CoT explanations can simultaneously clarify and mislead, underscoring the need for NLP systems to provide explanations that encourage scrutiny and critical thinking rather than blind trust. All code will be released publicly.