What Do Rationales Communicate? A Message-Intervention Study in Role-Specialized QA

Role-specialized QA pipelines increasingly pass rationales from a reasoner to a verifier, but it is unclear what this message actually buys: better answers, stronger support assessment, or a new failure surface.

The gist: faithful rationales add almost no answer accuracy over no rationale, while corrupted rationales strongly alter support judgments.

Diagnostic Framework

The study introduces a message-intervention diagnostic that fixes the evidence and candidate answer while varying only the rationale passed across the reasoner-verifier boundary to measure answer selection and support assessment separately. The core of this protocol treats the rationale as an explicit message sent from one role to another, rather than a hidden trace of model thinking. This allows researchers to test what happens when explanation-like text becomes a communication payload consumed by a later role.

The diagnostic is designed as a causal intervention on the message between roles. The task, evidence, candidate answer, verifier role, and output schema are fixed; only the rationale changes. If the verifier behavior changes under this intervention, the change is attributable to the rationale field rather than to a different retriever or reasoner.

Perturbation Conditions

Each example is evaluated under five distinct conditions to separate two primary questions: whether a message tests brittleness to wording or whether it responds to factual errors. These conditions include:

1. No rationale: The verifier receives only evidence and the candidate answer.

2. Original-rationale: The verifier receives the reasoner-generated rationale.

3. Harmless paraphrase: A meaning-preserving rephrasing of the rationale that preserves entities, relations, dates, and answer support.

4. Entity swap corruption: Replacing a key entity with an incompatible alternative (e.g., swapping a place or relation).

5. Answer-conflicting corruption: Altering the rationale to imply a different plausible answer than the candidate answer.

Metrics and Testing

The study defines paired metrics to separate answer selection from support assessment while holding evidence and candidate answers fixed:

AnsSens(c)

This metric measures how often the candidate answer changes under condition 'c' relative to the original rationale condition 'r'. Similarly, support sensitivity is measured by:

SuppSens(c)

The study also reports a coupling probability, denoted as:

Couple(c)

This asks how often answer changes accompany support flips. The researchers emphasize that they avoid making the difference between answer-span changes and binary support changes carry the main argument because the two outputs have different base rates and degrees of freedom.

Key Findings on Rationale Effects

The diagnostic shows that rationales are strongest as verification messages, not answer selection signals. Specifically:

1. Harmless paraphrases rarely change verifier judgments, shifting support by only 0–2.5% under a blind prompt.

2. Corrupted rationales strongly alter support judgments, shifting support by 10–22% under a blind prompt and up to 34–55% when an explicit rationale-checking prompt is used.

3. Final answers move less (typically 2–30%) compared to the strong shifts in support judgments.

Audits and Failure Modes

Human audits revealed two auditable failure modes that final EM alone cannot see:

1. Corruption-overtrust: The verifier keeps accepting a corrupted rationale, and blind human labels reject or mark unclear 9/10 audited corrupted-supported cases.

2. Correct-answer penalty: A correct answer is rejected because its rationale is corrupted.

A third qualitative pattern observed is local answer dominance, where direct evidence preserves the answer despite a broken upstream rationale.

Design Implications

The results suggest that before relying on a message field, one should test what receiver behavior it actually changes. After an answer is formed, rationales should be treated as claims to verify against evidence rather than collapsing them into a single binary failure state. The design rule suggested is: before relying on a message field, test what receiver behavior it actually changes. If the channel is active, systems should distinguish between rationale repair and support escalation. If it is ignored, passing rationales may add interface complexity without communication value.

Contribution Summary

The paper makes three main contributions:

1. Introducing a message-intervention protocol for testing what a rationale changes after it is passed to a verifier.

2. Defining paired metrics that separate answer selection from support assessment while holding evidence and candidate answers fixed.

3. Showing across three multi-hop QA datasets that harmless paraphrases rarely change verifier judgments, while corrupted rationales substantially change support judgments, identifying when this channel is active, amplified, inert, or task-coupled.

Based on the provided research paper, here are specific improvements for AI systems and what those improved systems can achieve:

---

) 1. Implement a Rationale Channel Diagnostic in Role-Specialized QA Pipelines:

The system should not automatically pass rationales from a reasoner to a verifier; instead, it must first execute the diagnostic protocol described in Section 2.1. This diagnostic involves testing the rationale as an explicit message against five conditions (No Rationale, Original Rationale, Harmless Paraphrase, Entity Swap Corruption, Answer Conflict Corruption).

• A system can be designed to classify the rationale channel into one of three auditable states:

List of three states and their implications:

1. Active Channel: The rationale strongly alters support judgments (e.g., 34–55% for corrupted rationales when explicitly checked). This state should trigger automated actions like Rationale Repair, Evidence Rechecking, or Support Escalation.

2. Harmless Channel: Harmless paraphrases leave support stable (e.g., 0–2.5% change under blind prompts). This channel is safe for use but provides no strong signal for intervention.

3. Inert Channel: The rationale has little communication value (e.g., DeepSeek-R1 boundary case). The system should discount this message, adding interface complexity without benefit, and potentially switch to a more independent verification method.

• What the improved AI can do: This allows systems to dynamically decide whether to trust a rationale as evidence for repair or simply ignore it as inert text, preventing the propagation of false information into downstream decision-making.

) 2. Introduce Paired Metrics for Separation of Concerns:

The system should move beyond single metrics (like final EM accuracy) and adopt the paired metrics defined in Section 2.3 to separate Answer Selection from Support Assessment, while holding evidence and candidate answers fixed.

• Metrics to track: Answer Sensitivity (how much the answer changes), Support Sensitivity (how much the support judgment changes), and Coupling Probability (P(∆Ans ∆S)).

• What the improved AI can do: This enables fine-grained debugging. If an answer is wrong but support is correct, it points toward a Correct-Answer Penalty issue driven by rationale quality. If both change, it indicates a deeper structural conflict. This allows developers to diagnose why an answer was selected versus why it was judged unsupported.

) 3. Implement Corruption-Aware Verifier Training:

Training should explicitly expose the model to the patterns identified in Section 4 and Figure 1—specifically, how factual errors in rationales lead to support flips (34–55% sensitivity under explicit checks).

• System Improvement: Use synthetic datasets where rationales are intentionally corrupted (entity swaps, answer conflicts) to train verifiers on the robust behavior shown by harmless paraphrases.

• What the improved AI can do: This creates a verifier that is less brittle to upstream message quality, increasing reliability in production environments where inputs might be generated by other LLMs or complex reasoning steps.

) 4. Develop a Support Bit Combines Answer and Rationale Checks:

Instead of relying on a single binary support label, the system should utilize the split-channel verification output (as shown in Table 8).

• System Improvement: The verifier should return three separate binary judgments: Answer Support, Rationale Faithfulness, and Overall Support.

• What the improved AI can do: This distinguishes between a correct answer with a bad explanation (Answer is supported but rationale is bad) and an overall unacceptable output. This prevents the system from incorrectly penalizing a correct answer simply because its supporting reasoning trace was flawed, while still flagging cases where the entire reasoning chain (including the rationale) is suspect.

) 5. Integrate Task-Boundary Checks for Contextual Sensitivity:

The system should dynamically adjust its sensitivity based on the task domain, as suggested by Figure 3 and Section 4.7.

• System Improvement: Implement a Task Sensitivity Monitor that assigns weighting to the rationale channel based on the QA type (e.g., high weight for MuSiQue/HotpotQA, lower weight for SciFact).

• What the improved AI can do: This prevents over-engineering in verification pipelines by applying stricter checks where they matter most (complex multi-hop reasoning) and allowing more flexibility where the task label itself is a support judgment (claim verification).