When Should LLMs Abstain? Chain-of-Self-Questioning for Selective Risk Control

arXiv:2609.17516 · cs.CL, cs.AI · Submitted 2026-09-15 · Read on arXiv

cs.CL, cs.AI

Submitted: 2026-09-15

Updated: 2026-09-15

Code: https://github.com/senolali/CoSQ

License: http://creativecommons.org/licenses/by/4.0/

The gist: Large language models can produce fluent answers when their factual support is weak.

Terminology

Abstract

Large language models can produce fluent answers when their factual support is weak. This paper introduces Chain-of-Self-Questioning (CoSQ), a prompt-only framework that makes answer commitment conditional on an explicit assessment of the information required to answer a question. We evaluate three CoSQ variants under seventeen conditions on the 817-item TruthfulQA multiple-choice validation set using eleven open-weight and hosted model families. In the final balanced-option protocol, Grounded-CoSQ at τ=0.90 reduces the mean unconditional wrong-commitment rate from 13.1% under chain-of-thought prompting to 8.9%, a 32.1% relative reduction, while increasing answered accuracy from 86.9% to 89.7% and answering 87.6% of questions. Both improvements hold for all eleven models and at every evaluated threshold. Critical-CoSQ and Adaptive-CoSQ provide neighboring operating points with 88.6% and 86.5% coverage, respectively, while remaining more reliable than the baseline. A secondary Natural Questions Short-Answer evaluation provides convergent open-form evidence. These findings show that self-assessment can support explicit, tunable answer-or-abstain decisions when an unsupported commitment is more costly than referral or review.

Sources

Related papers