Towards Reliable Long-Context Reasoning: Detecting Unanswerable Questions via FactGuard
summary
The gist
The gist The FactGuard framework introduces an innovative multi-agent collaborative methodology to autonomously generate answerable and unanswerable questions, developing the FactGuard-Bench dataset
In short
Researchers developed FACTGUARD, a multi-agent workflow to create and evaluate questions that are either answerable or unanswerable in long texts. They created FactGuard-Bench, a benchmark with 25,220 examples testing LLMs on recognizing missing or misleading evidence. Results show that training with this data significantly improves models' ability to correctly reject unanswerable questions.
Key concepts
- FACTGUARD
- A systematic multi-agent workflow designed to generate both answerable and unanswerable questions from long documents. It uses specialized agents in a collaborative process to ensure the generated questions are well-grounded in the source text, covering preparation, QA generation, and negative example creation.
- FactGuard-Bench
- A comprehensive benchmark dataset containing 25,220 examples of questions derived from texts up to 128K tokens long. It specifically targets two failure modes: lacking evidence and encountering misleading evidence in long contexts, making it a rigorous test for LLM reliability.
- Contextually Missing Negative Examples
- A way to create unanswerable questions by simply removing the supporting evidence from the original text. This tests whether an LLM can recognize that a question cannot be answered because the necessary information is absent from the provided context.
- Misleading Negative Examples
- Creating unanswerable questions by intentionally manipulating existing tuples through techniques like entity substitution or inserting impossible conditions. This tests a model's ability to reject questions based on false assumptions rather than just missing facts.
Terminology used across episodes
This episode discusses
- Towards Reliable Long-Context Reasoning: Detecting Unanswerable Questions via FactGuard · Paper Radio
- GPT-4 Technical Report
- Efficient Pretraining Data Selection for Language Models via Multi-Actor Collaboration
- Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
- LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding
- LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks
- ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate
- TigerBot: An Open Multilingual Multitask LLM
- DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
- Don't Just Say "I don't know"! Self-aligning Large Language Models for Responding to Unknown Questions with Explanations
- The Llama 3 Herd of Models · Paper Radio
- Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
- Pile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal Dataset
- MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework
- The FACTS Grounding Leaderboard: Benchmarking LLMs' Ability to Ground Responses to Long-Form Input
- Mixtral of Experts
- LooGLE: Can Long-Context Language Models Understand Long Contexts?
- AgentInstruct: Toward Generative Teaching with Agentic Flows
- Relation Extraction: A Survey
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- Astute RAG: Overcoming Imperfect Retrieval Augmentation and Knowledge Conflicts for Large Language Models
The paper
Towards Reliable Long-Context Reasoning: Detecting Unanswerable Questions via FactGuard · Read on arXiv
Qian-Wen Zhang, Fang Li, Jie Wang, Lingfeng Qiao, Yifei Yu, Di Yin, Xing Sun
Tencent Youtu Lab
Despite the success of Large Language Models (LLMs) in long-context tasks, their reliability in recognizing unanswerable questions remains a critical challenge. In information-seeking scenarios, models often suffer from overconfidence and hallucinations when evidence is missing or misleading. In this paper, we propose FACTGUARD, a systematic multi-task workflow for generating evidence-grounded answerable and unanswerable questions. We introduce FACTGUARD-BENCH, a benchmark comprising 25,220 examples with context lengths up to 128K, specifically targeting Lack of Evidence and Misleading Evidence scenarios. Evaluations of eleven representative open-source and proprietary LLMs reveal a significant reliability gap: models struggle to reject unanswerable questions as context scales, with the leading model achieving only 71.62% accuracy. By training with FACTGUARD-BENCH, we improve overall accuracy to 81.17% and significantly enhance models' reasoned rejection capabilities. Our work provides a scalable path toward building more faithful and trustworthy long-context LLMs. Our source is publicly available at https://github.com/biaoliu-kiritsugu/FactGuard.
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: Today's paper: "Towards Reliable Long-Context Reasoning".
Jane: The gist The FactGuard framework introduces an innovative multi-agent collaborative methodology to autonomously generate answerable and unanswerable questions, developing the FactGuard-Bench dataset to evaluate long-context LLM extraction capabilities.
Tom: First, who's behind it and why it matters.
Title and authors: Tom: Let's talk about how they set up the title and who wrote it. The name "FactGuard" suggests a system that acts like a guard, checking if an answer is solid or if the question is flawed >
Jane: It’s about leveraging multi-agent systems to generate questions that are both answerable and unanswerable. That’s what the title tells us about the core idea behind this paper >
Lu: The authors are Qian-Wen Zhang, Fang Li, Jie Wang, Lingfeng Qiao, Yifei Yu, Di Yin and Xing Sun. They are from Tencent YouTu Lab in Beijing >
Meng: I've seen a lot of work on multi-agent systems lately. The real question for me is how complex this collaboration really is when you’re trying to make sure the final output is useful data for training an AI >
Lalam: The team uses a console that aggregates the opinions from all these different agents to make the final decision about whether a piece of text should become an answerable question or a negative example >
The paper's summary: Tom: So, what is the actual process they describe? They use three main stages: preparation, QA generation, and then negative example generation. It sounds like they are building the data piece by piece >
Jane: Exactly. In the preparation stage, they select short text segments from huge documents and then refine that selection using quality scoring and topic selection >
Lu: The quality scoring uses quality agents to give each segment a score between one and five, while the topic agents select different categories like time or location from those segments >
Meng: So they are essentially pre-processing the raw text into manageable chunks with some basic checks before they even try to ask a question about it >
Lalam: And then in the QA generation stage, they have specialized QA agents produce tuples that include the fragment, the question, the answer, and the evidence to make sure everything is grounded in context >
The paper's improvements: Tom: The paper suggests a few key improvements over older methods. One is this multi-agent collaboration framework itself for generating challenging examples without needing manual annotation >
Jane: That’s huge because it bypasses the need for expensive human annotation, which is what makes creating large datasets so difficult right now >
Lu: They also focus on how to generate unanswerable questions through two specific ways: contextually missing evidence and misleading negative examples >
Meng: The misleading part sounds tricky. It means they are using rewriting agents to do things like substitution or inserting impossible conditions, which is a clever way to test a model's true understanding of the context >
Lalam: And they also have post-processing reviews to filter out questions that might have conflicting answers or common sense that doesn't really matter in this context >
Conclusion: Tom: So, to wrap up this paper on FactGuard, it’s about using these multi-agent systems to create a diverse set of long-context training data and a benchmark called FactGuard-Bench >
Jane: It shows that even the best models have critical shortcomings when it comes to unanswerable questions in those extended contexts, and this framework helps us measure that failure more clearly >
Lu: The results show that using SFT and DPO fine-tuning techniques can significantly improve the model’s reasoning ability for these difficult queries, boosting performance from around forty-one point two one percent to eighty-two point three nine percent in the best variant >
Meng: For us engineers, seeing that improvement means we have a clearer path on how to push those models past this barrier when we deploy them at scale >
Lalam: The overall quality of the synthetic data they generate is quite high, with ninety-three percent being good-quality unanswerable QA pairs, which shows the value of this method >
Tom: It seems like this work really shifts how we approach long-context reading comprehension by giving us a tool to systematically test and improve those models >
Jane: Indeed. We’re excited to see what they do next with FactGuard-Bench and how it pushes the limits of what current LLMs can handle >
More episodes
- 2406.20031-Pairwise Difference Learning for Classification
- 2311.01994-Distributionally Robust Regularization of Sparse Integer Programming Trained Learning Models
- 2312.02659-Supervised learning of spatial features with STDP and homeostasis using Spiking Neural Networks on SpiNNaker
- 2409.12500-LLMR: Knowledge Distillation with a Large Language Model-Induced Reward
- 2402.05804-InkSight: Offline-to-Online Handwriting Conversion by Teaching Vision-Language Models to Read and Write
- 2508.08100-Grid2Guide: A* Enabled Small Language Model for Indoor Navigation
- 2408.04140-UNLEARN Efficient Removal of Knowledge in Large Language Models
- 2504.03669-A self-learning framework for adaptive free-form pipe routing in aeroengines
- 2410.05203-Beyond FVD: Enhanced Evaluation Metrics for Video Generation Quality
- 2507.09227-PanoDiff-SR: Synthesizing Dental Panoramic Radiographs using Diffusion and Super-resolution