UrduFactCheck: An Agentic Fact-Checking Framework for Urdu with Evidence Boosting and Benchmarking

arXiv:2505.15063 · cs.CL · Submitted 2025-05-21 · Read on arXiv

cs.CL

Submitted: 2025-05-21

Updated: 2026-09-11

Comments: 15 pages, 4 figures, 5 tables, 6 Listings, In Findings of the Association for Computational Linguistics: EMNLP 2025

Journal ref: In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 22788-22802, Suzhou, China. Association for Computational Linguistics

DOI: 10.18653/v1/2025.findings-emnlp.1240

Code: https://github.com/mbzuai-nlp/UrduFactCheck

License: http://creativecommons.org/licenses/by/4.0/

The gist: The rapid adoption of Large Language Models (LLMs) has raised important concerns about the factual reliability of their outputs, particularly in low-resource languages such as Urdu.

Terminology

Abstract

The rapid adoption of Large Language Models (LLMs) has raised important concerns about the factual reliability of their outputs, particularly in low-resource languages such as Urdu. Existing automated fact-checking systems are predominantly developed for English, leaving a significant gap for the more than 200 million Urdu speakers worldwide. In this work, we present UrduFactBench and UrduFactQA, two novel hand-annotated benchmarks designed to enable fact-checking and factual consistency evaluation in Urdu. While UrduFactBench focuses on claim verification, UrduFactQA targets the factuality of LLMs in question answering. These resources, the first of their kind for Urdu, were developed through a multi-stage annotation process involving native Urdu speakers. To complement these benchmarks, we introduce UrduFactCheck, a modular fact-checking framework that incorporates both monolingual and translation-based evidence retrieval strategies to mitigate the scarcity of high-quality Urdu evidence. Leveraging these resources, we conduct an extensive evaluation of twelve LLMs and demonstrate that translation-augmented pipelines consistently enhance performance compared to monolingual ones. Our findings reveal persistent challenges for open-source LLMs in Urdu and underscore the importance of developing targeted resources. All code and data are publicly available at https://github.com/mbzuai-nlp/UrduFactCheck.

Sources

Related papers