AskQE: Question Answering as Automatic Evaluation for Machine Translation

arXiv:2504.11582 · cs.CL · Submitted 2025-04-15 · Read on arXiv

cs.CL

Submitted: 2025-04-15

Updated: 2026-09-20

Comments: ACL 2025 Findings

Code: https://github.com/dayeonki/askqe

License: http://creativecommons.org/licenses/by/4.0/

The gist: How can a monolingual English speaker determine whether an automatic translation in French is good enough to be shared? Existing MT error detection and quality estimation (QE) techniques do not

Terminology

Abstract

How can a monolingual English speaker determine whether an automatic translation in French is good enough to be shared? Existing MT error detection and quality estimation (QE) techniques do not address this practical scenario. We introduce AskQE, a question generation and answering framework designed to detect critical MT errors and provide actionable feedback, helping users decide whether to accept or reject MT outputs even without the knowledge of the target language. Using ContraTICO, a dataset of contrastive synthetic MT errors in the COVID-19 domain, we explore design choices for AskQE and develop an optimized version relying on LLaMA-3 70B and entailed facts to guide question generation. We evaluate the resulting system on the BioMQM dataset of naturally occurring MT errors, where AskQE has higher Kendall's Tau correlation and decision accuracy with human ratings compared to other QE metrics.

Sources

Related papers