Post-Training Language Models for Crosslingual Consistency

arXiv:2603.04678 · cs.CL, cs.AI · Submitted 2026-03-04 · Read on arXiv

cs.CL, cs.AI

Submitted: 2026-03-04

Updated: 2026-05-28

Comments: ICML 2026. The first two authors contributed equally. Codes available at: https://github.com/Betswish/ConsistencyRL

Journal ref: Proceedings of the 43rd International Conference on Machine Learning (ICML 2026), 2026

Code: https://github.com/Betswish/ConsistencyRL

License: http://creativecommons.org/licenses/by/4.0/

The gist: Language models often respond inconsistently to translation-equivalent prompts across languages, undermining the reliability of multilingual systems.

Terminology

Abstract

Language models often respond inconsistently to translation-equivalent prompts across languages, undermining the reliability of multilingual systems. To quantify this, we give an information-theoretic definition of crosslingual consistency as a divergence bound between a model's response distribution and its round-trip pushforward across languages. We then introduce penalized consistency optimization (PCO), a post-training procedure that couples this divergence with a Kullback-Leibler penalty to a fixed reference language model. Because direct optimization of PCO requires expensive on-policy roll-outs, we propose a tractable surrogate, direct consistency optimization (DCO), which can be optimized off-policy. Across diverse language models and 26 languages, DCO significantly improves crosslingual consistency, outperforms existing methods, and enables targeted alignment of low-resource languages.

Sources

Related papers