TransBERT: A Framework for Synthetic Translation in Domain-Specific Language Modeling

arXiv:2609.26347 · cs.CL, cs.AI, cs.LG · Submitted 2026-09-22 · Read on arXiv

cs.CL, cs.AI, cs.LG

Submitted: 2026-09-22

Updated: 2026-09-23

Comments: 17 pages

Journal ref: Findings of the Association for Computational Linguistics: EMNLP 2025, pages 19338-19354

DOI: 10.18653/v1/2025.findings-emnlp.1053

Code: https://github.com/jknafou/TransCorpushttps:

License: http://creativecommons.org/licenses/by/4.0/

The gist: The scarcity of non-English language data in specialized domains significantly limits the development of effective Natural Language Processing (NLP) tools.

Terminology

Abstract

The scarcity of non-English language data in specialized domains significantly limits the development of effective Natural Language Processing (NLP) tools. We present TransBERT, a novel framework for pre-training language models using exclusively synthetically translated text, and introduce TransCorpus, a scalable translation toolkit. Focusing on the life sciences domain in French, our approach demonstrates that state-of-the-art performance on various downstream tasks can be achieved solely by leveraging synthetically translated data. We release the TransCorpus toolkit, the TransCorpus-bio-fr corpus (36.4GB of French life sciences text), TransBERT-bio-fr, its associated pre-trained language model and reproducible code for both pre-training and fine-tuning. Our results highlight the viability of synthetic translation in a high-resource translation direction for building high-quality NLP resources in low-resource language/domain pairs.

Sources

Related papers