Adapting English Quality Classifiers for Multilingual LLM Pretraining Data Selection

arXiv:2610.11585 · cs.CL, cs.AI · Submitted 2026-10-08 · Read on arXiv

cs.CL, cs.AI

Submitted: 2026-10-08

Updated: 2026-10-08

Code: https://github.com/huggingface/datatrove

Terminology

• Embeddings — 30× in this paper · explained in 2 episodes
• Benchmark — 28× in this paper · explained in 2 episodes
• Generalization — 14× in this paper · explained in 13 episodes
• Adaptation — 11× in this paper · explained in 1 episode
• Transformer — 6× in this paper · explained in 2 episodes
• LLM-as-a-Judge — 4× in this paper · explained in 3 episodes
• Accuracy — 2× in this paper · explained in 1 episode
• Adam — 2× in this paper · explained in 2 episodes
• Cross-lingual Transfer — 2× in this paper · explained in 2 episodes
• APEX — 1× in this paper · explained in 1 episode
• Alignment — 1× in this paper · explained in 1 episode
• Decoupling — 1× in this paper · explained in 2 episodes
• Fine-Tuning — 1× in this paper · explained in 5 episodes
• LLM judge — 1× in this paper · explained in 2 episodes
• Large Language Model (LLM) — 1× in this paper · explained in 3 episodes
• Large Language Models (LLMs) — 1× in this paper · explained in 6 episodes
• Loss Function — 1× in this paper · explained in 2 episodes
• Perplexity — 1× in this paper · explained in 3 episodes
• Pruning — 1× in this paper · explained in 4 episodes
• Regularization — 1× in this paper · explained in 3 episodes
• Reinforcement Learning — 1× in this paper · explained in 4 episodes
• Representation Learning — 1× in this paper · explained in 2 episodes
• Scaling Laws — 1× in this paper · explained in 4 episodes
• Transfer Learning — 1× in this paper · explained in 10 episodes

Sources

• Apertus: Democratizing Open and Compliant LLMs for Global Language Environments
• An Expanded Massive Multilingual Dataset for High-Performance Language Technologies (HPLT)
• MuRating: A High Quality Data Selecting Approach to Multilingual Large Language Model Pretraining
• CulturalBench: A Robust, Diverse, and Challenging Cultural Benchmark by Human-AI CulturalTeaming
• Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
• XNLI: Evaluating Cross-lingual Sentence Representations
• Unsupervised Cross-lingual Representation Learning at Scale
• A New Massive Multilingual Dataset for High-Performance Language Technologies
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
• MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies
• Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations
• Bag of Tricks for Efficient Text Classification
• Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedback
• The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset
• Decoupled Weight Decay Regularization
• mmBERT: A Modern Multilingual Encoder with Annealed Language Learning
• Enhancing Multilingual LLM Pretraining with Model-Based Data Selection
• Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering
• Crosslingual Generalization through Multitask Finetuning
• CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languages

Related papers

• Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
• Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
• Subliminal Steering: Stronger Encoding of Hidden Signals
• MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• Untangling the Mechanisms of Misleading Context in Medical Question Answering