ReScraper: Unified Scraping and Cleaning of Web Data for Effective LLM Pretraining

arXiv:2609.34287 · cs.CL, cs.AI, cs.LG · Submitted 2026-09-28 · Read on arXiv

cs.CL, cs.AI, cs.LG

Submitted: 2026-09-28

Updated: 2026-09-28

Code: https://github.com/cxcscmu/ReScraper

Terminology

Core — 21× in this paper · explained in 2 episodes
Accuracy — 10× in this paper · explained in 1 episode
LLM judge — 4× in this paper · explained in 2 episodes
Ablation — 3× in this paper · explained in 1 episode
Perplexity — 3× in this paper · explained in 3 episodes
Aggregation — 2× in this paper · explained in 1 episode
Downstream Tasks — 2× in this paper · explained in 2 episodes
Pruning — 2× in this paper · explained in 4 episodes
Transformer — 2× in this paper · explained in 2 episodes
Adam — 1× in this paper · explained in 2 episodes
Alignment — 1× in this paper · explained in 1 episode
Benchmarking — 1× in this paper · explained in 3 episodes
Data Pruning — 1× in this paper · explained in 2 episodes
Decode — 1× in this paper · explained in 2 episodes
Distillation — 1× in this paper · explained in 2 episodes
Fine-Tuning — 1× in this paper · explained in 5 episodes
Foundation Model — 1× in this paper · explained in 3 episodes
Foundation Models — 1× in this paper · explained in 7 episodes
LLM-as-a-Judge — 1× in this paper · explained in 3 episodes
Provenance — 1× in this paper · explained in 3 episodes
Thinking mode — 1× in this paper · explained in 2 episodes
Transfer Learning — 1× in this paper · explained in 10 episodes
Two-stage training — 1× in this paper · explained in 2 episodes

Related papers