No Model Required: Text Entropy Rate Filtering Mitigates Iterative Fine-Tuning Collapse
cs.CL, cs.AI, cs.IT, math.IT, physics.data-an, stat.ML
Submitted: 2026-10-01
Updated: 2026-10-01
Code: https://github.com/lewismath/no-model-required
Terminology
Sources
- Large Language Models Suffer From Their Own Output: An Analysis of the Self-Consuming Training Loop
- Diversity Collapse in Multi-Agent LLM Systems: Structural Coupling and Collective Failure in Open-Ended Idea Generation
- The Llama 3 Herd of Models
- Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
- Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data
- Qwen2.5 Technical Report
- Escaping Model Collapse via Synthetic Data Verification: Near-term Improvements and Long-term Convergence
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering