Preventing Model Collapse: A Fisher-Rao Perspective on the Dynamics of Training with Synthetic Data
cs.LG, cs.SY, eess.SY
Submitted: 2026-09-16
Updated: 2026-09-16
Comments: 8 pages. Extended version of the paper accepted for presentation at the 2026 65th IEEE Conference on Decision and Control (CDC). This version contains the full proofs of the auxiliary lemmas, omitted from the conference version for space
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Terminology
Sources
- A Survey of Large Language Models
- Llama 2: Open Foundation and Fine-Tuned Chat Models
- Will we run out of data? Limits of LLM scaling based on human-generated data
- Heat Death of Generative Models in Closed-Loop Learning
- Large Language Models Suffer From Their Own Output: An Analysis of the Self-Consuming Training Loop
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: Local-Consistency Optimisation for Bayesian Neural Networks
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks