Diffusion models recover accurate mixture weights despite score function insensitivity
Andrew Dennehy, Ramchandran Muthukumar, Rebecca Willett, Nisha Chandramoorthy
cs.LG, math.PR, stat.ML
Submitted: 2026-07-16
Comments: 37 pages, 7 figures
Code: https://github.com/ax-ml/jax
License: http://creativecommons.org/licenses/by/4.0/
The gist: Score-based generative models exhibit a puzzling behavior: they often appear to cover all modes of a target multimodal distribution and yet may fail to learn the correct relative mode amplitudes,
Terminology
Abstract
Score-based generative models exhibit a puzzling behavior: they often appear to cover all modes of a target multimodal distribution and yet may fail to learn the correct relative mode amplitudes, which can be interpreted as mixture weights. We resolve this apparent paradox by relating the diffusion score matching (DSM) loss to the error in estimating mixture weights from generated samples. We show that, even when the target score is insensitive to mixture weights, generated samples can recover the weights accurately if the scores at intermediate noise levels are informative about the weights. Accordingly, we define the diffusion score sensitivity index (DSSI) as the variation in the DSM loss relative to changes in a parameter. We then show that the DSSI governs the accuracy with which the parameter of the target distribution can be estimated from generated samples. For Gaussian mixtures in arbitrary dimensions, we prove that the mixture weight estimation errors are on the same order as the DSM loss under mild conditions. Empirically, we show the emergence of sensitivity during the noising process of benchmark data distributions under typical noise schedules, and that these sensitivity values predict how well a well-trained model recovers mixture weights. Furthermore, we show that the choice of noise schedule can reduce diffusion sensitivity, leading to mode amplification. Although we focus on mixture weights, the proposed sensitivity framework governs the recovery of any qualitative parameter of the target distribution.
Sources
- Learning noisy tissue dynamics across time scales
- On the Collapse Errors Induced by the Deterministic Sampler for Diffusion Models
- Deeper Diffusion Models Amplify Bias
- Machine learning emulation of precipitation from km-scale UK regional climate simulations using a diffusion model
- Cosmological Field Emulation and Parameter Inference with Diffusion Models
- How Faithful is your Synthetic Data? Sample-level Metrics for Evaluating and Auditing Generative Models
- Convergence of score-based generative modeling for general data distributions
- Spontaneous Symmetry Breaking in Generative Diffusion Models
- Denoising Diffusion Implicit Models
- DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps
- Your diffusion model secretly knows the dimension of the data manifold
- MCMC for multi-modal distributions
- Improved Noise Schedule for Diffusion Training
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: Local-Consistency Optimisation for Bayesian Neural Networks
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks