Equilibrium Training of Energy-Based Models with Parallel Trajectory Tempering
cs.LG, cond-mat.dis-nn, cond-mat.stat-mech
Submitted: 2026-07-29
Updated: 2026-09-07
Comments: 11 pages, 7 figures
License: http://creativecommons.org/licenses/by-nc-nd/4.0/
The gist: Energy-Based Models (EBMs) provide an interpretable framework for generative modeling of scientific data, but poor Markov Chain Monte Carlo mixing often limits their reliability.
Terminology
Abstract
Energy-Based Models (EBMs) provide an interpretable framework for generative modeling of scientific data, but poor Markov Chain Monte Carlo mixing often limits their reliability. We introduce a training algorithm based on Parallel Trajectory Tempering (PTT), which exploits the continuity of the optimization path to maintain equilibrium sampling throughout learning. This enables stable and fast training on highly multimodal and data-scarce scientific datasets. Combined with reservoir sampling and adaptive optimization, PTT has a computational cost comparable to Persistent Contrastive Divergence, making it a practical replacement for standard training methods. It also provides direct estimates of thermalization times, equilibrium samples from trained models, and accurate log-likelihoods at essentially no additional cost. Experiments on Restricted Boltzmann Machines show that PTT consistently outperforms existing EBM training approaches. On discrete tabular data, it also surpasses state-of-the-art deep generative models, yielding higher-quality samples and greater robustness to overfitting and limited data. Our results make equilibrium maximum-likelihood training of EBMs practical and computationally efficient.
Sources
- Distributional simplicity bias and effective convexity in Energy Based Models
- Uncovering statistical structure in large-scale neural activity with Restricted Boltzmann Machines
- Gaussian-Bernoulli RBMs Without Tears
- Cascade of phase transitions in the training of Energy-based models
- Explaining the effects of non-convergent sampling in the training of Energy-Based Models
- Bayesian Flow Networks
- PRIVET: PRoximIty leakage detection Via Extreme value Theory
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: Local-Consistency Optimisation for Bayesian Neural Networks
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks