Reparameterization through Coverings and Topological Weight Priors
cs.LG
Submitted: 2026-04-26
Updated: 2026-09-01
Journal ref: Data Analytics and Topology Volume 2 2026 193-204
License: http://creativecommons.org/licenses/by/4.0/
The gist: We generalise the reparameterization trick (RT) applied in variational autoencoders (VAEs) letting these have latent spaces of non-trivial topology - i.e.
Terminology
Abstract
We generalise the reparameterization trick (RT) applied in variational autoencoders (VAEs) letting these have latent spaces of non-trivial topology - i.e. that of base manifolds covered with other ones, on which some technique for RT is available. That is possible since covering maps are measurable - moreover, this allows to establish an inequality on KL-divergence between pushforward (PF) densities on the base latent manifold, bounding it with KL-divergence between pullbacks on the cover, in some cases making the KL-term of VAE's ELBO analytically tractable, despite the topological non-triviality of the supporting latent manifold. Our development follows a route close but somewhat alternative to reparameterization on Lie groups, the latest proposal for which is to reparameterize PFs of normal densities from the Lie algebra - "through" the exponential map, seen by us as a particular case of what we propose to call reparameterization via covering (RVC). We demonstrate the working of our approach by constructing a VAE with the latent space of Klein bottle (not a Lie group) topology, which we call KleinVAE, successfully learning an appropriate artificial dataset. We discuss potential applicability of such topology-informed generative models as weight priors in Bayesian learning, particularly for convolutional vision models, where said manifold was peculiarly shown to have some relevance.
Sources
- The Deep Weight Prior
- Hyperspherical Variational Auto-Encoders
- Explorations in Homeomorphic Variational Auto-Encoding
- Bayesian Neural Network Priors Revisited
- Exposition and Interpretation of the Topology of Neural Networks
- Normalizing Flows on Riemannian Manifolds
- Density estimation on smooth manifolds with normalizing flows
- Adam: A Method for Stochastic Optimization
- Field choice problem in persistent homology
- Structured Weight Priors for Convolutional Neural Networks
- Diffusion Variational Autoencoders
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Learning Topology-Preserving Data Representations
- Learning Geometry and Topology via Multi-Chart Flows
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: Local-Consistency Optimisation for Bayesian Neural Networks
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks