Autoencoders in Function Space
stat.ML, cs.LG
Submitted: 2024-08-02
Updated: 2026-08-31
Comments: 54 pages, 24 figures. Includes corrections for error in legends of figure 7 and figures in appendix B.5 that were present in v3
Journal ref: Journal of Machine Learning Research 26(165):1--54 (2025)
Code: https://github.com/htlambley/functional
License: http://creativecommons.org/licenses/by/4.0/
The gist: Autoencoders have found widespread application in both their original deterministic form and in their variational formulation (VAEs).
Terminology
Abstract
Autoencoders have found widespread application in both their original deterministic form and in their variational formulation (VAEs). In scientific applications and in image processing it is often of interest to consider data that are viewed as functions; while discretisation (of differential equations arising in the sciences) or pixellation (of images) renders problems finite dimensional in practice, conceiving first of algorithms that operate on functions, and only then discretising or pixellating, leads to better algorithms that smoothly operate between resolutions. In this paper function-space versions of the autoencoder (FAE) and variational autoencoder (FVAE) are introduced, analysed, and deployed. Well-definedness of the objective governing VAEs is a subtle issue, particularly in function space, limiting applicability. For the FVAE objective to be well defined requires compatibility of the data distribution with the chosen generative model; this can be achieved, for example, when the data arise from a stochastic differential equation, but is generally restrictive. The FAE objective, on the other hand, is well defined in many situations where FVAE fails to be. Pairing the FVAE and FAE objectives with neural operator architectures that can be evaluated on any mesh enables new applications of autoencoders to inpainting, superresolution, and generative modelling of scientific data.
Sources
- Wasserstein GAN
- Machine Learning for Inverse Problems and Data Assimilation
- Understanding Variational Inference in Function-Space
- Continuum Attention for Neural Operators
- Sampling via Gradient Flows in the Space of Probability Measures
- Regularized KL-Divergence for Well-Defined Function-Space Variational Inference in Bayesian neural networks
- Sobolev Training for Neural Networks
- Towards a Neural Statistician
- From Variational to Deterministic Autoencoders
- Multilevel Diffusion: Infinite Dimensional Score-Based Diffusion Models for Image Generation
- Gaussian Error Linear Units (GELUs)
- Denoising Diffusion Probabilistic Models
- Diffusion Generative Models in Infinite Dimensions
- Adam: A Method for Stochastic Optimization
- Auto-Encoding Variational Bayes
- Neural Operator: Learning Maps Between Function Spaces
- Discretization Error of Fourier Neural Operators
- PointCNN: Convolution On $\mathcal{X}$-Transformed Points
- Fourier Neural Operator for Parametric Partial Differential Equations
- Score-based Diffusion Models in Function Space
Related papers
- Behavior of prediction performance metrics with rare events
- Optimal Estimation of Generic Dynamics by Path-Dependent Neural Jump ODEs
- A Posterior-Dynamics Framework for Imaging Inverse Problems with Pretrained Diffusion Priors
- One Permutation Is All You Need: Fast, Deterministic Feature Importance and Model Stress-Testing
- Online Conformal Prediction for Non-Exchangeable Panel Data
- Deep Time-Series Forecasting in 10 Years: A Survey