Nonlinear Dimensionality Reduction Techniques for Bayesian Optimization
math.OC, cs.LG, cs.NA, math.NA
Submitted: 2025-10-17
Updated: 2026-09-11
Comments: Substantial revision from the first version
Code: https://github.com/L-Lok/Nonlinear-Dimensionality-Reduction-Techniques-for-Bayesian-Optimization
Project page: https://lindonroberts.github.io
License: http://creativecommons.org/licenses/by/4.0/
The gist: Bayesian optimisation (BO) enables sample-efficient global optimisation of expensive black-box functions but remains challenging in high dimensions.
Terminology
Abstract
Bayesian optimisation (BO) enables sample-efficient global optimisation of expensive black-box functions but remains challenging in high dimensions. We investigate nonlinear dimensionality reduction to a sequence of low-dimensional latent-space BO (LSBO) problems. Early LSBO used linear random and supervised embeddings; building on Grosnit et al., we employ variational autoencoders (VAEs), deep metric loss for structured latent manifolds, and retraining to adapt the encoder-decoder pair to newly sampled regions. We couple LSBO with sequential domain reduction (SDR) directly in latent space (SDR-LSBO), narrowing search domains as evidence accumulates. Implemented in GPU-accelerated BoTorch with Matérn-5/2 Gaussian-process surrogates, our methods improve benchmark optimisation quality, and retraining can enhance BO performance. Comparisons with adaptive supervised linear random embeddings demonstrate the effectiveness of VAE-based BO for nonlinear low-dimensional structures. We analyse BO-VAE with a fixed pretrained representation, decomposing ambient-space simple regret into latent BO error and a fixed VAE-induced representation gap. Under a PAC-Bayes-certified reconstruction condition and standard fixed-prior assumptions for expected improvement with a Matérn-5/2 kernel, latent BO error vanishes as the evaluation budget increases, whereas the representation gap remains fixed and may impose a non-vanishing error floor. Visualisations empirically assess accessibility of the ambient optimum through the learned decoder. To our knowledge, this is the first study combining SDR with VAE-based LSBO. Our analysis clarifies metric shaping and retraining choices critical for scalable latent-space BO. For reproducibility, source code is available at https://github.com/L-Lok/Nonlinear-Dimensionality-Reduction-Techniques-for-Bayesian-Optimization.git.
Sources
- High-Dimensional Bayesian Optimisation with Variational Autoencoders and Deep Metric Learning
- A Tutorial on Bayesian Optimization
- Vanilla Bayesian Optimization Performs Great in High Dimensions
- Auto-Encoding Variational Bayes
- Convergence of Gaussian Process Regression with Estimated Hyper-parameters and Applications in Bayesian Inverse Problems
- Grammar Variational Autoencoder
- Active Learning of Linear Embeddings for Gaussian Processes
- Global optimization using random embeddings
- Good practices for Bayesian Optimization of high dimensional structured spaces
- Dimensionality Reduction Techniques for Global Bayesian Optimisation
- Tutorial on Variational Autoencoders
- Adam: A Method for Stochastic Optimization
- Generating Sentences from a Continuous Space
- TVAE: Triplet-Based Variational Autoencoder using Metric Learning
- A dimensionality reduction technique for unconstrained global optimization of functions with low effective dimensionality
Related papers
- Lions and Muons: Optimization via Stochastic Frank-Wolfe under Heavy-Tailed Noise
- Adam-HNAG: A Convergent Reformulation of Adam with Accelerated Rate
- Incremental Learning in Mirror Flows
- Online Control via Counterfactual Tracking
- Asynchronous Replanning in Two Population Linear Quadratic Mean Field Games: Information Requirements and Stability
- Petrov-Galerkin operator inference with application to stability-encouraging identification