beta-VAEs as Effective Theories: Tolerance-Dependent Dimension

In a beta-VAE, increasing the regularization strength acts as a spectral cutoff by collapsing low-utility latent coordinates. In the linear Gaussian VAE, the collapse order matches the ranking of reconstruction utilities exactly, because both are set by the PCA spectrum. We ask which parts of this picture survive in fully connected nonlinear VAEs trained on WorldClim. We find that nonlinear interactions shift and broaden collapse onsets, so thresholds no longer coincide exactly with utilities. However, the common ordering is preserved over the resolved ranks, so the spectral cutoff still acts as a utility cutoff and the effective-description logic carries through. The resulting effective-dimension curves reveal a head–tail tradeoff: increasing depth concentrates utility into the first few coordinates but worsens tail fidelity.

The paper fixes reconstruction as the task, reconstruction distortion as the metric, and fully connected beta-VAEs as the model class, with depth as the main architectural variable. With task, metric, and model class fixed, tolerance becomes the main variable, and the paper asks how effective dimension changes as the allowed reconstruction error is varied. This suggests an effective-theory viewpoint: choosing a reconstruction tolerance amounts to choosing a resolution, so that some variation is represented explicitly while the rest is left unresolved in the residual term. Unlike in field theory, the residual is not absorbed into counterterms to keep predictions cutoff-independent; it is allowed to increase as the cutoff is raised. In a VAE, the KL weight beta plays the role of a tunable cutoff over latent coordinates. The measured object is the rank–distortion curve: reading it at fixed tolerance epsilon gives the effective dimension deff (epsilon), while the associated marginal utilities give the ordered spectrum of reconstruction importance.

The linear Gaussian analysis in Ref. [15] shows that this selection picture is exact in the linear VAE case. As T is raised, latent coordinates collapse one by one. Each collapse threshold equals both the marginal reconstruction utility of that coordinate and the corresponding PCA explained-variance ratio. The collapse spectrum, PCA spectrum, and pruning-utility spectrum therefore coincide component by component. The paper asks which parts of this exactly calibrated picture survive in nonlinear VAEs, testing on the WorldClim bioclimatic dataset using rank–distortion curves to ask how many variables are needed to reach a given normalized reconstruction error.

The nonlinear scan uses the same normalized control parameter as the linear theory: T = betasigma dec2 / V = beta / (V/sigma dec2), where V is the total data variance. Thus T is the information price in normalized distortion units. The KL regularization acts spectrally: coordinates collapse one by one as this price is raised, rather than being uniformly shrunk. For each latent coordinate k, the paper monitors the posterior mean-square amplitude, posterior variance, posterior log-variance, and rate. The primary order parameter is the scale-invariant signal fraction Mk2(T) = muk(x)2 / (muk(x)2 + sigmak(x)2), which measures the signal fraction of the aggregate coordinate without assuming a fixed latent scale. A coordinate is classified as active when Mk2 > 0.1. For squared-error reconstruction, normalized distortion is reported as D̃ K = ‖x − x̂(K)‖2 / V, where x̂(K) is the reconstruction obtained after retaining the first K ranked latent coordinates and pruning the rest. The marginal utility of rank K is ΔD̃ K = D̃ K−1 − D̃ K.

The linear Gaussian VAE provides the calibrated baseline. The reconstruction geometry is quadratic and diagonal in the PCA basis, so each eigendirection behaves independently, and activating one mode does not change the utility of another. The signal-fraction branch follows the one-mode form Mk2(T) = [1 − T/Tk]+, and the collapse threshold, reconstruction utility, and PCA weight coincide: Tk = ΔD̃ k = lambdak/V. The linear result is a calibrated null model: the equilibrium beta scan resolves the PCA spectrum mode by mode. The nonlinear experiments keep this measurement protocol and ask which parts of this calibration remain once the decoder can mix modes. Such departures are called nonlinear spectral deformations: collapse thresholds can shift relative to PCA, the coordinate order can change, leading modes can absorb a larger fraction of the total distortion reduction, and onset branches can soften into crossovers when a new coordinate activates in the background created by already-active coordinates.

The paper distinguishes a local instability from a finite reconstruction utility. Let A denote the set of already-active coordinates, and let qk be a small activity amplitude for a collapsed coordinate k. Locally, the objective can be expanded as L A+k = L A + ½[T − gk(A, T)]qk2 + O(qk4), where gk(A, T) is the local reconstruction gain available to coordinate k in the background of the active set. The infinitesimal onset is controlled by T = gk(A, T). The finite utility measured by truncation is instead ΔD̃ k(A) = D̃(A) − D̃(A ∪ k). In the linear Gaussian case these two notions coincide because the quadratic reconstruction operator is global and diagonal in the PCA basis: gk(A, T) = ΔD̃ k(A) = ΔD̃ k. A nonlinear representation has no reason to obey this additivity; the relevant quadratic operator is local to the current active background and can change along the scan, with gk(A, T) = gk(0) + ∑ l∈A J kl(T) + ∑ l,m∈A J klm(T) + ⋯.

The main empirical object is the ranked utility spectrum for WorldClim. Unless noted otherwise, the single-architecture scans use a representative fully connected VAE with four hidden layers of width 64 in both encoder and decoder, and 32 latent coordinates. The depth comparison keeps the width fixed at 64 and the latent dimension fixed at 32 while varying depth over 2, 4, 8, and 16 hidden layers. Fig. 1 shows the signal-fraction scan used to rank coordinates: the first coordinates activate in a clear sequence as T is lowered, but the branches do not appear to follow exact one-mode Landau curves. The paper uses Mk2 primarily as a scale-invariant ranking observable, not as the main source of fitted thresholds.

Fig. 2 measures utilities directly by pruning: for each point along the scan, only the first K ranked latents are retained and the rest pruned. The left panel follows the resulting normalized distortion as a function of T; the right panel records the best distortion reached at each retained rank. The marginal drops in this curve define ΔD̃ K. This pruning measurement is a property of the trained representation, not a comparison with separately retrained K-latent models. Fig. 3 shows a rescaled overlay for the information rate Rk(T) using the utilities from Fig. 2, with the black curve being the one-mode Landau prediction with no fitted threshold. The overlay checks agreement between the order-parameter ranking and the rate ranking. There is no parameter fitting involved in Fig. 3: the point is that a utility measured from reconstruction predicts the rate scale of the active branch.

For each rank, the paper fits the unrescaled rate branch to the fixed-slope form Rk(T) ≃ ½[−log(T/T c,k rate)]+ over the window 10−2 ΔD̃ k ≤ T ≤ ΔD̃ k. Fig. 4 compares these fitted thresholds with the directly measured utilities. The diagonal agreement is summarized by an identity R2 on log-scaled values, log ΔD̃ k versus log T c,k rate. On the test split, the check gives R2 ≃ 0.98 over the plotted ranks. For the representative WorldClim run, the threshold ranking agrees with the utility ranking through the first thirteen modes; beyond that point, neighboring fitted thresholds begin to swap while the pruning utilities remain clearly separated.

A compact sparsity diagnostic is the required rank at a prescribed normalized distortion. Fig. 5 shows the measured rank–distortion curves; lower curves are more spectrally sparse. The plotted comparison fixes the hidden width and varies depth. The first few ranks form a depth-sensitive head: increasing depth can concentrate more utility into the leading effective variables. Beyond rank 8, however, the deepest models often saturate at a higher residual floor. Reading the same curves at fixed tolerance gives the effective dimension, deff(epsilon) = min K ∶ D(K)/D0 ≤ epsilon. Table I reports this fixed-tolerance read-off from 5% down to 10−5. For the PCA baseline, the required ranks are 6, 9, 10, 12, 15, and 17 for targets 5%, 1%, 0.5%, 0.1%, 0.01%, and 0.001%, respectively. For the VAE, the lowest ranks among the compared architectures are 2, 5, 5, 8, 14, and a dash (target not reached) for the same targets.

Fig. 6 suggests that the fall-off can be described locally by a power law. On log–log axes, the head–tail separation appears as a crossover near Kc ≃ 4: the first few ranks form the head, and subsequent ranks follow the residual tail until architecture-dependent floors appear. For WorldClim, the residual distortion falls approximately as D(K) ∼ K−4 over ranks 4–17. The corresponding spectral exponent is closer to 5 since the marginal utility spectrum ΔD̃ K is the discrete derivative of D(K) and is therefore roughly one power steeper.

The paper concludes that for WorldClim, the nonlinear scan confirms the cutoff picture, but direct reconstruction utilities are more robust than fitted collapse thresholds. Utilities are therefore the preferred spectrum: they directly measure the feature importance of each ranked latent coordinate, remain usable at higher ranks, and avoid the fit-window and outlier sensitivity of threshold extraction. Thresholds serve mainly as a check of the utility–threshold duality. This changes the practical role of the scan: the scan is the calibration step, not the end goal. Once the spectral-cutoff picture has been checked for a given model class and normalization, one need not extract thresholds or run a full scan on every dataset. Instead, one can choose the normalized regularization strength to match the smallest reconstruction utility one is willing to trust, for example a noise floor or modeling tolerance.

The main output is not a single preferred bottleneck size, but a tolerance-dependent effective dimension. Effective dimension is therefore not a fixed property of the data alone; it depends on the task, distortion metric, model class, and tolerated reconstruction error. Architecture changes the shape of that dependence: at fixed width, depth mainly redistributes utility toward the head of the spectrum, often at the cost of a higher tail floor. This is a latent sparsity tradeoff: deeper models concentrate more reconstruction value into the first few coordinates, but can leave a less efficient residual tail.

These results support an effective-theory reading of the beta-VAE scan. The value of T specifies the price of resolving latent information; the trained model then decides which learned variables remain explicit and which variation is absorbed into the residual distribution. The scan does not assume an effective description in advance; it reveals one by showing which variables survive at the chosen cutoff. This selection mechanism also clarifies what distinguishes VAEs from deterministic and sparse autoencoders. VAEs introduce probabilistic latent degrees of freedom, average over them in the objective, and penalize their departure from a prior. A deterministic autoencoder can learn a low-dimensional bottleneck, but the bottleneck dimension is fixed by design. Sparse autoencoders impose sparsity at the level of individual codes, with each input activating only a small subset of dictionary elements, and the active subset changing from one data point to the next; this does not by itself produce a small globally ranked set of coordinates shared across the dataset.

In the VAE, this cutoff picture comes from a prior-relative rate cost for making an entire coordinate input-dependent across the dataset. A coordinate either earns its KL cost or collapses toward the prior globally. Rather than asking whether a system is emergent in the abstract, one can ask how many effective variables are required to reproduce the behavior of interest at a given tolerance. Because the present work ranks variables by reconstruction utility, it can retain redundant or nuisance variation whenever that variation helps reproduce the input distribution. A natural next step is to embed the same variational bottleneck inside a supervised predictor and rank variables by task utility, since the bottleneck would then lie on the prediction path, and pruning would measure the importance of variables for the model actually used for inference. In that setting, choosing beta becomes an operating-point question: one would trade off task performance, active dimension, and the interpretability of the surviving coordinates.

The WorldClim experiments use the 19 bioclimatic variables from WorldClim at 10 arc-minute resolution. The data split uses seed 0 and a spatial-block protocol with equal-area blocks of side length 500 km, area-weighted sampling, a target of 500,000 training samples, and an equal split of the non-training blocks between validation and test. The representative nonlinear run uses fully connected networks with four hidden layers, 64 hidden units per layer, and 32 latent coordinates, with SiLU activations, Kaiming initialization, and initial log-variance bias −4. The reconstruction loss is mean-squared error with fixed decoder variance convention sigma dec2 = 1. The scan evaluates an independent grid of normalized temperatures, with T running from 1 to 10−6. Each value of T is trained from a fresh initialization rather than warm-started from neighboring scan points, so the ordering of the grid is not an annealing path and does not introduce scan hysteresis. In the saved resolved configuration this corresponds to 121 scan points and raw beta values from 19.0 to 1.9 × 10−5, because the measured WorldClim variance scale is V = 19. Each scan point is trained with Adam at learning rate 3 × 10−4, with a maximum budget of 106 optimizer updates, relative tolerance 10−3, and patience fraction 0.01. Latent coordinates are ranked by the saved SNR-like score, equivalently by the signal fraction Mk2, using descending score order. The active threshold used for scan diagnostics is 0.1. Ranked-pruning curves retain the first K coordinates in this order and prune the rest.

Improvements to AI Systems Based on This Paper:

1. Tolerance-Aware Adaptive Bottlenecking: Implement a dynamic latent dimensionality selector that adjusts the number of active latent variables based on a user-specified reconstruction tolerance (ε), rather than fixing the bottleneck size a priori. The system would read off the effective dimension deff(ε) from a pre-computed rank–distortion curve, enabling automatic compression that trades fidelity against computational cost in real time.

2. Utility-Based Pruning for Inference: Replace arbitrary or heuristic latent pruning with a principled ranking by marginal reconstruction utility (ΔD̃ K). After training, the system can prune low-utility coordinates at inference time without retraining, directly using the measured utility spectrum to decide which latents to keep—improving efficiency while guaranteeing a bounded distortion increase.

3. Nonlinear Spectral Deformation Detection: Add a diagnostic module that compares fitted collapse thresholds (T c,k rate) against directly measured utilities (ΔD̃ k) to detect when nonlinear interactions break the linear VAE calibration. This allows the system to flag when a learned representation has shifted, broadened, or reordered its spectral cutoffs, preventing overconfident use of PCA-like assumptions in complex data.

4. Depth-Aware Architecture Selection: Use the head–tail tradeoff finding to guide model design: for tasks requiring high precision on few dominant features, choose deeper networks (which concentrate utility into leading coordinates); for tasks requiring uniform fidelity across many features, prefer shallower networks (which maintain a lower tail floor). The system can recommend depth based on the target distortion profile.

5. Threshold-Free Spectral Ranking: Adopt the paper’s conclusion that direct reconstruction utilities are more robust than fitted thresholds. Implement a pruning-based utility measurement as the primary ranking mechanism for latent coordinates, avoiding fit-window sensitivity and outlier issues. This yields more stable feature importance scores for interpretability and downstream tasks.

6. Calibrated Scan as a One-Time Setup: Once a spectral-cutoff check is validated for a given model class and normalization, skip full β-scans on new datasets. Instead, set the regularization strength T directly to match the smallest acceptable utility (e.g., noise floor), reducing training cost while maintaining the same effective-description logic.

7. Task-Utility Embedding for Supervised Learning: Extend the variational bottleneck to supervised predictors by placing it on the prediction path and ranking latent coordinates by task utility (not reconstruction utility). This enables automatic selection of task-relevant features, with β as an operating-point knob trading off task performance, active dimension, and interpretability—useful for model compression and explanation.

8. Power-Law Tail Modeling for Extrapolation: Use the observed power-law decay (D(K) K−4 for WorldClim) to predict distortion at untested ranks or to estimate the spectral exponent from marginal utilities. This allows the system to extrapolate effective dimension to stricter tolerances without additional training, saving compute.

9. Hysteresis-Free Scan Protocol: Adopt the paper’s practice of training each β value from fresh initialization (not warm-started) to avoid scan hysteresis. This ensures the measured rank–distortion curves reflect true equilibrium behavior, improving reproducibility and reliability of spectral diagnostics.

10. Scale-Invariant Activity Classification: Use the signal fraction M k2(T) = μ k2/(μ k2 + σ k2) as the activity criterion instead of raw amplitude or variance. This makes the system’s latent activity detection invariant to arbitrary latent scale choices, improving robustness across different architectures and initializations.

What the Improved AI System Can Do:

• Automatically determine the minimal number of latent variables needed to meet a user-specified reconstruction error, without retraining.

• Prune redundant latent coordinates at inference time with a guaranteed distortion bound, reducing memory and compute.

• Detect when its learned representation deviates from linear (PCA-like) assumptions and adapt its interpretation accordingly.

• Recommend network depth based on the desired tradeoff between head concentration and tail fidelity.

• Provide stable, threshold-free feature importance rankings for interpretability.

• Reuse a single calibration scan across multiple datasets of the same type, saving significant training time.

• In supervised settings, select task-relevant variables dynamically, balancing performance and interpretability.

• Predict performance at stricter tolerances via power-law extrapolation, enabling forward planning without extra compute.

• Produce reproducible spectral analyses free from scan-order artifacts.

In a beta-VAE, increasing the regularization strength acts as a spectral cutoff by collapsing low-utility latent coordinates. In the linear Gaussian VAE, the collapse order matches the ranking of reconstruction utilities exactly, because both are set by the PCA spectrum. We ask which parts of this picture survive in fully connected nonlinear VAEs trained on WorldClim. We find that nonlinear interactions shift and broaden collapse onsets, so thresholds no longer coincide exactly with utilities. However, the common ordering is preserved over the resolved ranks, so the spectral cutoff still acts as a utility cutoff and the effective-description logic carries through. The resulting effective-dimension curves reveal a head--tail tradeoff: increasing depth concentrates utility into the first few coordinates but worsens tail fidelity.