A Variational Analysis of Kernel Learning with Learnable Linear Transformations

Summary

The paper A Variational Analysis of Kernel Learning with Learnable Linear Transformations by Yang Li and Feng Ruan studies a generalization of the classical kernel ridge regression problem. In the classical version, one aims to find the minimizer f of the loss function

1/2 E[Y - f(X) 2] + λ/2 f2 H,

where 0 < λ < 1 is a fixed small parameter, and H is a fixed a priori choice of reproducing kernel Hilbert space (RKHS) induced by a kernel function K. The choice of H reflects an a priori guess on the function class for f, and the effectiveness of this method depends crucially on selecting an appropriate kernel. This limitation has motivated kernel learning, in which one searches over a parametrized family of kernels to better adapt to the data distribution.

This paper studies the special case of optimizing among the family of kernels K U(x, x') = K(Ux, Ux'), parametrized by a learnable linear transformation U ∈ End(V). For each U ∈ End(V), the authors consider the functional

I(f, U, λ) = 1/2 E[Y - f(UX) 2] + λ/2 f2 H,

minimized over f ∈ H, and denote the minimal value by J(U, λ) = min f∈H I(f, U, λ). Although kernel ridge regression is linear in f for fixed U, the dependence of J(U, λ) on U is highly nonlinear. The paper's goal is to analyze this dependence, and in particular, how to optimize the choice of U, thereby shedding light on which U are preferred by the objective J, and how they extract latent structure in the data.

A particularly nice case is when the RKHS H is rotationally invariant. In this case, J(U, λ) depends only on the inner product Σ = U T U, and the kernel ridge regression admits an equivalent formulation. One minimizes the loss function

I(F, Σ, λ) = 1/2 E[Y - F(X) 2] + λ/2 F2 H Σ,

where the choice of the Hilbert norm varies with Σ. Then J(U, λ) = J(Σ; λ):= min F∈H Σ I(F, Σ, λ). The authors denote Sym2+ as the space of inner products on V, and refer to the local minimizers Σ of J as vacua in analogy with physics. One of the principal objectives is to understand the landscape of vacua.

The paper presents a sequence of examples highlighting two basic mechanisms: scale adaptation and variable selection. In Example 1.1, a normal variable X N(0, σ2) and Y = sin(σ−1X) illustrate the intrinsic tension between approximation and regularization in a fixed RKHS, motivating the need to learn an extra parameter U. The paper notes that in order to break the approximation-regularization trade-off, we should aim to learn the appropriate scale parameter, or equivalently to learn the appropriate Hilbert norm. Example 1.2 (Two scale problem) and Example 1.3 (Multi-scaled problem) present data with several distinct scales, where no single choice of norm can simultaneously resolve all scales. Example 1.4 (Variable selection and multi-index problem) supposes V = W ⊕ W' is high dimensional while W is low dimensional, and X W' is independent of both X W and Y, so that E[YX] = E[YX W]. One hopes to fit Y by f(UX) where the image of U lands in W, so that U captures the essential degrees of freedom.

The paper addresses two main problems: the static problem of understanding how J(Σ; λ) depends on Σ and the landscape of vacua under a priori structural assumptions, and the dynamical problem of designing a gradient flow on the space Sym2+ of Σ which converges to the vacua of J. The main contribution is a foundational study on the static problem.

Section 2 starts with a quick introduction of the kernel ridge regression problem where the RKHS is defined by a translation invariant kernel. The Hilbert space H is defined in terms of the Fourier transform by

H = f: f2 H = ∫ f̂2(ω)/k V(ω) dω < +∞,

where the function k V is strictly positive and satisfies the L1-integrability condition ∫ k V(ω)dω = 1. Lemma 2.1 (Sobolev embedding) states that for any f ∈ H, f L∞ ≤ f H, and f is continuous, and f → 0 at infinity. Lemma 2.2 states that the functions in the unit ball of H have uniform modulus of continuity.

The paper characterizes the minimizer in terms of an integral equation derived from the Euler-Lagrange equation. Lemma 2.5 (Euler-Lagrange) states that the minimizer f U for I(f, U, λ) satisfies E[(Y - f U(UX))ḡ(UX)] = λ(f U, g) H for all g ∈ H. Plugging in the kernel function K(x, ·) as the test function g, the authors deduce the integral equation

f U(x) = (f U, K(x, ·)) H = 1/λ E[(Y - f U(UX))K(x, UX)].

Proposition 2.6 characterizes the function f U ○ U as the unique finite norm solution to the integral equation F(x) = 1/λ E[(Y - F(X))K(U(x - X))].

The paper proves a quantitative estimate on the modulus of continuity of J(Σ; λ) and the minimizer function. Lemma 2.7 states that for any U, Ũ, there is a uniform bound

E[f U(UX) - f Ũ(ŨX)2] ≤ 1/λ2 E[Y2]E[K(U(X' - X)) - K(Ũ(X' - X))2],

where X' denotes an independent copy of X. Proposition 2.8 (Quantitative continuity of J) states that

J(U, λ) - J(Ũ, λ) ≤ 1/(2λ) E[Y2]E[K(U(X' - X)) - K(Ũ(X' - X))2] 1/2.

Corollary 2.9 states that the minimizer f U depends continuously on U, in the Hilbert space H-norm.

Section 2.5 recalls the representer theorem for finite atomic measures. If X takes value in a finite number of points a1,..., a m ⊂ V, with P(X = a i) = p i, and Y = y i conditional on X = a i, then Proposition 2.10 states that

J(U, λ) = 1/2 ∑ p iy i2 - 1/2 ∑ (λM + P)−1 ij p i y i p j ȳ j,

where M = (M ij) is the m × m inner product matrix with entries M ij = K(U(a i - a j)), and P = diag(p1,..., p m).

Section 2.6 discusses the law of large numbers. Lemma 2.11 shows the almost sure convergence of the empirical I m(f, U, λ) to I(f, U, λ) uniformly in f and U on compact sets. Proposition 2.12 shows that the minimizer f U,m ∈ H (resp. J m(U, λ)) converges uniformly in U to f U ∈ H (resp. J(U, λ)) almost surely.

Section 3 studies the limiting behavior of J(Σ; λ) when Σ tends to infinity in the space of inner products, while remaining bounded on a (possibly empty) proper subspace W ⊂ V. The main theorem is Theorem 3.6, which states that for a sequence of U tending to infinity,

lim J(U, λ) = 1/2 E[Y2 1 X W' ∉ a i] + ∑ J i(U11∞, λ),

where U11∞ is the limit of U11, and J i are the minima of dimensionally reduced problems associated to the atoms a i of the marginal distribution of X W'. Perhaps surprisingly, the answer is sensitive to whether the marginal distribution of X on the quotient space V/W contains any discrete part. If there is no discrete part, for instance if X is a continuous variable, then lim J(Σ; λ) = 1/2 E[Y2], which is the global supremum of J. If there is a nontrivial discrete part, then the kernel ridge regression problem decouples to a number of dimensionally reduced problems in the asymptotic limit, whose minimizers become orthogonal in the RKHS asymptotically.

The proof technique relies on relating the kernel ridge regression problem to an optimal interpolation problem. Lemma 3.1 (Interpretation of H W) states that if f ∈ H, then its restriction to the subspace W ⊂ V lies in H W, and for any g ∈ H W, g H W = min f∈H f H: f W = g. Lemma 3.2 (Optimal interpolation) gives the formula for the minimizer f* of the optimal interpolation problem in Fourier transform. Corollary 3.3 (Asymptotic orthogonality in the optimal interpolation problem) states that if g i H W are uniformly bounded, and min i≠j y i - y j → +∞, then lim inf (f*2 H - ∑ g i2 H W) ≥ 0.

Theorem 3.14 (Approximate minimizer) states that

lim k→+∞ E[(f(m) - f U)(UX)2] + λ f(m) - f U2 H ≤ E[Y2 1 X W' ∉ a i ∞ m+1].

Corollary 3.15 (Asymptotic trivial fitting for the non-atomic part) states that lim E[f U(UX)2 1 X W' ∉ a i ∞1] = 0.

Section 4 focuses on the rotationally invariant kernel case. The main result is the first variation formula of J in Theorem 4.2, which generalizes [12, Lemma 3.6] by relaxing the condition on the kernel function. The theorem states that at any Σ ∈ Sym2+,

D Σ J(Σ, λ) = -1/(2λ) E[r Σ(X, Y)r Σ(X', Y')D Σ K(X - X'2 Σ)]

= -1/(2λ) E[r Σ(X, Y)r Σ(X', Y')K'(X - X'2 Σ)(X - X') ⊗ (X - X')],

where r Σ(X, Y) = Y - F Σ(X), and (X', Y') is an independent copy of (X, Y). The paper provides two arguments under slightly different conditions, one using integral equation techniques, the other using finite measure approximation via the law of large numbers.

Proposition 4.3 (Differentiability in Σ for the minimizer) states that for Σ ∈ Sym2+ and Σ' close to Σ, F Σ' - F Σ - ⟨D Σ F Σ, Σ' - Σ⟩ E = o(Σ' - Σ), where the differential D Σ F Σ is characterized as the unique finite norm solution to the integral equation

f(x) + 1/λ E[f(X)K(X - x2 Σ)] = 1/λ E[r Σ(X, Y)D Σ K(x - X2 Σ)].

Proposition 4.7 discusses when the first variation formula can be continuously extended to the boundary of Sym2≥0. Lemma 4.5 states that the functional J(Σ; λ) extends continuously to the boundary of Sym2≥0, and the minimizer F Σ can be defined for Σ on the boundary, such that F Σ depends continuously on Σ in the · E topology.

Section 4.4 introduces the partial compactification. Definition 4.11 states that the partial compactification Sym2≥0 consists of Sym2≥0 and the points at infinity, where the points at infinity correspond to a pair (W, Σ W), with W ⊂ V a proper subspace of V which is allowed to be zero, and Σ W a positive semidefinite form on W. Proposition 4.13 states that the function J extends to a continuous function on the partial compactification Sym2≥0.

Section 4.5 discusses completely monotone kernels, including Gaussian kernels, Laplace kernels and Sobolev kernels. The celebrated theorem of Schoenberg says that given a radial profile function K, then K(x) = K(x2) defines a RKHS for all dimensions of V, if and only if this Gaussian sum representation formula holds, if and only if the profile function K is a completely monotone function. Example 4.14 gives the Gaussian kernel, Example 4.15 gives the kernel K(x) = 1/(1+x2 V) α, and Example 4.16 gives the Sobolev kernels defined by k V(ω) = k d,γ(ω2) = (4π) d/2 Γ(γ + d/2)/Γ(γ) (1 + 2πω2)-γ-d/2.

Section 5 focuses on the landscape of vacua. Definition 5.1 states that a vacuum in Sym2≥0 is a local minimizer of J in Sym2≥0, and a vacuum at infinity is a point at infinity (W, Σ W), such that for any sequence Σ i ∈ Sym2+ converging to (W, Σ W), whenever i is large enough, then J(Σ i; λ) ≥ J((W, Σ W), λ).

Lemma 5.2 (Necessary condition for boundary vacuum) states that if Σ is a vacuum on S l, then it is a local stationary point of J restricted to the boundary stratum S l, and ⟨D Σ J(Σ), A⟩ ≥ 0 for all positive semi-definite A ∈ Sym2 V*. Lemma 5.3 (Sufficient condition for boundary stratum) states that if Σ is a local minimizer of J restricted to the stratum S l, and ⟨D Σ J(Σ), ω ⊗ ω⟩ > 0 for all ω ≠ 0 ∈ W* ⊂ V*, then Σ is a vacuum.

Lemma 5.4 states that given a direct sum decomposition V = W ⊕ W', so that X = (X W, X W'), if X W is independent of both Y and X W', then for any U ∈ End(V), J(U, λ) ≥ J(Ũ, λ) where Ũ has the U11 entry set to zero. Corollary 5.5 states that if J achieves its global minimum in Sym2≥0, then there is some boundary vacuum Σ ∈ Sym2≥0 achieving this minimum, whose null subspace contains W.

Lemma 5.6 states that the function F Σ(x) descends to a function on W' ≃ V/W, and is the unique C0-solution to the integral equation f(x) = 1/λ E[(Y0 - f(X))K(x - X2 Σ)], where Y0 = E[YX W']. Lemma 5.7 states that if K is a completely monotone kernel function, then the term (43) pairs non-positively with any positive semi-definite form in Sym2 W*. Lemma 5.8 states that if E[X WX W'] and E[X W ⊗ X WX W'] are both independent of X W', and K is a completely monotone kernel function, then the term (42) pairs non-negatively with any positive semi-definite form in Sym2 W*.

Section 5.2 revisits the case where X is a discrete variable taking a finite number of values. Corollary 5.9 (Asymptotic for large Σ) states that if min i,j a i - a j Σ is large, then

J(Σ; λ) = 1/2 ∑ (λp i)/(λ + p i) y i2 - λ/2 ∑ i≠j (p i y i p j ȳ j)/((λ + p i)(λ + p j)) K(a i - a j2 Σ) + O(M - I2).

The limiting value is lim Σ→+∞ J(Σ; λ) = 1/2 ∑ (λp i)/(λ + p i) y i2. The paper notes that in the large Σ regime, the m particles interact via a potential of the form-λ/2 ∑ (p i y i p j ȳ j)/((λ + p i)(λ + p j)) K(a i - a j2 Σ).

Section 6 takes a more operator theoretic approach to the Euler-Lagrange equation. The Euler-Lagrange equation (2.5) is cast in the operator theoretic framework as (λI + T)f U = Y, where T is a bounded self-adjoint operator defined by E[f(UX)ḡ(UX)] = (Tf, g) H, and Y ∈ H is defined by Riesz representation, E[Yḡ(UX)] = (Y, g) H. Lemma 6.1 states that Y2 H = E[YȲ'K(UX, UX')], where (X', Y') is an independent copy of (X, Y). Corollary 6.2 (Decay estimate for the minimizer) states that λf U2 H + E[f U(UX)2] ≤ 1/λ Y2 H = 1/λ E[YȲ'K(UX, UX')], and in particular, if X is a continuous variable, then λf U2 H + E[f U(UX)2] → 0 as U → ∞.

Proposition 6.3 (Operator norm estimate) provides bounds on the operator T, including Tf2 H ≤ f2 C0 E[K(U(X - X'))] ≤ f2 H E[K(U(X - X'))], and the Hilbert-Schmidt norm of T acting on H is T2 HS = E[K(U(X - X'))2]. Lemma 6.4 provides sufficient conditions for the operator norm of T to be small in the case of completely monotone kernels, using bounds on the probability density of the marginal distribution of X on V/W.

Section 6.2 introduces cluster operators. The authors let χ i = 1 X0=i, and define a self-adjoint operator T i by E[χ i f(UX)ḡ(UX)] = (T i f, g) H, inducing the decomposition T = ∑ T i, with each T i describing one 'cluster'. Proposition 6.7 provides bounds on the cluster operators, including T i T j f2 H ≤ E[χ j f(UX)2]E[χ i χ'j K(U(X - X'))2]. Lemma 6.8 states that if K is a completely monotone kernel function and dist Σ(supp(χ i X), supp(χ j X)) ≥ a, then E[χ i χ'j K(X - X'2 Σ)] ≤ K(a2)E[χ i]E[χ j]. Lemma 6.9 provides a similar estimate when the marginal distribution of χ i X on V/W has L∞-bound by p0.

Theorem 6.10 (Non-interaction between far separated clusters) constructs an approximate solution f̃ U = ∑ f i to the Euler-Lagrange equation, where each f i is the solution to the decoupled equation (λI + T i)f i = Y i. The theorem states that the deviation between f U and f̃ U satisfies

λf U - f̃ U2 H + E[Z - (f U - f̃ U)(UX)2] ≤ E[Z2],

where E[Z2] ≤ 1/λ2 E[Y2](∑ 1≤i≠j≤m E[χ i χ'j K(UX, UX')2]). Corollary 6.11 gives the bound

λf U - f̃ U2 H + E[(f U - f̃ U)(UX)2] ≤ 4/λ2 E[Y2] ∑ 1≤i≠j≤m E[χ i χ'j K(UX, UX')2].

Theorem 6.12 (Minimum value) states that

J(U, λ) - ∑ J i ≤ 1/λ2 E[Y2] ∑ 1≤i≠j≤m E[χ i χ'j K(UX, UX')2],

and

J(U, λ) - ∑ J i ≥ -2/λ2 E[Y2] ∑ 1≤i≠j≤m E[χ i χ'j K(UX, UX')2].

Section 6.4 discusses the dimensional reduction mechanism. Lemma 6.13 gives the Hilbert-Schmidt norm of T - T̃, and Lemma 6.14 bounds Y - Ỹ2 H. Lemma 6.15 states that if K(x) = K(x2 V) and Σ = U T U, then

λf U12 H + E[f U1(UX)2] ≤ 1/λ E[Y12] × E[K(X - X'2 Σ) - K(X̃ - X'2 Σ) - K(X - X̃'2 Σ) + K(X̃ - X̃'2 Σ)2] 1/2.

Section 7 formulates the notion of 'scale detectors'. Definition 7.1 defines the Λ-neighbourhood of Σ as N(Σ, Λ):= Σ' ∈ Sym2+: Λ−1Σ ≤ Σ' ≤ ΛΣ. Definition 7.2 states that a vacuum Σ ∈ Sym2+ is called an (ϵ, Λ)-scale detector, if it has the local minimizing property J(Σ', λ) ≥ J(Σ, λ) for all Σ' ∈ N(Σ, Λ), and the strict gap property J(Σ', λ) ≥ J(Σ, λ) + ϵEY2 for all Σ' ∈ ∂N(Σ, Λ). Definition 7.3 and 7.4 extend this to boundary vacua.

Lemma 7.5 states that any (ϵ, Λ)-scale detector Σ must be bounded below by (ϵλ)/(2βK'C0) ·2 V, where β = sup ω∈V*, ω V=1 E[⟨X, ω⟩4] 1/2. Lemma 7.6 states that under quantitative bounds on the probability density of X, any (ϵ, Λ)-scale detector Σ in Sym2≥0 is bounded above by Σ ≤ (C K p0)2(λϵ)−4 ·2 V.

Theorem 7.7 states that under the essential dependence assumption, and the hypotheses of Lemma 7.5 and 7.6, any global minimizing vacuum Σ0 lies within the neighbourhood N(·2 V, Λ), and such Σ0 is an (ϵ, Λ2)-scale detector.

Theorem 7.8 (Multiple vacua existence) states that in the one-dimensional multi-scale setting, there are a small constant ϵ and a large constant Λ, such that if σ i/σ i+1 is sufficiently small for all i, and min i≠j a i - a j ≥ C−1σ m, then:

1. Any (ϵ, Λ)-scale detector Σ satisfies Λ−1σ i−2 < Σ = U2 < Λσ i−2 for some i ∈ 1,..., m.

2. For each i = 1,..., m, there is an (ϵ, Λ2)-scale detector Σ with Λ−1σ i−2 < Σ = U2 < Λσ i−2.

The proof of Theorem 7.8 proceeds in four steps: replacing the small scale clusters by atoms, decoupling the bigger scale clusters, analyzing the simpler kernel ridge regression problem, and providing a quantitative version.

The paper concludes with an appendix (Appendix A) that discusses the connections between the kernel learning problem and representation learning in two-layer neural networks. The paper notes that the kernel learning problem studied in this paper may be viewed as a tractable analogue of a two-layer neural network, where U plays a role analogous to the collection of learned first-layer weights, while f plays the role of a learned second layer. The paper emphasizes that "the present work provides a concrete example in which favorable local minimizers of a two-layer model induce representations encoding meaningful 'structural information' of the data, including intrinsic scales, predictive low-dimensional structure, and clustering structure in the underlying data distribution—phenomena that are rigorously characterized in the present paper."

Based on the variational analysis in this paper, I can identify several specific improvements to AI systems, particularly in kernel-based learning, feature selection, and multi-scale data modeling.

1. Adaptive Kernel Learning for Multi-Scale Data

• Improvement: Replace fixed-kernel ridge regression with a learnable linear transformation U (or equivalently, a learnable inner product = U T U) that is optimized via the first-variation formula (Theorem 4.2) and the continuity properties (Proposition 2.8).

• What the improved system can do: Automatically detect and adapt to multiple inherent scales in data (e.g., signals with both low-frequency and high-frequency components, or clusters with vastly different variances). The system will no longer suffer from the approximation-regularization trade-off described in Example 1.1; instead, it will learn a scale parameter that matches the data's intrinsic scale, leading to better prediction accuracy with lower regularization error.

2. Automatic Variable Selection and Dimensionality Reduction

• Improvement: Use the boundary vacua analysis (Section 5.1) and the first-variation test at boundary points (Lemma 5.2, 5.3) to identify and prune irrelevant input dimensions. Specifically, the system can evaluate whether a candidate low-rank (with null space W) is a local minimizer by checking the sign of the directional derivative along omega omega for omega in W*.

• What the improved system can do: In high-dimensional settings (e.g., d = 10 4), the system can automatically determine which features are essential for predicting Y (as in the multi-index model, Example 1.4). It will converge to a low-rank representation, effectively performing variable selection without explicit regularization penalties (like L1), and will be robust to irrelevant features that are independent of Y.

3. Cluster-Aware Learning and Decoupling

• Improvement: Leverage the cluster interaction estimates (Proposition 6.3, Lemma 6.8, 6.9) and the decoupling theorem (Theorem 6.10, 6.12) to decompose the learning problem into independent subproblems when data exhibits distinct clusters (e.g., different classes or regimes).

• What the improved system can do: For datasets with natural groupings (e.g., different experimental conditions, patient subgroups, or sensor modes), the system will automatically learn separate kernel parameters for each cluster, leading to faster convergence and better generalization. It will also provide a quantitative measure of when clusters are far enough apart to be treated independently, preventing unnecessary interference.

4. Robustness to Degenerate and Divergent Transformations

• Improvement: Use the asymptotic analysis (Theorem 3.6) and the partial compactification (Section 4.4) to handle cases where U becomes singular (rank-deficient) or diverges to infinity.

• What the improved system can do: The system will gracefully handle situations where some features are constant or where the data distribution is nearly atomic. It will correctly identify when a degenerate transformation is optimal (e.g., when Y depends only on a few features) and when a divergent transformation is optimal (e.g., when data is discrete with well-separated atoms). This prevents numerical instability and ensures the optimizer does not get stuck in poor local minima.

5. Improved Optimization Dynamics via Gradient Flow

• Improvement: Use the first-variation formula (Theorem 4.2) to design a gradient flow on the space of inner products, as outlined in the companion paper [28]. The flow can be equipped with a canonical Riemannian metric derived from the variational structure.

• What the improved system can do: The system will have a principled optimization algorithm that provably converges to stationary points (vacua) of the kernel learning objective. It will be able to escape shallow local minima (which are not scale detectors) and converge to meaningful representations that capture the intrinsic scales and feature subspaces, as demonstrated in Theorem 7.8 for multi-scale problems.

6. Quantified Uncertainty and Stability of Learned Representations

• Improvement: Use the quantitative continuity estimates (Lemma 2.7, Proposition 2.8) to provide error bars on the learned U and on the prediction function f U.

• What the improved system can do: The system can report a confidence interval for the learned scale parameters and feature importance, based on the modulus of continuity of J with respect to U. This is crucial for high-stakes applications (e.g., medical diagnosis, financial forecasting) where knowing the uncertainty of the model's internal representation is as important as the prediction itself.

7. Handling Non-Atomic vs. Discrete Distributions

• Improvement: Use the sharp distinction in asymptotic behavior (Theorem 3.6) to automatically detect whether the data distribution has a discrete component (atoms) or is purely continuous.

• What the improved system can do: The system will automatically switch between two regimes: (a) for continuous data, it will avoid overfitting to noise by not letting U diverge to infinity; (b) for discrete data with atoms, it will exploit the decoupling into lower-dimensional problems, leading to better performance on tasks like classification with well-separated classes.

8. Multi-Index Model Recovery

• Improvement: Use the variable selection mechanism (Section 5.1.1) and the first-variation test to recover the true low-dimensional subspace W in multi-index models (Example 1.4).

• What the improved system can do: In regression problems where Y depends only on a few linear combinations of X, the system will recover the exact subspace W (up to estimable error), even when the number of irrelevant features is large. This is a significant improvement over standard kernel methods that use all features and suffer from the curse of dimensionality.

9. Improved Generalization with Small Regularization

• Improvement: Use the asymptotic orthogonality and decoupling results (Theorem 3.14, Corollary 6.11) to show that the learned representation U leads to better generalization bounds.

• What the improved system can do: With a small regularization parameter lambda, the system will achieve lower test error on unseen data, because the learned U effectively reduces the problem to a lower-dimensional one, reducing the effective VC dimension and improving sample complexity.

10. Theoretical Guarantees for Feature Learning

• Improvement: Provide a rigorous mathematical foundation for why feature learning works in kernel models, as opposed to fixed kernels.

• What the improved system can do: The system will not just be a black-box optimizer; it will provide certificates that the learned representation U is a local minimizer (vacuum) of the population objective, and that it corresponds to a scale detector or variable selector with a quantitative energy gap. This is a step towards making AI systems more interpretable and trustworthy.

The classical kernel ridge regression problem aims to find the best fit for the output Y as a function of the input data X in R d, with a fixed choice of regularization term imposed by a given choice of a reproducing kernel Hilbert space, such as a Sobolev space. Here we consider a generalization of the kernel ridge regression problem, by introducing an extra matrix parameter U, which aims to detect the scale parameters and the feature variables in the data, and thereby improve the efficiency of kernel ridge regression. This naturally leads to a nonlinear variational problem to optimize the choice of U. We study various foundational mathematical aspects of this variational problem, including its Euler-Lagrange equation, continuity and first variation, limiting behavior under degenerate or diverging transformations, and the structure of its local minimizers. Particular attention is given to two data-distribution settings, namely multi-scale and multi-index models, where the learned transformation U encodes intrinsic scale parameters and the essential low-dimensional feature variables, respectively.