Minimax Additive Regression under Unknown Dependent Designs

Minimax Additive Regression under Unknown Dependent Designs investigates how to estimate additive models when both the design and the regression function are unknown, even when they depend on a potentially non-product random design. This work establishes minimax optimal rates for prediction in this complex setting by introducing coupled smoothness classes and adapting Riesz-basis constructions to handle dependence, allowing for dimension growth with sample size.

The gist

Minimax optimal rates for prediction in additive regression under unknown dependent designs are established by constructing thresholded least-squares estimators that recover the centered additive components at the same aggregate upper rate, depending on whether marginal densities are known or unknown.

Key Theoretical Framework and Representation

The analysis begins by defining the model as a random-design nonparametric regression:

1. The model is given by (1):

2. The true function is additive: f⋆(x) = f⋆0 + Xd j=1 f⋆j(xj), x ∈ [0, 1]d, where EP [f⋆j(Xj)] = 0 for identifiability.

3. The analysis separates the problem into two cases: known and unknown marginal densities.

To handle the non-orthogonal geometry induced by a general distribution P, the authors build on Ferrere et al. (2026) to introduce a distribution-adapted Riesz basis (Theorem 2.6). This representation decomposes each additive component as:

)& f⋆j = X+∞ m=1 θ(j)m ψ(j)m, where ψ(j)m = ϕm(x)/pj(x). The coefficients θ are the Fourier coefficients of the weighted components. This allows for the definition of regularity classes based on coefficient constraints (Definition 2.7).

Regularity Classes and Dimension Growth

The paper introduces a coupled parameter class, Cβ,γ = [p∈Pγ,L × Fβ,R(p)], where:

1. β controls the spectral regularity of the signal in the Riesz representation (Sobolev constraint on coefficients).

2. γ controls the regularity of the marginal densities pj (Hölder regularity).

3. The dimension d is allowed to grow with n at a rate satisfying Assumption 2.13: d = o n(2β/(2β+1)) log n.

Minimax Rates and Bounds

The paper establishes minimax optimal upper and lower bounds under suitable dimension-growth conditions:

)& Known Density Regime:

)& When marginal densities are known, thresholded least squares attains the classical additive minimax rate, with linear dependence on d: M(n, d, F) ≲ d n(-2β/(2β+1)).

)& Unknown Density Regime:

)& When marginal densities are unknown, an estimator based on sample splitting and thresholded least squares in an estimated dictionary is introduced. This estimator recovers the centered additive components at the same aggregate upper rate: M(n, d, Cβ,γ) ≲ d n(-2β/(2β+1)) + d n(-2γ/(2γ+1)).

Component Recovery and Lower Bounds

The paper demonstrates that while the estimators estimate components centered with respect to estimated marginal weights (Definition 4.5), the true components can be recovered:

)& Recovering True Components:

)& The debiased estimator ¯fj,M consistently estimates the true component at a rate of M(n, d, F) ≲ d n(-2β/(2β+1)) + d n(-2γ/(2γ+1)).

The minimax lower bound analysis distinguishes between two regimes:

)& Smooth Regime (γ ≥ β):

**)& The lower bound achieves the rate M(n, d, Cβ,γ) ≳ d n(-2β/(2β+1)), leading to minimax optimality when γ ≥ β. **

)& Rough Regime (γ < β):

**)& Assuming a dimension growth rate d = o n(2γ/(2γ+1)), the lower bound is M(n, d, Cβ,γ) ≳ d n(-2γ/(2γ+1)), leading to minimax optimality. **

Density Estimation Guarantee

The construction of marginal density estimators relies on piecewise polynomial projection followed by clipping (Lemma G.6). This yields a deterministic bound on the pointwise risk:

)& Uniform Pointwise Density Risk:

**)& The risk satisfies sup t∈[0,1] Ep[pej(t)] − pj(t)2 ≤ (s + 2)L!/(2γ - γ + κmax(s + 1)2 J n0.

Based on the provided scientific paper, here are specific improvements that could be made to AI systems by applying its theoretical framework, and what those improved systems could achieve:

)System Improvements Derived from the Paper

The core contribution of this paper is providing rigorous minimax optimal rates for additive regression under complex, dependent random designs where the dimension can grow with sample size. This framework moves beyond traditional curse of dimensionality limitations by coupling signal smoothness with design distribution regularity.

Here are specific improvements and capabilities:

AI System: Additive Regression Models on Non-Product Designs (e.g., complex sensor arrays, time-series data).

2. Capability: Estimation of high-dimensional regression functions where the covariates are statistically dependent but the underlying model is additive (e.g., modeling how multiple correlated features influence an output).

AI System: Robust Nonparametric Modeling with Unknown Covariate Distributions (e.g., medical imaging, climate science).

4. Capability: Developing estimators that perform optimally even when the joint probability distribution of the input data is completely unknown, provided only uniform bounds on its density exist (Assumption 2.3). This allows for real-world applications where precise distributional knowledge is impossible to obtain.

AI System: Adaptive Dictionary Learning and Model Selection (e.g., feature engineering, sparse neural networks).

6. Capability: Implementing thresholded least-squares estimators that automatically select the most relevant additive components from a large dictionary of features (the Riesz basis representation), achieving optimal prediction error while simultaneously estimating the underlying signal's smoothness.

AI System: Uncertainty Quantification in High-Dimensional Additive Models (e.g., reinforcement learning, Bayesian neural networks).

8. Capability: Providing statistically rigorous confidence intervals and prediction bounds that correctly account for both estimation uncertainty (bias/variance) and the uncertainty arising from estimating the marginal distribution of the input features. This is crucial for safety-critical applications where knowing how much you don't know is as important as a point estimate.

AI System: Dimension-Growing Predictive Modeling (e.g., deep learning architectures with dynamically growing layers).

10. Capability: Designing neural network structures or functional ANOVA models that can handle scenarios where the number of input features grows slowly relative to the sample size, maintaining optimal predictive accuracy without requiring the dimension to be fixed beforehand.

AI System: Sparse Interaction Modeling (e.g., interpreting complex interaction effects in biological systems).

12. Capability: Recovering individual additive effects even when covariates are highly dependent, by separating signal regularity from design density irregularity using the coupled smoothness class framework. This allows AI to correctly attribute variance to specific input features rather than misattributing it to correlated neighbors.

)How the Improved AI System Can Do It (Specific Applications)

Improved AI System Specific Action Enabled by Paper's Theory Real-World Application Example

:---:---:---

Additive Regression on Non-Product Designs (e.g., correlated sensor data) Estimate the additive components of a regression model where input features are dependent, utilizing the Riesz basis representation to handle non-orthogonal geometry. Predicting machine failure by correlating temperature, vibration, and pressure sensors that are known to be statistically dependent in a factory setting.

Nonparametric Modeling with Unknown Covariate Distributions (e.g., climate/medical data) Construct estimators that achieve minimax rates even when the joint density of the inputs is unknown, relying only on uniform bounds on densities (Assumption 2.3). Developing a predictive model for disease progression using patient biomarkers where the exact correlation structure between biomarkers is not perfectly known beforehand.

Adaptive Dictionary Learning and Model Selection (e.g., Feature Selection) Use thresholded least-squares estimators to automatically select only the components of the additive model that contribute meaningfully to prediction, without requiring prior knowledge of which features are important. Building a sparse feature set for a recommendation engine where many potential features are correlated, allowing the system to focus computational power on only the most predictive ones.

Uncertainty Quantification in High-Dimensional Models (e.g., Bayesian NNs) Implement bias-variance decomposition derived from the Riesz representation to precisely quantify the error contributed by model approximation versus noise and estimation of input distributions. A self-driving car system that provides a prediction along with a calibrated confidence score indicating whether the uncertainty is due to noisy sensor data or insufficient model complexity.

Dimension-Growing Predictive Modeling (e.g., Dynamic Architectures) Design dynamic architectures whose complexity (dimension) can grow with the number of samples, ensuring that the learning rate remains optimal rather than degrading as in fixed-dimensional models. An AI system monitoring evolving financial markets where new types of risk factors or assets emerge over time, requiring a model capacity that scales with data volume.

Sparse Interaction Modeling (e.g., Biological Systems) Recover individual additive effects in systems where interactions are complex and dependent, by leveraging the distinction between signal smoothness and design density irregularity. Analyzing gene expression data to identify which specific genetic factors drive a phenotype, despite known correlations between many genes influencing the same pathway.

We study additive regression under an unknown and potentially non product design distribution, allowing the number of covariates to grow with the sample size. We consider a coupled class that separately controls the smoothness of the marginal densities and of each additive component multiplied by the corresponding marginal density. Under joint-density bounds that hold uniformly in the dimension and suitable dimension-growth conditions, we establish matching minimax bounds for prediction. With known marginal densities, the classical additive rate is attainable. When the marginals are unknown, this rate is preserved if the densities are at least as smooth as the weighted components. Otherwise, marginal-density smoothness determines the minimax rate over the coupled class. Finally, we recover all additive components with total squared error of the same order as the prediction error.