Overparameterized Multiple Linear Regression as Hyper-Curve Fitting

arXiv:2404.07849 · stat.ML, cs.LG · Submitted 2026-02-25 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "Overparameterized Multiple Linear Regression as Hyper-Curve Fitting".

Jane: The paper was written by Elisa Atza and Neil Budko from Delft University of Technology.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Title: Tom: Alright, welcome back to the show, everybody. Today we’re digging into a paper that just landed on arXiv with a title that sounds like a mouthful: “Overparameterized Multiple Linear Regression as Hyper-Curve Fitting.” Jane, I’ll be honest, when I first saw that title, I thought, okay, this is going to be another one of those papers that’s all math and no heart.

Jane: And then you read it and found out it’s actually all math and a whole lot of heart. Tom, this paper is genuinely exciting because it’s asking a question that’s been bugging people in machine learning for years. When you have way more predictors than data points, why does a simple linear model still work so well? And the authors, Elisa Atza and Neil Budko from TU Delft, they’ve come up with an answer that flips the whole picture upside down.

Tom: Upside down how? Because I thought overparameterized linear regression was this well-studied thing, you know, minimum norm solutions, benign overfitting, all that.

Jane: Right, but here’s the twist. They say that when your dataset is fundamentally overparameterized, the model isn’t actually fitting a hyperplane at all. It’s fitting a hypercurve, a one-dimensional curve living in a high-dimensional space. And that curve can be parameterized by a single scalar, which they suggest should be the dependent variable itself.

Tom: So instead of thinking about y as a function of all the x’s, you think about each x as a function of y. That’s the inverse regression idea, right?

Jane: Exactly. And they prove that under certain conditions, the predictions from this inverse regression model are identical to the predictions from the standard multiple linear regression. So you get the same answers, but you get a much clearer view of what each predictor is actually doing.

Tom: That’s huge for interpretability. I mean, in the standard model, you get these beta weights and you’re supposed to trust them, but they’re these convoluted combinations of everything. Here, you can literally plot each predictor as a function of y and see if it’s linear, quadratic, or just noise.

Jane: And that’s what I love about this paper. It’s not just theory for theory’s sake. They use this to identify which predictors are “improper,” meaning they don’t fit the curve model, and they show that removing those predictors improves prediction accuracy on real chemometric data. We’re talking about NIR spectra of yarn, predicting density, and they get the test error down from zero point two eight to zero point zero five just by removing the bad columns.

Tom: Wait, that’s a massive improvement. So the model was being dragged down by these predictors that were violating the assumptions, and the hypercurve view lets you spot them and kick them out?

Jane: Precisely. And that’s the kind of practical payoff that makes me want to keep reading. But we’re just getting started. Next segment, we’re going to dig into the actual math of how they define these fundamentally overparameterized datasets and why the equivalence holds.

Tom: I’m on board. Let’s get into the weeds.

Summary: Jane: So we’re back, and we’re still talking about “Overparameterized Multiple Linear Regression as Hyper-Curve Fitting.” Tom, last segment I teased the big idea, but now I want to get into the actual structure of the paper, because there’s a really clean progression here.

Tom: Please, walk me through it. Because I’m still trying to wrap my head around how a linear model can be a curve.

Jane: Okay, so they start with a definition. They call it a Fundamentally Overparameterized dataset, or FOP for short. The idea is that no matter how many samples you add, the rank of the data matrix stays below the number of predictors. So it’s not just that you happen to have more columns than rows; it’s that the data itself is constrained to live in a lower-dimensional space.

Tom: So the data is secretly low-rank, even though you have hundreds of columns.

Jane: Exactly. And then they introduce the PARCUR model, which is just a parametric hypercurve. You pick a scalar parameter, say s, and every variable, including y, is a function of s. Then they show that if you choose s to be y itself, you get the inverse regression model, where each predictor x j is a function of y.

Tom: And that’s the IR model. So the predictors are polynomials in y, or whatever basis you choose.

Jane: Right. And here’s the key theorem. They prove that if your training set is complete, meaning it has the maximal rank possible for that FOP dataset, then the predictions from the standard MLR model are exact. Not approximately exact, but exact. And they also prove that the MLR and PARCUR predictions are identical when you use the same training data.

Tom: So the two models are mathematically equivalent, even though they look completely different on the surface.

Jane: Yes. And that’s not just a curiosity. It means you can use the PARCUR model to analyze the data in a way that’s much more interpretable, and you know that whatever conclusions you draw about the predictors will also apply to the MLR model.

Tom: But what about when the training set isn’t complete? Because in practice, you rarely have a complete set, right?

Jane: That’s where it gets interesting. They show that even with an incomplete training set, the prediction error has a specific structure. And they identify a condition where the prediction of y is still exact, even if the training set isn’t complete. It’s this condition that y transpose times (X X transpose) inverse times y equals one. It’s a bit technical, but it shows that the model can be surprisingly robust.

Tom: And then they move on to the case where the predictors are polynomials. They classify them into three types: low-degree polynomials, high-degree polynomials, and non-functional relationships, like data on a cone.

Jane: Right, and they prove that even if you have high-degree or non-functional predictors, as long as you have enough low-degree polynomial predictors, the dataset is still FOP. So the model can still make exact predictions, even though some of the predictors are completely violating the linear assumptions.

Tom: That’s the “illusion of understanding” they talk about. You think the model is working because it’s capturing something real, but actually it’s just the low-degree predictors doing all the work.

Jane: Exactly. And that’s why the next part of the paper is so important. They introduce a regularization scheme based on truncating the polynomial degree, and they show how to use cross-validation to find the optimal degree. That’s the practical tool that makes all of this usable.

Tom: So we’ve got the theory, we’ve got the regularization, and then they go even further with predictor removal. That’s what we’re going to dig into next.

Jane: And that’s where the real-world impact shows up. Let’s keep going.

Improvements: Tom: We’re back, still on “Overparameterized Multiple Linear Regression as Hyper-Curve Fitting.” Jane, last segment we covered the core theory and the regularization. Now I want to talk about the part that actually got me excited, which is the predictor removal algorithm. Because that’s where the paper goes from clever math to something you can actually use in the lab.

Jane: Absolutely. And the way they frame it is really smart. They’ve already used the training X-data to tune the polynomial degree r-star. So they still have the training y-data and the test X-data available. They use the test X-data to compute a column-wise prediction error for each predictor, which they call chi j. That tells you how well each predictor is being predicted by the model.

Tom: So if a predictor is being predicted badly, it’s probably not following the curve. It’s either too noisy or it’s non-functional.

Jane: Right. And then they set a threshold tau, and any predictor with chi j above that threshold gets removed. But here’s the clever part: they don’t just pick tau arbitrarily. They tune it using the training y-data, because that data hasn’t been used for anything yet. They compute the prediction error on y as a function of tau, and they pick the tau that minimizes that error.

Tom: So they’re using the y-data to find the best threshold, and the X-data to find the best polynomial degree. That’s a really clean separation of the data into different roles.

Jane: Exactly. And in their synthetic experiments, it works beautifully. They have two hundred polynomial predictors, two non-functional ones, and noise in a third of the columns. The algorithm correctly identifies all the noisy and non-functional predictors and removes them. And the prediction error on the test set drops significantly.

Tom: And then they apply it to the yarn dataset, which is this classic chemometric benchmark. twenty-eight NIR spectra, two hundred sixty-eight wavelengths, predicting yarn density. They use twenty-one samples for training and seven for testing.

Jane: Right. And with n equals twenty-one they can’t use cross-validation, so they visually inspect the polynomial fits and pick r-star equals five. Then they compute the chi j errors and find that a whole band of wavelengths is being discarded. And when they look at those discarded predictors, they see this cone-shaped data, just like the non-functional examples from the synthetic experiments.

Tom: So that frequency band in the NIR spectrum is actually sitting on a higher-dimensional manifold, not a curve. And the algorithm catches that and removes it.

Jane: Yes. And the result is dramatic. The test error drops from zero point two eight to zero point zero five. That’s a huge improvement just by removing the bad predictors. And it also tells you something about the data itself. Those wavelengths are not useless in general, they’re just not useful for a linear curve model. They might be perfect for a nonlinear or higher-dimensional model.

Tom: That’s the part I love. The algorithm isn’t just improving predictions; it’s telling you where your model is wrong and where you might need a different kind of model.

Jane: And that’s a really valuable diagnostic tool. But we’ve got one more segment to wrap this up, and I want to bring in some other voices to talk about what this means for the field.

Conclusion: Tom: Alright, we’re in the final stretch, and we’re still talking about “Overparameterized Multiple Linear Regression as Hyper-Curve Fitting.” Jane, we’ve covered the theory, the regularization, and the predictor removal. Let’s bring in Lu and Meng to get their take on the bigger picture.

Jane: Good idea. Lu, you’ve been quiet this whole time. What do you think about the implications of this work?

Lu: I think the most exciting implication is that it gives us a principled way to think about when linear models work and when they don’t. The hypercurve perspective is so clean. It says that a linear model is only appropriate when the data actually lives on a curve parameterized by the dependent variable. And when it doesn’t, you can detect that and either remove the offending predictors or switch to a higher-dimensional model.

Tom: So it’s not just a better regression method; it’s a diagnostic tool for understanding the geometry of your data.

Lu: Exactly. And I think the natural extension is to a two-parameter hyper-surface model. The authors mention that at the end, and I think that’s where the real potential is. If you can fit a surface instead of a curve, you can capture more complex relationships while still keeping the interpretability.

Meng: But let’s talk about the practical side. This algorithm requires you to sort your data by the dependent variable, and that’s not always easy when you have noise. The paper mentions that the Wiener filter only works well above one hundred fifty samples. So for small datasets, you’re stuck with visual inspection, which is subjective.

Jane: That’s a fair point, Meng. The paper is honest about that limitation. But for larger datasets, the cross-validation approach works, and the results are solid.

Meng: And the predictor removal step, that’s the part I’d want to test in production. The fact that it uses the training y-data to tune the threshold is clever, but I’d want to see how robust that is across different noise levels and different types of data.

Lu: I think it’s robust enough to be useful. And even if the threshold isn’t perfect, the paper shows that removing the worst predictors almost always helps. The improvement on the yarn dataset, from zero point two eight to zero point zero five, is hard to argue with.

Tom: And Lalam, what do you think? You’ve been listening to all of us. What’s the most impactful vision here?

Lalam: I think the most impactful vision is that this paper gives us a way to make machine learning models more trustworthy in high-stakes fields like medicine and agriculture. When you have thousands of predictors and only a few hundred samples, you need to know which predictors are actually meaningful. This paper provides a systematic method to identify and remove the ones that are just noise or non-functional. That could lead to more reliable diagnostics, better crop yield predictions, and more targeted experiments.

Jane: That’s a beautiful way to put it. And I think that’s the real legacy of this paper. It’s not just a mathematical curiosity; it’s a practical tool for making sense of complex, high-dimensional data.

Tom: Alright, so to wrap it up: “Overparameterized Multiple Linear Regression as Hyper-Curve Fitting” gives us a new way to think about linear models, a way to regularize them, and a way to clean up our predictor sets. And it does all of that with clear math and real-world validation.

Jane: And we’re saying goodbye to this paper, but we’re definitely going to be thinking about it for a while. Thanks for listening, and we’ll see you next time with another exciting paper.

Tom: Take care, everyone.

Elisa Atza, Neil Budko

Delft University of Technology

stat.ML, cs.LG

Submitted: 2026-02-25

Updated: 2026-08-17

Comments: 18 pages, 8 figures, version 2 (IOP style, revised), Python code and data available at: https://github.com/the-iterator/hyper-curve-regression-yarn

Code: https://github.com/the-iterator/hyper-curve-regression-yarn

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 42/100

Key concepts

Fundamentally Overparameterized Dataset (FOP)
This is a dataset where the data itself is constrained to live in a lower-dimensional space. It means that even if you have many more predictors than data points, the underlying structure of the data limits its complexity.
Hypercurve/PARCUR Model
Instead of fitting a standard linear plane, this model treats the relationship as a one-dimensional curve parameterized by the dependent variable (y). This provides a much clearer and more interpretable view than traditional regression models.
Predictor Removal Algorithm
This is a practical method that calculates an error score for each predictor using test data. Predictors with high error are removed, allowing the model to eliminate non-functional or noisy variables and improve overall prediction accuracy.

Terminology

Summary

Summary

This paper demonstrates that applying a fixed-effect multiple linear regression (MLR) model to an overparameterized dataset is mathematically equivalent to fitting a hyper-curve parameterized by a single scalar. This reformulation shifts the focus from global coefficients to individual predictors, allowing each to be modeled as a function of a common parameter. The authors prove that this overparameterized linear framework can yield exact predictions even when the underlying data contains nonlinear dependencies that violate classical linear assumptions. By employing parameterization in terms of the dependent variable and a monomial basis, the approach is validated on both synthetic and experimental datasets. The results show that the hyper-curve perspective provides a robust framework for regularizing problems with noisy predictors and offers a systematic method for identifying and removing ’improper’ predictors that degrade model generalizability.

The paper introduces the PARametric hyper-CURve (PARCUR) model, which states that all data are functions of some scalar parameter s, describing a parametric hyper-curve in a (p+1)-dimensional space. Under a monotone transformation of variables, the PARCUR model is equivalent to the Inverse Regression (IR) model, where the independent variables xj are considered functions of the dependent variable y. The IR model is the most easily interpretable version of the PARCUR model as the functions xj(y) provide insight into the change of each individual predictor variable as a function of the dependent variable y.

The paper defines a Fundamentally OverParameterized (FOP) dataset of rank q as one where, for any m, the data-matrix Sm has the property rank(Sm) ≤ q ≤ p. A training dataset is complete if rank(Sn) = q. Theorem 2.1 establishes that the prediction produced by the MLR model is exact for any data matrix from the FOP dataset if and only if the training dataset is complete and rank(X) = q. Theorem 2.2 proves that the predictions made by the MLR and PARCUR models are equal for any basis in the column function space with an invertible basis matrix.

In Section 3, the paper considers data containing polynomial column functions. The authors establish that a dataset is a FOP dataset if it contains k predictors that are polynomials in y of degrees greater than q-1 or have a non-functional relation to y, and its remaining p-k predictors are polynomials in y of degree r, where 1 ≤ r ≤ q-k-1. Each of the p+1 data columns belongs to one of three classes: polynomials in y of degree less than or equal to n-1, polynomials of degree higher than n-1, or non-functional dependencies between xj and y. The paper demonstrates that having extremely high-degree polynomials or non-functional dependencies among predictors does not prevent the MLR and IR models from making exact predictions as long as there are also polynomial data of sufficiently low degree and the training set is complete.

Section 4 addresses polynomial regularization for noisy data. The paper assumes additive Gaussian noise and investigates three cases: noise only in the dependent variable, i.i.d. noise in the data matrix, and independent but not identically distributed noise. The regularization approach depends on the number of available training samples. For n ≥ 150, the Wiener filter and cross-validation are effective; for n < 150, the optimal truncation index r* is determined subjectively. The optimal truncation index corresponds to the polynomial degree for which the minimum of the validation error ρ(Xv) is attained. The paper notes that the IR model always over-fits the training y-data because the dependent variable is the second column of the Vandermonde basis matrix, so regularization is validated on the X-data columns.

Section 5 proposes a novel predictor removal algorithm. The paper introduces the column-wise test set prediction error χj and defines the optimal threshold τopt as the one for which the feature-removal error ρ(y) with r = r* is minimal. The hyper-parameters are tuned using different data: the optimal degree r* uses the training X-data, which predictor to remove uses the test Xt-data, and the optimal threshold τopt uses the training y-data. The paper demonstrates that the removal of ’improper’ predictors improves both the adequacy and predictive power of the model.

Section 6 applies the regularized IR model with predictor removal to the experimental chemometric Yarn dataset, which consists of 28 Near InfraRed (NIR) spectra of Polyethylene terephthalate (PET) yarns measured at p = 268 wavelengths, with 21 training samples and 7 test samples. The optimal degree r* = 5 was determined by visual inspection. The removal of ’improper’ predictors significantly improved the quality of prediction, with the prediction error ρ(yt) on the test dataset going down from 0.28 to 0.05 after the predictor removal procedure. The paper demonstrates the presence of both curve-like and higher-dimensional manifolds in this dataset, with some predictor data exhibiting a ’conical’ shape, indicating that the NIR data in certain frequency bands is situated on a hyper-cone.

The conclusions state that the hyper-curve PARCUR model provides an alternative, predictor-centered look at overparameterized multiple linear regression. The column-centered nature of the approach allows for a rigorous algorithm for detecting ’improper’ predictors, and removing these predictors improves not only the adequacy but also the predictive power of the model. The polynomial IR version of the PARCUR model appears to work well with chemometric data but may also be suitable for other ’smooth’ predictors. The main problem with applying the IR model is the difficulty of sorting the dependent variable by magnitude in the presence of noise. The paper suggests that a two-parameter hyper-surface model would be a natural extension of the present single-parameter hyper-curve model.

Improvements for AI systems

Based on the paper, here are specific improvements for AI systems:

1. Improved Feature Selection for High-Dimensional Regression

  • Implement the column-wise prediction error (χj) as a new feature importance metric, replacing or augmenting traditional coefficient magnitudes

  • Add automatic detection and removal of improper predictors (those with non-functional or high-degree polynomial relationships to the target) using the threshold optimization method

  • This reduces overfitting and improves generalization when training data is scarce (n < p)

2. New Regularization Strategy: Polynomial Degree Truncation

  • Replace or supplement L1/L2 regularization with a basis-function truncation approach that limits the polynomial degree of the inverse regression model

  • Use cross-validation on the predictor matrix X (not the target y) to find the optimal truncation degree, since y is always overfitted in this formulation

  • This handles noise in both dependent and independent variables simultaneously

3. Hyper-Curve Parameterization for Nonlinear Data

  • Train models by parameterizing all variables as functions of a single scalar (e.g., the target variable), rather than assuming a linear hyper-plane

  • This allows exact predictions even when underlying data has nonlinear dependencies that violate classical linear assumptions

  • Implement the equivalence between MLR and PARCUR models to gain interpretability of individual predictor-target relationships

4. Detection of Higher-Dimensional Manifolds

  • Use the residual structure after fitting the hyper-curve to identify subsets of predictors that lie on higher-dimensional manifolds (e.g., cones, surfaces)

  • This enables automatic flagging of data regions where a linear or single-parameter model is inadequate, suggesting where nonlinear or multi-parameter models would be more appropriate

5. Robustness to Noisy Predictors

  • Apply the Wiener filter and sorting-by-target strategy for datasets with n ≥ 150 to denoise the parameterization

  • Use the feature removal error on training y-data to tune the removal threshold, even when test data is unavailable

  • This improves prediction accuracy by 5-10x in chemometric applications (e.g., reducing test error from 0.28 to 0.05)

6. Interpretable Model Coefficients

  • Replace opaque regression weights with per-predictor polynomial coefficients (constant, linear, quadratic terms) that directly show the functional form of each predictor's relationship to the target

  • This enables domain experts to validate whether relationships are physically meaningful, reducing illusions of understanding from spurious correlations

What the improved AI system can do:

  • Achieve near-exact predictions on overparameterized datasets (n < p) even with noisy or non-functional predictors

  • Automatically identify and remove predictors that degrade generalization, improving test accuracy

  • Provide interpretable, per-feature functional relationships (not just weights)

  • Detect when data lies on higher-dimensional manifolds, guiding model selection

  • Handle both noise in the target and predictors without requiring known noise levels

  • Work effectively with small training sets (n < 150) using subjective degree selection, and larger sets using cross-validation

Sources

Related papers