Deep Learning as Neural Low-Degree Filtering: A Spectral Theory of Hierarchical Feature Learning

arXiv:2605.13612 · cs.LG, cond-mat.dis-nn, stat.ML · Submitted 2026-05-13 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Deep Learning as Neural Low-Degree Filtering".

Jane: Understanding how deep neural networks learn useful internal representations is a central open problem in theory,

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So, we’re moving into segment two where we look at the title and authors of "Deep Learning as Neural Low-Degree Filtering: A Spectral Theory of Hierarchical Feature Learning." We talked about how it attempts to explain the core mechanism, so now let’s see who is behind this work.

Jane: Before we get into the names, I think it’s important to know that this paper brings together a team from different backgrounds at EPFL, which tells us they are approaching the problem from a multi-faceted perspective.

Lu: I find it interesting that having authors from statistical physics and information learning laboratories on the same page suggests they are aiming for a rigorous, theoretical foundation for this work. Tom That makes sense; when you’re trying to model something as abstract as deep learning dynamics, you need strong mathematical backing.

Meng: From an engineering view, I just want to know if these researchers have experience translating these spectral ideas into something that actually runs on real hardware without massive overhead. Jane That’s a fair question, Meng; the hope is that they’ve found ways to keep the complexity tractable by focusing only on the leading eigen-directions of the moment operator.

Lalam: For me, it’s exciting because it shows that deep theoretical research isn't just happening in abstract math; it’s being applied to concrete challenges like feature selection in neural networks. Tom It really demonstrates how foundational theory can have a direct impact on practical AI development.

Jane: They are really tackling the central open problem of understanding internal representations, and their focus on making this tractable is what makes this paper significant for the broader field of AI research.

Lu: I think it’s worth noting that when you see authors from different fields collaborating, you often get cross-disciplinary insights that can lead to novel conceptual leaps in how we frame a problem. Tom That’s exactly what I hope to see here; new ways of framing the problem that unlock new possibilities for AI development.

Meng: From my side, if this approach is too abstract, it might be hard to implement immediately, so I need to know how close they are getting to a practical mechanism without needing a massive theoretical overhead. Jane They are trying to balance the theoretical rigor with the need for a tractable surrogate mechanism that doesn't require impossibly high computational resources.

Lalam: It’s encouraging because it shows that we can make deep concepts accessible and applicable, which is what we want for the AI community right now. Tom So, as we move on, let’s see how this theoretical framework actually helps us understand those feature selection steps in more detail.

The paper's summary: Jane: Now that Tom and Jane have set the stage, we can dive into the actual summary of "Deep Learning as Neural Low-Degree Filtering: A Spectral Theory of Hierarchical Feature Learning." Essentially, this is where they lay out exactly what Neural LoFi is.

Tom: They define it as a stylized limit of gradient-based training where hierarchical feature learning becomes an explicit iterative spectral procedure. Lu That means we are looking at how the layers decouple and see the dynamics become more predictable at each step.

Meng: So, they’re saying that given the current representation, each layer forms a label-weighted moment operator on those features and selects its leading eigendirections. Jane Which is essentially performing a weighted principal component analysis to pick the directions with maximal low-degree correlation to the label.

Lalam: I see this as a way of summarizing that complex deep learning process into something much more manageable, which is very helpful for anyone trying to grasp the core idea quickly without getting bogged down in every detail. Tom Exactly, it abstracts away the complexity while keeping the underlying principles intact.

Jane: They then add a second step called the Lift Step where those projected features are re-expanded through a nonlinear random feature map. Lu This lift step is what allows the representation to become richer for the next layer, which is crucial for continuing the learning process.

Tom: So, instead of just projecting and moving on, they combine these two steps into this iterative procedure that acts as a surrogate for early gradient-based feature selection. Jane It’s a powerful tool because it provides an explicit mechanism to study feature learning beyond the lazy regime.

Lu: That framework is very compelling because it provides a concrete way to study multi-layer feature learning that was previously only vaguely defined by tensor-program or dynamical mean-field theory approaches, which are more general but less specific in practice. Tom So they’re providing a specific, tractable path for analysis.

Meng: From an engineering side, I like the idea that it gives us a concrete model to analyze the feature selection process rather than just guessing what those features are doing. Jane It moves it from intuition to evidence-based selection.

Lalam: This makes the paper't really useful because it’s not just abstract math; it grounds these concepts in how deep networks actually function. Tom That grounding is key for making this work for real-world AI development.

Lu: I think the core value here is providing a specific mechanism to study feature learning that was missing from existing literature, which opens up new theoretical possibilities.

Jane: So, the summary boils down to Neural LoFi being an explicit iterative spectral surrogate that allows us to look at deep networks in a way we couldn't before.

The paper's improvements: Tom: Now we move into segment four where we’re discussing what the paper actually suggests as improvements to this framework, which is where things get really actionable for us.

Jane: They suggest replacing standard feature extraction methods with the Neural LoFi operator at each layer instead of using fixed mechanisms like simple Principal Component Analysis or fixed random projections.

Lu: This means we’re moving towards a system that automatically discovers task-adaptive features that are simple in the geometry induced by previous layers. Tom So, it’s not just picking a feature, but picking one that fits the current structural constraints of the representation.

Meng: From an engineering standpoint, I like this because it suggests we can dynamically determine the optimal feature retention rank using their relevance-complexity trade-off criterion derived from LoFi. Jane This criterion tells us exactly when to stop adding features so we don't get wasteful over or under-parameterized.

Tom: That’s a huge improvement over setting a fixed feature count, because it gives us a data-driven way to manage the feature budget based on how predictive those features are relative to their complexity. Jane It allows for optimal scaling for given sample sizes, which is something we desperately need in resource-constrained environments.

Lu: They also propose the principle of low-degree compositionality as a guiding principle, implying that depth is effective when the target structure is not only compositional but also visible through those low-degree correlations in the current representation. Tom So they are emphasizing that it’s about building simpler sub-problems sequentially.

Jane: And this leads to an adaptive kernel construction, where each layer builds its own kernel based on the low-complexity features found in the previous layer. Lu This suggests a system where the geometry itself evolves as it learns.

Meng: If we can implement that evolving geometry, it means our AI agents could be much more flexible in handling novel or complex inputs because they won't be stuck with a single fixed statistical structure. Tom That flexibility seems like a big win for generalization in real-world scenarios.

Lalam: This is exciting because it moves us away from static representations and towards dynamic ones that are constantly adapting their internal structure based on the task at hand. Jane So, the representation itself becomes part of the learning process, evolving alongside the data.

Lu: I think this adaptive kernel construction could be a major concept for future work in creating truly flexible AI systems capable of handling diverse inputs effectively.

Tom: So, to wrap up on improvements: we get layer-wise spectral filtering, dynamic feature ranking based on relevance and complexity, and that adaptive kernel approach. Jane These are the three main ways they suggest moving beyond fixed methods.

Conclusion: Tom: Alright team, we’re wrapping up with the conclusion of "Deep Learning as Neural Low-Degree Filtering: A Spectral Theory of Hierarchical Feature Learning." We’ve seen that the authors have provided a rigorous mathematical surrogate for understanding feature learning.

Jane: They conclude that this framework provides a tractable surrogate mechanism for studying multi-layer feature learning beyond the lazy regime. Lu In simple terms, they’re showing us how deep networks select features follow a sequence of low-degree correlations and compositionality principles.

Meng: The practical implications are about moving from fixed methods to dynamic ones that adapt to the task structure through those adaptive kernels. Tom So we’ve seen the roadmap for building more flexible and efficient systems.

Lalam: This paper offers a way to understand these complex internal representations without needing backpropagation or end-to-end optimization, which is really useful for understanding AI's inner workings at a time when transparency is so important. Jane It gives us a clear lens into the process.

Lu: I think the principle of low-degree compositionality they describe as a key insight, emphasizing that depth is effective when the target structure is visible through those specific correlations. Tom That’s a deep theoretical concept we can build on for future research in building more structured AI.

Meng: From an engineering side, I see the ability to use this diagnostic criterion to monitor feature emergence as a way to know exactly when our model has discovered something new that warrants attention. Jane That means knowing precisely where the learning process is progressing beyond just observing error reduction.

Tom: So, we’ve got a solid theoretical framework for Neural LoFi that explains how deep networks learn useful internal representations through low-degree filtering and compositionality principles. Lu This gives us a powerful tool to analyze and build more structured AI systems.

Jane: It really helps us bridge the gap between abstract theory and the actual training dynamics of these complex models.

Lalam: For me, it's about giving our AI a better internal roadmap for how to improve itself based on understanding its own feature selection process. Tom A roadmap for future AI development.

École Polytechnique Fédérale de Lausanne (EPFL)

cs.LG, cond-mat.dis-nn, stat.ML

Submitted: 2026-05-13

Updated: 2026-09-04

Comments: 79 pages, 16 figures, companion codes in https://github.com/IdePHICS/Neural-LoFi-Theory

Code: https://github.com/IdePHICS/Neural-LoFi-Theory

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 89/100

The gist: Understanding how deep neural networks learn useful internal representations is a central open problem in theory, and this paper addresses that challenge by proposing a mathematically tractable

Key concepts

Neural LoFi
A stylized limit of gradient-based training where hierarchical feature learning becomes an explicit iterative spectral procedure. It means each layer forms a label-weighted moment operator on features and selects its leading eigendirections, essentially performing weighted principal component analysis to pick features with maximal low-degree correlation to the label.
Lift Step
A second step in the Neural LoFi process where the projected features are re-expanded through a nonlinear random feature map. This step is crucial because it allows the representation to become richer for the next layer, enabling continued learning.
Adaptive Kernel Construction
A proposed improvement where each layer builds its own kernel based on the low-complexity features found in the previous layer. This suggests that the geometry of the representation evolves as it learns, allowing AI agents to be more flexible with novel inputs.
Low-Degree Compositionality
A guiding principle suggesting that depth is effective when the target structure is visible through low-degree correlations in the current representation. It implies building simpler sub-problems sequentially.

Terminology

Summary

Understanding how deep neural networks learn useful internal representations is a central open problem in theory, and this paper addresses that challenge by proposing a mathematically tractable surrogate for feature learning called Neural Low-Degree Filtering (Neural LoFi). This framework provides an explicit, layer-wise mechanism for studying hierarchical feature learning beyond the lazy regime, offering a concrete predictive model of how deep networks select and refine features.

How it works

Neural LoFi is defined as a stylized limit of gradient-based training that operates iteratively across layers. Given the current representation z-1(x, mu, mu), each layer performs two distinct steps to generate the next representation z(x):

  1. The Filter Step: The layer forms a label-weighted moment operator, b =, and projects the current features onto its leading eigen-directions. This is equivalent to a weighted principal component analysis that selects low-degree correlation features.

  2. The Lift Step: The resulting projected features are then lifted through a nonlinear random feature map sigma(R g), which re-expands the coordinates into a richer representation for the next layer.

This process is motivated by the fact that under specific initializations, gradient descent dynamics can be approximated as a linear projection onto b followed by an exponential weighting of eigenvalues (Proposition 1).

The core theoretical insights

The analysis of Neural LoFi yields two primary lessons regarding the nature of deep learning:

  • A relevance–complexity trade-off: The next layer selects features that are simple in the geometry induced by the current representation while having large low-degree correlation with the label. This is formalized through a variational problem.

  • The principle of low-degree compositionality: Depth is effective when the target structure is not only compositional but also visible through low-degree correlations in the current representation.

This mechanism allows for a kernel interpretation, where Neural LoFi constructs a sequence of task-adaptive kernels, rather than relying on a single fixed kernel.

The emergence criterion

A crucial contribution of this work is providing a quantitative diagnostic for when new concepts become learnable. The paper establishes an explicit criterion for feature emergence: A new direction becomes learnable when its population correlation rho L rises above the empirical noise floor tau Lk(n). This threshold is controlled by the residual effective dimension of the current kernel, providing a data-driven diagnostic for concept emergence.

Empirical validation and application

The theory was tested on real datasets, including binary CIFAR-10 and CelebA. The results demonstrated that Neural LoFi:

  • Improves over lazy random-feature baselines, achieving superior generalization in the low-data regime.

  • Recovers meaningful structured filters without requiring backpropagation or end-to-end optimization.

  • Provides a mechanism for feature selection that is layerwise, backpropagation-free, and directly applicable to both fully connected and convolutional architectures.

Improvements for AI systems

Based on the rigorous analysis of Deep Learning as Neural Low-Degree Filtering, here are specific, actionable improvements to AI systems and the resulting capabilities.

1. Implementation of Layer-Wise Spectral Filtering (The LoFi Operator)

  • Improvement: Replace standard feature extraction mechanisms (like simple Principal Component Analysis or fixed random projections) with the Neural Low-Degree Filtering (LoFi) operator at each layer.

  • The LoFi mechanism involves two steps: projecting the current representation onto the leading eigenvectors of a label-weighted moment operator (C), and then lifting those selected features through a nonlinear random feature map (sigma(R g)).

  • What the Improved System Can Do:

  • Task-Adaptive Feature Discovery: The system will automatically discover task-relevant features that are simple in the geometry induced by previous layers. Unlike fixed kernel methods, this mechanism is supervised and adapts to the target labels, ensuring it finds features that are highly predictive of y relative to their complexity.

  • Guaranteed Compositional Learning: It enforces a low-degree compositionality principle, enabling the system to learn complex functions by progressively building simpler sub-problems in intermediate representations, making it highly effective for hierarchical data structures (e.g., in NLP or structured image analysis).

2. Integration of Feature Selection (Optimal Rank Determination)

  • Improvement: Instead of using a fixed number of features (k) or relying on standard regularization, use the relevance-complexity trade-off criterion derived from LoFi to dynamically determine the optimal feature retention rank (k).

  • This criterion dictates that a new direction is learnable only when its population correlation rho rises above the empirical noise floor tau eff(n), where tau eff is controlled by the residual effective dimension of the C operator.

  • What the Improved System Can Do:

  • Optimal Feature Budgeting: The system avoids over-parameterization and under-parameterization. It knows exactly how many features are necessary to capture a signal before those features become statistically indistinguishable from noise, maximizing efficiency for given sample sizes.

3. Feature Emergence Monitoring

  • Improvement: Implement the Feature Emergence Criterion (rho tau eff(n)) as a real-time diagnostic tool during training or model evaluation.

  • This involves tracking the overlap between the features learned from a small subset of data (the current n) and a large-sample reference set, monitoring when this overlap crosses the predicted threshold derived from Equation (116).

  • What the Improved System Can Do:

  • Predictive Diagnostics: The system can predict when specific concepts (e.g., cheekbones in CelebA or edge detectors in CIFAR-10) will become reliably learned, allowing engineers to know exactly when a plateau phase ends and the actual feature-learning phase begins, moving beyond simple error monitoring.

4. Analyzing Learning Dynamics (Saddle-to-Saddle Equivalence)

  • Improvement: Use the saddle-to-saddle dynamics analysis derived in Section B to interpret gradient descent performance, particularly in models with low information exponents (IE 2).

  • The system analyzes the convergence path of its internal weights, identifying whether they are progressing through a saddle-to-saddle cascade (the expected behavior for efficient learning) or getting stuck.

  • What the Improved System Can Do:

  • Root Cause Analysis: It provides a mathematically rigorous explanation for why certain models (like those with low IE) succeed: it interprets the success not as a smooth descent, but as the sequential recovery of specific, high-correlation directions.

5. Adaptive Kernel Construction (Kernel LoFi)

  • Improvement: Utilize the Kernel LoFi approach to create a sequence of task-adaptive kernels (K 0 to K 1 to).

  • Instead of using a single, fixed kernel defined before seeing the labels, each layer uses its own kernel constructed from the low-complexity features found in the previous layer.

  • What the Improved System Can Do:

  • Supervised Kernel Learning: The system learns not just a mapping, but an evolving geometry. This is vital for complex tasks where different stages of a signal require different types of statistical analysis (e.g., transforming raw pixels into structured motifs).

Feature Original AI System Limitation LoFi Improvement Resulting Capability

:---:---:---:---

Feature Selection (Section 2.1) Fixed, heuristic, or random feature selection. Failure to distinguish signal from noise. Optimal Rank Determination: Using the relevance-complexity trade-off (rho tau eff). Optimal efficiency; knowing exactly how many features are necessary for task performance.

Learning Mechanism (Section 2.2) Implicit, decoupled dynamics; black box learning process. Layer-Wise Spectral Filtering: Explicitly selecting leading eigen-directions of the label-weighted moment operator C. Guaranteed hierarchical compositionality; transforming a high-degree problem into a sequence of simpler spectral recoveries.

Diagnostics (Section 3.1) Only measures final error or general activation patterns. Feature Emergence Criterion: Tracking overlap against predicted tau eff(n) thresholds. Predictive diagnostics; knowing exactly when specific concepts emerge from the noise floor at a given data scale n.

Architecture (Section 3.2) Assumes uniform performance across layers. Low-Degree Compositionality: Applying LoFi to structured inputs (e.g., CNN) without backpropagation. Recovering known visual structures (edges, contrast) even in early stages of feature learning, providing a first-principles understanding of visual processing.

Abstract

Understanding how deep neural networks learn useful internal representations from data remains a central open problem in the theory of deep learning. We introduce Neural Low-Degree Filtering (Neural LoFi), a stylized limit of gradient-based training in which hierarchical feature learning becomes an explicit iterative spectral procedure. In this limit, the dynamics at each layer decouple: given the current representation, the next layer selects directions with maximal accessible low-degree correlation to the label. This yields a tractable surrogate mechanism for deep learning, together with a natural kernel-space interpretation. Neural LoFi provides a mathematically explicit framework for studying multi-layer feature learning beyond the lazy regime. It predicts how representations are selected layer by layer, explains how emergence of concepts arises with given sample complexity, and gives a concrete mechanism by which depth progressively constructs new features from old ones through low-degree compositionality. We complement the theory with mechanistic experiments on fully connected and convolutional architectures, showing that Neural LoFi improves over lazy random-feature baselines, recovers meaningful structured filters, and predicts representations aligned with early gradient-descent feature discovery with real datasets.

Sources

Related papers