Foundations of Independent Component Analysis

This paper presents a comprehensive, self-contained mathematical treatment of the foundations of linear Independent Component Analysis (ICA), aimed at readers with a background in measure-theoretic probability theory. The paper develops the theory of characteristic functions, proves several identifiability results for ICA models under successively stronger assumptions, and presents an algorithm for source recovery.

The core ICA model postulates that an observed p-dimensional random vector X is an affine image of a k-dimensional random vector Z with mutually independent components: X = AZ + µ. The identifiability question asks how much of the pair (A, L(Z)) is determined by the law of X alone. Some ambiguity is unavoidable: sources can be rescaled, permuted, and shifted. The paper's central finding is that, under suitable assumptions, this is the only ambiguity, with one important exception: Gaussian sources, which can be rotated arbitrarily without changing the law of X.

The paper's logical structure is built on a single fundamental result, Theorem 4.2, which is proved in Appendix B. This theorem establishes a dichotomy for representations with independent, non-constant sources: any column of one mixing matrix that is not proportional to a column of the other forces its corresponding source to be Gaussian. For proportional columns, the sources are related by a factor that is the exponential of a polynomial, which by Marcinkiewicz' theorem implies a Gaussian relationship.

From this foundation, the paper branches into two lines of development:

1. Strengthening assumptions on sources to achieve stronger identifiability results.

2. Fixing convenient assumptions to develop an estimation algorithm.

Key Identifiability Results:

• Non-Constant Sources (Theorem 4.2): This is the foundational result, establishing the column dichotomy described above.

• Non-Gaussian Sources (Theorem 5.5): With the stronger assumption that sources are non-Gaussian, the mixing matrix is identifiable up to permutation and scaling. However, the sources themselves are only identifiable up to an additional componentwise additive Gaussian noise. This is a genuine ambiguity, not an artefact of the proofs.

• Gaussian-Free Sources (Theorem 6.12): The paper introduces the concept of Gaussian-free random variables, which are those from which no non-degenerate Gaussian noise can be split off. Theorem 6.7 shows that every real-valued random variable decomposes essentially uniquely into a Gaussian-free part and independent Gaussian noise. Under the assumption that sources are Gaussian-free, the paper proves its strongest identifiability result: sources are identifiable up to permutation, scale, and translation, even in the presence of additive Gaussian noise with an arbitrary, possibly non-diagonal and degenerate covariance matrix. This removes the last remaining ambiguity from Theorem 5.5.

Estimation Algorithm:

The paper then focuses on the complete noiseless ICA model (square invertible mixing matrix, no noise). It derives the maximum likelihood objective and shows that the standard preconditioner in the gradient step is not an approximate inverse Hessian but the exact gradient for a natural right-invariant metric on the group of invertible matrices. This leads to the equivariant gradient descent ICA algorithm.

The paper proves a local stability theorem (Theorem 7.20) for this algorithm, showing that a separating solution is a locally asymptotically stable equilibrium of the mean dynamics precisely when the identifiability conditions from Corollary 7.4 are met. This theorem introduces the crucial stability quantity ζj, which depends on the source distribution and the chosen model score (nonlinearity). The paper shows that:

• For Gaussian sources, ζj = 1, placing them exactly on the stability boundary.

• For correctly specified sources (using the true score), ζj ≥ 1, with equality only for Gaussian sources, meaning the algorithm is stable if and only if at most one source is Gaussian.

• For the cubic nonlinearity, the stability criterion is exactly that the source is sub-Gaussian (kurtosis < 0).

The paper concludes by showing how a causal ordering (LiNGAM) removes the permutation ambiguity, and provides a brief overview of generalisations of the ICA model, including nonlinear ICA, multidimensional ICA, and multi-view ICA.

Improvement 1: Robust Identifiability-Aware Source Separation

• What: Integrate Theorem 6.12 (Gaussian-free decomposition) into the AI system's generative model. Instead of assuming sources are purely non-Gaussian, the system explicitly decomposes each latent variable into a Gaussian-free component and an independent Gaussian residual.

• What the improved AI system can do: In blind source separation (e.g., separating mixed audio or biomedical signals), the system can now recover sources even when additive Gaussian noise is present with unknown, non-diagonal covariance. It will not confuse Gaussian noise with signal, and it can output both the cleaned source and the noise covariance, eliminating the ambiguity that plagues standard ICA. This is critical for EEG/fMRI denoising where noise is spatially correlated.

Improvement 2: Stability-Conditioned Learning Rate Adaptation

• What: Use Theorem 7.20 and the stability quantity ζj to dynamically adjust the learning rate or step size per source component during gradient descent. The system computes ζj from the current source estimate and the chosen nonlinearity (e.g., true score or cubic).

• What the improved AI system can do: In online or mini-batch ICA training, the system automatically slows down updates for components near the stability boundary (ζj ≈ 1, e.g., near-Gaussian sources) and accelerates for well-separated sub-Gaussian sources. This prevents divergence and oscillation, leading to faster convergence and higher final separation quality. For the cubic nonlinearity, the system can even pre-screen sources and reject those with positive kurtosis, guaranteeing stability.

Improvement 3: Causal Ordering Disambiguation for Latent Factors

• What: Incorporate the LiNGAM causal ordering result (from the conclusion) into the AI system's latent variable model. After ICA separation, the system applies a causal ordering algorithm to resolve the permutation ambiguity, not just for visualization but for downstream causal inference.

• What the improved AI system can do: In causal discovery from observational data (e.g., gene regulatory networks or economic indicators), the system can output not just independent components but a directed acyclic graph (DAG) with a unique causal order. This allows the AI to answer interventional questions (What happens if we perturb X?) without needing randomized experiments, directly improving decision-making in personalized medicine or policy simulation.

Improvement 4: Gaussian-Source Detection and Dimensionality Reduction

• What: Use the dichotomy in Theorem 4.2 (non-proportional columns force Gaussian sources) to build a diagnostic module that identifies which latent dimensions are Gaussian and thus unidentifiable.

• What the improved AI system can do: In representation learning (e.g., autoencoders or variational autoencoders), the system can automatically detect and flag Gaussian latent dimensions. It can then either (a) remove them to achieve a fully identifiable representation, or (b) explicitly model them as noise, improving the interpretability of the latent space. This is useful for fairness (avoiding spurious correlations) and for model compression (dropping unidentifiable dimensions).

Improvement 5: Natural-Gradient Equivariant Optimization for Deep Networks

• What: Apply the paper's insight that the preconditioner is the exact gradient for a right-invariant metric (Section 7) to general neural network training, not just ICA. Replace standard Adam or SGD preconditioners with a metric derived from the group structure of the weight matrices.

• What the improved AI system can do: For any deep learning task with invertible weight matrices (e.g., normalizing flows, recurrent networks), the system achieves faster and more stable convergence by respecting the geometry of the parameter space. It avoids the common pitfall of approximate inverse Hessians, leading to better conditioning and reduced sensitivity to learning rate tuning. This improves training of generative models and sequence models.

Improvement 6: Marcinkiewicz-Based Distributional Validation

• What: Use the exponential-of-polynomial result (from Theorem 4.2) as a built-in validity check for the learned source distributions. The system verifies that any non-Gaussian source's characteristic function is not of that restricted form.

• What the improved AI system can do: In any ICA-based pipeline (e.g., speech enhancement, financial factor analysis), the system can automatically detect overfitting or model misspecification. If a learned source's distribution is too close to the Gaussian-exponential-polynomial family, the system raises a warning that the separation is unreliable, prompting the user to collect more data or change the model. This increases trustworthiness in high-stakes applications like fraud detection or medical diagnostics.

We present the mathematical foundations of linear independent component analysis (ICA) models based on standard literature in a self-contained note. It is aimed at readers with a background in measure-theoretic probability theory. We first develop the theory of the characteristic functions of probability measures on R d, including their analyticity and the way in which they determine and characterise the distributions. We then focus on several identifiability results of ICA models with successively strengthened assumptions on the sources: from merely non-constant, to non-Gaussian, to Gaussian-free independent sources. Under the strictest assumptions, we show that the independent sources are identifiable up to translation, permutation, scales and signs, and this even in the presence of additive Gaussian noise. Furthermore, we present the online equivariant gradient descent ICA algorithm for recovering the independent sources from data, in the standard complete noiseless non-Gaussian ICA setting.