InfoAtlas: A Foundation Model for Zero-Shot Statistical Dependence Estimate
cs.LG, cs.AI, stat.ML
Submitted: 2026-05-29
Updated: 2026-09-12
Comments: Accepted to ICML 2026
License: http://creativecommons.org/licenses/by/4.0/
The gist: Measuring statistical dependency between high-dimensional random variables is a fundamental task in data science and machine learning.
Terminology
Abstract
Measuring statistical dependency between high-dimensional random variables is a fundamental task in data science and machine learning. Neural mutual information (MI) estimators offer a promising avenue, but they typically require costly iterative optimization for each new dataset, making them impractical for real-time applications. We present InfoAtlas, a foundation model-like architecture that eliminates this bottleneck by directly inferring MI in a single forward pass. Pretrained on large-scale synthetic data with rich dependence patterns, InfoAtlas learns to identify diverse dependence structures and predict MI directly from the dataset. Comprehensive experiments demonstrate that InfoAtlas matches state-of-the-art neural estimators in accuracy while achieving 100 times speedup, can flexibly handle varying dimensions and sample sizes through a single unified model, and generalizes effectively to complex, real-world scenarios. By reformulating MI estimation as an inference task, InfoAtlas establishes a foundation for real-time dependency analysis.
Sources
- Chronos: Learning the Language of Time Series
- Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
- Beyond Normal: On the Evaluation of Mutual Information Estimators
- On the Properties and Estimation of Pointwise Mutual Information Profiles
- Information Bottleneck Analysis of Deep Neural Networks via Lossy Compression
- Density estimation using Real NVP
- Microsoft COCO Captions: Data Collection and Evaluation Server
- Neural Approximate Sufficient Statistics for Implicit Models
- MINDE: Mutual Information Neural Diffusion Estimation
- Representation Learning with Contrastive Predictive Coding
- ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skills
- InfoNet: Neural Estimation of Mutual Information without Test-Time Optimization
- Perceiver IO: A General Architecture for Structured Inputs & Outputs
- Solving Empirical Bayes via Transformers
- On Mutual Information Maximization for Representation Learning
- Effortless, Simulation-Efficient Bayesian Inference using Tabular Foundation Models
- PointOdyssey: A Large-Scale Synthetic Dataset for Long-Term Point Tracking
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: Local-Consistency Optimisation for Bayesian Neural Networks
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks