Sanity Checking Causal Representation Learning on a Simple Real-World System
cs.LG, cs.AI, stat.ME
Submitted: 2025-02-27
Updated: 2026-09-14
Comments: 24 pages, 12 figures
Code: https://github.com/simonbing/CRLSanityCheck
License: http://creativecommons.org/licenses/by/4.0/
The gist: We evaluate methods for causal representation learning (CRL) on a simple, real-world system that satisfies the basic problem setup of CRL.
Terminology
Abstract
We evaluate methods for causal representation learning (CRL) on a simple, real-world system that satisfies the basic problem setup of CRL. The system consists of a controlled optical experiment producing a variety of measurements where the underlying causal factors---the control inputs to the experiment---are known, providing a ground truth. We select methods representative of different approaches to CRL and find that they all fail to consistently recover the underlying causal factors. To understand the failure modes of the evaluated algorithms, we perform an ablation on the data by substituting the real data-generating process with a simpler synthetic equivalent. The results reveal a reproducibility problem, as most methods already fail on this synthetic ablation despite its simple data-generating process. Additionally, we observe that common assumptions on the mixing function are crucial for the performance of some of the methods but do not hold in the real data. Our efforts highlight the contrast between the theoretical promise of the state of the art and the challenges in its application. We hope the benchmark serves as a simple, real-world sanity check to further develop and validate methodol- ogy, bridging the gap towards CRL methods that work in practice.
Sources
- Identifying General Mechanism Shifts in Linear Causal Representations
- Nonparametric Partial Disentanglement via Mechanism Sparsity: Sparse Actions, Interventions and Sparse Temporal Dependencies
- Partial Disentanglement via Mechanism Sparsity
- Score-based Causal Representation Learning with Interventions
- Marrying Causal Representation Learning with Dynamical Systems for Science
- Unifying Causal Representation Learning with the Invariance Principle
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: Local-Consistency Optimisation for Bayesian Neural Networks
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks