Exploring Sparse Autoencoders in Text-Based Causal Confounding Adjustment

arXiv:2609.01322 · cs.CL, cs.LG · Submitted 2026-09-01 · Read on arXiv

cs.CL, cs.LG

Submitted: 2026-09-01

Updated: 2026-09-01

Comments: Long paper accepted at EMNLP 2026 main conference, 25 pages, 16 figures

Code: https://github.com/mianzg/sae-text-confounder

License: http://creativecommons.org/licenses/by/4.0/

The gist: In many settings, studying causal questions based on text data requires adjusting for confounding information within texts.

Terminology

Abstract

In many settings, studying causal questions based on text data requires adjusting for confounding information within texts. Yet there is a tradeoff in constructing text representations for adjustment: they must be sufficiently large and/or dense to preserve the confounding variables necessary for unbiased effect estimation, but sufficiently small and/or sparse to satisfy finite-sample overlap and yield low-variance estimates. To address this tradeoff, we turn to sparse autoencoders (SAEs), and propose a novel causal adjustment pipeline that iteratively selects a minimal set of SAE features via conditional independence tests. We find that SAE representations achieve better adjustments (lower bias and and higher coverage) than alternative representations in standard semi-synthetic evaluations with binary confounders, and their interpretability offers opportunities for falsification. We also introduce a more realistic semi-synthetic evaluation that uses multi-label data as the unobserved confounders and find off-the-shelf adjustment methods require increased investigation for these more complex settings. Code: https://github.com/mianzg/sae-text-confounder

Related papers