Beyond a Bag of Features: Set-Level Instability in Sparse Autoencoders

This paper investigates whether the interpretability of individual sparse autoencoder (SAE) latents extends to the set-level analysis of active latent signatures, testing the bag-of-features hypothesis for SAE representations.

Motivation and Research Questions

The authors revisit the work of Shani et al. (2026), who showed that LLM representations broadly recover human category boundaries while failing to reflect fine-grained typicality structure. Shani et al. used cosine similarity over dense model representations. The authors replace this with overlap over active SAE latent sets as a more interpretable similarity measure, asking whether this changes the picture. They also investigate whether SAE features compose via simple bag-of-features semantics outside idealized settings.

Preliminary Studies

The authors first test whether SAE signatures are meaningful units of analysis. In a controlled toy-model study using synthetic residual-stream states generated from known ground-truth feature directions, they find that sufficiently wide SAEs can satisfy union consistency—the property that composing two features yields a signature close to the union of the individual feature signatures. Results show high Jaccard, recall, and precision between the composed signature and the expected union. This holds across various configurations, including a larger scaling setup (D=1000) and a setup where ground-truth directions are sampled from a trained Gemma-Scope SAE decoder dictionary. In the latter case, union consistency is typically recovered for sufficiently wide SAEs, though the strongest failure mode appears at intermediate widths.

On natural text, the authors sample 30,000 snippets from The Pile and construct local neighborhoods using a SAE latent-set overlap metric. The resulting neighborhoods are often semantically interpretable, grouping contexts related to topics such as political conferences, mathematical proofs, and proper noun disambiguation.

Experiment 1: Typicality Judgements

Using the human-concepts dataset from Shani et al. (2026), the authors evaluate whether SAE set similarities recover human category boundaries and within-category typicality. They compute Jaccard similarity between word-level SAE signatures and compare against cosine similarity over dense representations.

Results show that SAE activation sets do not recover human category boundaries more reliably than dense residual-stream representations. The highest SAE-based AMI (Adjusted Mutual Information) remains below the highest embedding-based AMI. For within-category typicality, human typicality rankings and SAE set-based similarity scores are essentially uncorrelated, with per-category Spearman rank coefficients fluctuating around zero for most categories.

Notably, SAE-based Jaccard similarities more closely track model-internal similarity structure, as shown by positive correlations between SAE-based and dense cosine similarity rankings. However, since these correlations are not one... SAE activations appear to capture a related but partially distinct structure rather than simply reproducing dense-state similarity.

Experiment 2: Set Consistency under Semantically Related Inputs

The authors construct sequences of increasingly specific prompts by prefixing adjectives to nouns (e.g., shovel → yellow shovel), testing whether adding compatible adjectives preserves the base noun's active latents while adding new ones—the union-of-properties hypothesis.

Results show that lost-latent rates are substantial, typically between 20% and 60%, increasing with k and often also with model depth. Adding a semantically compatible adjective can deactivate a large fraction of latents that were active for the less specific noun prompt, directly contradicting the union-of-properties expectation.

Investigating the mechanism, the authors find that a substantial subset of lost latents was previously recoverable from the computation, but disappears at layer l at the noun position where loss is measured. The fraction of lost latents active upstream varies with the layer at which the latent is lost, as it is low in early layers, rises to roughly 60% in middle layers, and decreases again in later layers.

Analyzing pre-activations, for previously active lost latents evaluated at the loss layer, the negative fraction is relatively small, typically 10%–20%, while the near-zero fraction is larger and tends to increase in later layers. On the noun signature specifically, around 80% of latents fall into the near-zero regime, with only few negative latents, suggesting latents are removed, rather than suppressed by a negative component. For never-previously-active lost latents, both near-zero and negative fractions are generally low, indicating most such latents do not follow a simple pattern of being persistently near zero or persistently in the negative regime.

The authors consider alternative explanations. Geometric interference from non-orthogonal latents is tested via controlled steering experiments; results suggest positive-latent interference is a plausible contributing factor, but does not offer a sufficient explanation of the lost-latent rates observed under real inputs. Feature absorption is also considered incomplete, as the qualitative trends are broadly consistent across SAE types, including Matryoshka SAEs designed to mitigate absorption. Auto-interpretability analysis shows lost latents typically receive slightly lower auto-interpretability scores than latents active in both t0 and t1, providing weak evidence that lost latents are less well captured by current automatic explanations.

Conclusion

The authors conclude that active latent sets reveal meaningful structure, but their dynamics are context-sensitive and model-internal, rather than straightforwardly equivalent to human concepts or symbolic semantic feature sets. The results constrain a naive bag-of-features interpretation of SAE signatures. While SAE signatures can be meaningful in controlled settings and produce coherent semantic neighborhoods in natural text, they do not align with human conceptual structure better than dense representations, and their compositional behavior violates simple union-of-properties expectations under semantically compatible refinements.

Limitations

The authors note their conclusions are tied to the datasets, semantic constructions, and SAE families considered here. The typicality experiments rely on human psychology datasets limited to their concepts and items. The lost-latent experiments study only adjective modification of nouns and may not directly generalize to paraphrases, task-relevant abstractions, or other forms of semantic consistency. The analysis focuses primarily on residual-stream SAEs and does not exhaustively evaluate other hook locations or SAE families such as BatchTopK or gated SAEs.

Improvement 1: Context-Aware Feature Composition Module

The improved AI system can dynamically modulate its internal feature activations based on input context, rather than assuming static bag-of-features semantics. Specifically, it will implement a gating mechanism that detects when adding semantically compatible modifiers (e.g., yellow shovel) should suppress or transform existing active latents, mimicking human-like context sensitivity. This allows the system to avoid over-activating irrelevant features during compositional tasks, improving performance in fine-grained reasoning (e.g., visual question answering with color/shape modifiers) and reducing hallucinated attribute combinations.

Improvement 2: Human-Aligned Similarity Calibration

The improved AI system can adjust its internal similarity metrics (e.g., for retrieval or clustering) to better match human category boundaries and typicality judgments. By learning a correction layer on top of SAE-based Jaccard similarities—trained on human typicality rankings—the system can down-weight model-internal quirks (e.g., lost-latent phenomena) and up-weight features that align with human conceptual structure. This enables more intuitive search results, better few-shot classification on human-defined categories, and more natural language generation when ranking candidate responses by semantic closeness.

Improvement 3: Compositional Robustness via Latent Persistence Monitoring

The improved AI system can track the stability of its active latent sets across sequential inputs (e.g., noun → adjective+noun) and flag cases where >20% of latents are unexpectedly lost. It will then trigger a fallback mechanism—such as re-encoding the input with a wider SAE or switching to dense-representation similarity—to avoid brittle behavior. This makes the system more reliable in multi-step reasoning tasks (e.g., planning, dialogue state tracking) where maintaining consistent semantic features across turns is critical, reducing errors from feature dropout.

Improvement 4: Hybrid Similarity for Interpretable Decision-Making

The improved AI system can combine SAE set-overlap and dense cosine similarity with a learned weighting, based on the finding that SAE similarities capture partially distinct structure. This hybrid metric will be used in explainable AI interfaces, where the system can show both feature overlap and vector similarity explanations for its decisions. This allows users to see when model-internal structure diverges from human intuition, improving trust and enabling better human oversight in high-stakes applications (e.g., medical diagnosis support, legal document analysis).

Improvement 5: Adaptive SAE Width Selection

The improved AI system can dynamically choose the optimal SAE width (e.g., from a set of trained models) based on the input's semantic complexity, leveraging the finding that union consistency holds only for sufficiently wide SAEs. For simple inputs (e.g., single nouns), it uses narrower SAEs for efficiency; for complex compositional inputs (e.g., multi-property descriptions), it switches to wider SAEs to preserve feature unions. This reduces computational cost while maintaining interpretability and accuracy in tasks like semantic parsing or knowledge graph construction.