Interpretable Embeddings of Speech Enhance and Explain Brain Encoding Performance of Audio Models

Introduction to the show: ident: Genomics Radio. Generated commentary on the latest computational biology and genomics papers.

Ines: Hi, I'm Ines.

Marcus: And I'm Marcus, here with Yuki.

Ines: We're looking at the paper: "Interpretable Embeddings of Speech Enhance and Explain Brain Encoding Performance of Audio Models".

Marcus: Here's the gist of the paper. The gist Speech foundation models (SFMs) are increasingly hailed as powerful computational models of human speech perception, but their representations are inherently black-box, so this study built linear encoding models from six interpretable feature families and contextualized embeddings from three state-of-the-art SFMs to quantify electrocorticography (ECoG) response variance shared between feature classes; How it works The researchers constructed linear encoding models from six interpretable feature families: mel-spectrogram, Gabor filter bank features, speech presence, phonetic, syntactic, and semantic features; They also incorporated contextualized embeddings from three state-of-the-art SFMs (Whisper, HuBERT, WavLM) to quantify electrocorticography (ECoG) response variance shared between feature classes; Variance-partitioning analyses revealed several key insights regarding this comparison; Key Findings from Variance Partitioning The variance partitioning analyses revealed three main findings concerning the alignment of SFMs with the brain; First, "the SFMs’ alignment with the brain can be mostly explained by their ability to learn and encode simple interpretable speech features"; Second, "SFMs exhibit a systematic trade-off between encoding of brain-relevant low-level and high-level features across layers"; Finally, "our results show that SFMs learn brain-relevant semantics which cannot be explained by lower-level speech features, with this capacity increasing with model size and context length"; Analysis of Interpretable Features The study derived six families of interpretable predictors reflecting the language processing hierarchy in speech comprehension; These features included mel-spectrograms, Gabor filter bank (GBFB) features, speech presence, phonetic features, syntactic features, and semantic features; Each feature explained unique neural variance relative to the others; For instance, "the mel-spectrogram" feature set showed a ratio that peaked in early layers for HuBERT (0.90) and WavLM (0.80), and at layer 5 for Whisper (0.79); In contrast, "phonetic features, syntactic, and semantic features" exhibited a more gradual rise with layer depth and peaked in later layers than phonetic features; Trade-offs in Encoding Dynamics The results indicated a trade-off between low-level and high-level feature encoding across SFM representations; Specifically, "gains in high-level representations emerge at the expense of lower-level features, rather than through a simple monotonic accumulation of information"; This was observed because "Whisper and WavLM progressively lose fine acoustic detail but expand their temporal receptive windows with depth"; Conversely, "HuBERT’s acoustic representations simply degrade in later layers"; Evidence for Semantic Encoding The study demonstrated that SFMs encode brain-relevant semantics that cannot be explained by lower-level speech features; This was quantified by showing that "SFM representations significantly reduce the unique variance explained by the semantic embedding, providing strong evidence that they indeed encode brain-relevant semantic information"; Furthermore, "the increase

in unique neural variance: with model size" suggests that larger models encode more brain-relevant semantic information; The findings suggest a principled approach to building more interpretable, accurate, and efficient encoding models of the brain by augmenting SFM embeddings with interpretable features; This strategy mitigates the limitations of relying on individual layers by integrating complementary information from hand-crafted features while still benefiting from the representational advantages conferred by scaling; The unique neural variance explained by SFMs in later layers is primarily localized to the superior temporal gyrus and inferior frontal gyrus, and increases with model size; This suggests that "the remaining unique variance in later layers likely reflects high-level, non-linear representations that are not captured by the current interpretable feature sets"; The research concludes that incorporating interpretable features consistently improves encoding performance across all layers; Limitations The study's findings are based on data from nine clinical ECoG participants who listened to a single 30-min English podcast; Replication using larger and more diverse datasets will be important for assessing the generalizability of these results; Additionally, the researchers acknowledge that "a statistically significant portion remains unexplained" by interpretable features, and identifying the nature of this residual variance is an important direction for future research; While SFMs account for only about 40% of the unique variance explained by semantic features in IFG, with respect to non-semantic features, indicating that their semantic representations remain partial; The researchers also restricted their analyses to SFMs because "AudioLLMs are primarily optimized for interactive or task-specific use cases, such as question-answering or dialogue, rather than for producing stable, general-purpose audio features"; Broader Impact Statement This study relies on electrocorticography (ECoG) data collected from patients with epilepsy who underwent invasive monitoring for clinical purposes; The use of such data requires careful attention to ethical considerations, including the assurance that appropriate informed consent and clinical oversight were in place during data collection, and that secondary analyses such as ours respect the dignity and autonomy of the participants; This work advances the understanding of how SFMs encode speech information relevant for neural processing; The findings suggest room for improvement, as "fine-tuning SFMs on fMRI data can improve their alignment with language regions of the brain"; The researchers also showed that "augmenting SFM representations with interpretable features significantly improves brain encoding model performance"; This strategy provides a more compact and interpretable representation of the same information, making them a principled complement to SFM embeddings; The study ultimately shows that "the later layers of SFMs are the most useful for capturing components of neural variance that cannot be explained by interpretable counterparts"; This paper demonstrates that while the vast majority of neural variance captured by speech foundation models (SFMs) is attributable to interpretable features, nearly all in the earliest layers, still a substantial portion (approximately 80%) in the deeper layers; The unique neural variance explained by SFMs—beyond what is accounted for by interpretable features—is primarily localized to the superior temporal gyrus and inferior frontal gyrus, and increases with model size; The researchers found a systematic trade-off where gains in high-level representations emerge at the expense of lower-level features across layers; Furthermore, they confirmed that SFMs encode brain-relevant semantics that cannot be reduced to lower-level features, and this capacity improves with model size and context length; They conclude that combining interpretable feature sets with the later layers of large SFMs offers a principled approach for building more interpretable and accurate brain encoding models; The overall conclusion is that augmenting SFM representations with interpretable features significantly improves brain encoding model performance; The paper provides a framework suggesting that the residual variance in deeper layers reflects high-level, non-linear representations not captured by current interpretable feature sets; Page 1 --- The gist Speech foundation models (SFMs) are increasingly hailed as powerful computational models of human speech

Paper summary: Ines: So we're talking about this paper called "Interpretable Embeddings of Speech Enhance and Explain Brain Encoding Performance of Audio Models." It looks like the main thing they’re trying to do is take these speech foundation models, which are really good at understanding human speech, and figure out what makes them align with brain activity.

Marcus: Right. The core idea is that these models are black boxes, so we need a way to see what's actually happening inside them when they relate to how the brain responds. They built linear encoding models using six different sets of interpretable features and compared those against embeddings from three big speech foundation models: Whisper, HuBERT, and WavLM.

Yuki: From a population genetics perspective, I wonder if this comparison helps us understand what specific acoustic or linguistic patterns are being prioritized by these large models in relation to human hearing.

Ines: Exactly. They claim that their variance partitioning analyses show three main things: first, the alignment with the brain is mostly explained by how well the models learn and encode simple interpretable speech features; second, there's a systematic trade-off between encoding low-level and high-level features across different layers of these models; and finally, they found that these models learn brain-relevant semantics that you can't explain just by looking at lower-level speech features, with this capacity actually getting bigger as the model gets larger or the context gets longer.

Marcus: That trade-off part is interesting because it suggests that when you go deeper into a model, you’re trading some of those fine acoustic details for broader information.

Ines: Right. The paper goes on to detail these six interpretable feature families they used: mel-spectrograms, Gabor filter bank features, speech presence, phonetic features, syntactic features, and semantic features. Each one explained unique neural variance relative to the others; for example, the mel-spectrogram feature set showed a ratio that peaked in early layers for HuBERT and WavLM at zero point nine zero and zero point eight zero respectively; but at layer five for Whisper it was around zero point seven nine.

Paper summary: Yuki: So you're saying the basic sound structure is important early on, but as the model gets deeper, it starts focusing more on the grammar and meaning of what’s being said?

Ines: That seems to be the direction they're pointing; phonetic features, syntactic features, and semantic features showed a more gradual rise with layer depth compared to just phonetic features.

Marcus: And that leads into another point they make about encoding dynamics. They found a trade-off where gains in high-level representations emerge at the expense of lower-level features rather than just information accumulating simply as you go deeper.

Yuki: So, when Whisper and WavLM progressively lose fine acoustic detail but expand their temporal receptive windows with depth, that’s what causes that trade-off?

Ines: Precisely; while HuBERT’s acoustic representations actually just degrade in later layers, the other two models show this shift toward high-level information.

Marcus: That brings us to the evidence for semantic encoding. They showed that SFMs encode brain-relevant semantics that lower-level speech features simply can't explain, and they quantified this by showing that SFM representations significantly reduce the unique variance explained by the semantic embedding.

Yuki: So, even though they are powerful models, their ability to capture actual meaning—the semantics—is something separate from just recognizing phonemes or syllables.

Ines: Yes, and it gets stronger with model size; the increase in unique neural variance with model size suggests that larger models encode more brain-relevant semantic information.

Marcus: This means if you're building a bigger foundation model, you’re getting a better capture of what the brain actually cares about in terms of meaning.

Yuki: So, what does this mean for us studying language processing across different populations? Does it suggest that the semantic structure learned by these models is robust enough to reflect deeper cognitive organization?

Ines: It suggests a principled way to build better encoding models of the brain by adding these interpretable features onto the SFM embeddings. This strategy helps mitigate the limitation of only looking at individual layers by integrating complementary information from those hand-crafted features while still using the representational advantages that come from scaling up.

Paper summary: Marcus: So, what's their final verdict on this approach? What are they suggesting we do with this work?

Ines: They suggest augmenting SFM representations with interpretable features to build more accurate and efficient brain encoding models. They also point out that the unique neural variance in the later layers of SFMs is primarily localized to the superior temporal gyrus and inferior frontal gyrus, and this variance increases with model size.

Yuki: That localization helps narrow down where we should be looking for these high-level representations in human brain studies.

Marcus: And they also admit a limitation on their study setup. They based this work on data from nine clinical ECoG participants who listened to a single thirty-minute English podcast, and they note that replication using larger and more diverse datasets will be important for assessing the generalizability of these results <ref:2507.16080#pg1>.

Ines: Right, and they also flag that while SFMs account for only about forty percent of the unique variance explained by semantic features in the IFG compared to non-semantic features, indicating their semantic representations are still partial.

Marcus: So what's the big picture here? They’re suggesting that combining these interpretable feature sets with the later layers of large SFMs provides a framework for building more interpretable and accurate brain encoding models.

Ines: It essentially shows that while most of the neural variance captured by speech foundation models is due to interpretable features, there's still about eighty percent in the deeper layers that needs explaining, and that residual variance is what's localized to those specific areas of the brain.

Yuki: That residual variance might be where we need to focus our next generation of population studies if we want to really connect these models back to species-level language organization.

Marcus: So, the main conclusion here is that augmenting SFM representations with interpretable features significantly improves brain encoding model performance, and it gives us a more compact representation of the same information.

Conclusion: Ines: So we've seen how they built these linear encoding models using six different feature families and compared them to embeddings from Whisper, HuBERT, and WavLM to see how well they match electrocorticography data.

Marcus: Right. It boils down to the idea that these massive speech models aren't just learning random noise; they’re actually picking up on specific patterns in the sound data that correlate with brain activity.

Ines: The paper's title is "Interpretable Embeddings of Speech Enhance and Explain Brain Encoding Performance of Audio Models." It tells us they built these linear models to make the black box more transparent.

Marcus: And it claims that by adding these interpretable features—like phonetic or semantic ones—they can explain a lot more of what the speech model is doing, especially in those deeper layers.

Ines: What this means for me as a computational biologist is that we can start to pinpoint exactly which parts of speech processing are driving neural responses, instead of just guessing.

Marcus: For my work with cohorts, it suggests that if you want to understand how different groups process language, you need those interpretable features because the raw embeddings alone don't tell the whole story about the brain interaction.

Yuki: From a population genetics view, if these models are picking up on semantics that scale with model size or context length, it implies that the species might be organizing its language processing in a way that matches this complexity.

Ines: That’s what they found—the semantic capacity grows with the model. It suggests the brain isn't just reacting to simple acoustic sounds; it’s engaging with complex meaning structures.

Marcus: But there are caveats, and they mention that while these models explain a lot, they still don't account for everything in those deeper layers of neural variance.

Ines: Yeah, so the paper suggests augmenting the speech foundation model embeddings with these interpretable features is a principled way to get a more accurate picture of brain encoding.

Marcus: It provides this clearer representation that’s still benefiting from the scaling power of the models while making it more understandable for us to use in our own studies.