CORTEG: Foundation Models Enable Cross-Modality Representation Transfer from Scalp to Intracranial Brain Recordings

summary

The gist

The gist The CORTEG framework introduces a cross-modality transfer method that adapts large pretrained EEG foundation models to ECoG, enabling efficient, cross-patient learning for intracranial brain

In short

CORTEG adapts large, pretrained EEG foundation models to ECoG for cross-patient learning in brain-computer interfaces. It combines a pretrained EEG backbone with electrode-aware spatial adapters and a dual-stream tokenizer to process both low-frequency and high-gamma signals. This allows the model to achieve state-of-the-art performance on regression tasks while enabling rapid, data-efficient calibration for new patients.

Key concepts

EEG FM Backbone: This is a large, pretrained foundation model specifically trained on scalp EEG data. It learns general temporal representations of brain activity patterns. This pretraining provides a strong starting point for understanding neural dynamics across different recording modalities.
KNNSoftFourier Spatial Adapter: This component maps the physical locations of ECoG electrodes onto the learned representation space of the pretrained EEG model. It ensures that the model can effectively utilize electrode-specific spatial information from ECoG recordings by aligning it with its existing knowledge base.
Dual-Stream Tokenizer: Raw ECoG signals are split into two streams: low-frequency signals and high-gamma activity. The tokenizer processes these separately, allowing the model to leverage the EEG FM's strength in handling low frequencies while incorporating the high-gamma dynamics crucial for decoding intracranial data.
Leave-One-Subject-Out Fine-Tuning (LOOFT): This is a parameter-efficient strategy where a population model is adapted to a single new patient by fine-tuning only a small subset of parameters. This allows the system to calibrate to an individual in minutes, using existing knowledge from many patients without needing extensive retraining.

Terminology used across episodes

Reinforcement Learning (RL) — explained in 25 episodes
Sample Complexity — explained in 19 episodes
Federated Learning — explained in 17 episodes
Retrieval-Augmented Generation (RAG) — explained in 17 episodes
Quantum Fisher Information (QFI) — explained in 16 episodes
Interpretability — explained in 15 episodes

This episode discusses

The paper

CORTEG: Foundation Models Enable Cross-Modality Representation Transfer from Scalp to Intracranial Brain Recordings · Read on arXiv

Liuyin Yang, Qiang Sun, Bob Van Dyck, Eva Calvo Merino, Marc M. Van Hulle

Laboratory for Neuro- & Psychophysiology, Department of Neurosciences, KU Leuven

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "CORTEG: Foundation Models Enable Cross-Modality Representation Transfer from Scalp to Intracranial Brain Recordings".

Jane: The gist The CORTEG framework introduces a cross-modality transfer method that adapts large pretrained EEG foundation models to ECoG, enabling efficient, cross-patient learning for intracranial brain interfaces.

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So we’re wrapping up on this paper, "CORTEG: Foundation Models Enable Cross-Modality Representation Transfer from Scalp to Intracranial Brain Recordings." It’s about taking those large, general scalp EEG models and making them work for brain recordings from inside the skull, ECoG.

Jane: It’s really neat how they tackle that modality gap by using a cross-modality transfer framework to bridge the difference between what you see on the scalp and what's happening deep inside.

Lu: The authors used a pretrained EEG foundation model as a backbone and then hooked it up with an electrode-aware spatial adapter to map those ECoG coordinates into that learned space.

Meng: From my side, I’m looking at how they handle the data streams, decomposing the ECoG signal into low frequency stuff and high gamma activity. That’s important for deployment because you can’t just feed raw data in there.

Lalam: What I see is a dual-stream tokenizer that handles both those low frequency signals and the high gamma activity separately before they get merged for decoding.

Tom: And they fine-tuned this whole thing using a parameter-efficient strategy called leave-one-subject-out with LoRA to adapt it to a new patient really fast. That’s what makes it practical.

Jane: The results show that this setup achieves state of the art mean performance on both their regression tasks, which were finger trajectory and audio envelope decoding.

Lu: They found that the pretraining from scalp EEG actually gave them gains over just starting with a random model architecture for ECoG.

Meng: But I’m watching those numbers too. The gain is bigger on the audio task and when you only have a small number of subjects to calibrate to, which is where I care about deployment at scale.

Tom: It means we don't have to build entirely new decoders from scratch for every single patient; we can repurpose existing knowledge.

Jane: So what does this mean for us? It suggests that pretraining on one type of brain signal could actually be a really powerful starting point for understanding another, different recording modality.

Conclusion: Tom: So we’re wrapping up on CORTEG, which is about taking those big, general scalp EEG models and making them work for brain recordings from inside the skull, ECoG.

Lu: The authors used a pretrained EEG foundation model as a backbone and then hooked it up with an electrode-aware spatial adapter to map those ECoG coordinates into that learned space.

Lalam: What I see is a dual-stream tokenizer that handles both those low frequency signals and the high gamma activity separately before they get merged for decoding.

Tom: And they fine-tuned this whole thing using a parameter-efficient strategy called leave-one-subject-out with LoRA to adapt it to a new patient really fast. That’s what makes it practical.

Jane: The results show that this setup achieves state of the art mean performance on both their regression tasks, which were finger trajectory and audio envelope decoding.

Lu: They found that the pretraining from scalp EEG actually gave them gains over just starting with a random model architecture for ECoG.

Tom: It means we don't have to build entirely new decoders from scratch for every single patient; we can repurpose existing knowledge.

Lu: The latent representations they found suggest the pretrained model learns a lower-dimensional structure for finger movements, which is more compact than what random training would yield.

Meng: That compression is good because it means the model needs fewer parameters to learn from just one new patient’s data.

Tom: We still have limitations on things like MNI space localization and they didn't show how well it transfers across different types of tasks.

Jane: It’s a solid step, definitely showing that scalp EEG pretraining isn't just some isolated technique; it can be repurposed for intracranial recording decoding in a way that makes adaptation much more efficient.

More episodes

← Home