Leakage-Audited Benchmarking Reveals Limited Evidence for Cross-Subject Auditory-Evoked EEG Vowel Perception Decoding

arXiv:2605.00865 · eess.SP, cs.CL, cs.CV, cs.LG, cs.SD, q-bio.NC · Submitted 2026-04-22 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "Leakage-Audited Benchmarking Reveals Limited Evidence for Cross-Subject Auditory-Evoked EEG Vowel Perception Decoding".

Jane: The paper was written by the authors from Northeastern University.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Jane: We also have Lu with us today — senior AI researcher at Tsinghua.

Tom: We also have Meng with us today — lead engineer at a mysterious AI startup.

Jane: We also have Lalam with us today — the in-house Large Language Model.

Tom: Alright, let's get started.

Title and Authors: Tom: Alright, welcome back to the show, everyone. Today we’re digging into a paper with a title that’s a mouthful but really says it all: “Leakage-Audited Benchmarking Reveals Limited Evidence for Cross-Subject Auditory-Evoked EEG Vowel Perception Decoding.” Jane, when you first read that title, what jumped out at you?

Jane: Oh, Tom, the word “leakage-audited” is the star of the show for me. It tells you right away that these authors, Xiaoyang Li and Zeyan Tao from Northeastern University, weren’t just running another model comparison. They were checking their own homework, making sure no information from the test participants snuck into the training process.

Tom: And that’s so important because in EEG research, especially with something like decoding vowels from brain waves, it’s really easy to accidentally cheat. You might think your model is reading the person’s mind, but it’s actually just memorizing the background noise or the specific way one participant blinks.

Jane: Exactly. And the title also says “limited evidence,” which is a pretty bold statement. They’re basically saying, look, we tried really hard, we used a bunch of different models, and we still couldn’t find strong proof that a computer can look at one person’s brain activity and reliably tell which of five vowels they’re hearing, without ever having seen that person before.

Tom: That’s the “cross-subject” part, right? It’s the hardest challenge. It’s one thing to train a model on your own brain and have it work on your own brain. It’s a whole other thing to train it on fifteen people and have it work on a sixteenth person they’ve never met.

Jane: Right, and that’s what makes this paper so valuable. It’s not a flashy “we built a mind-reading machine” story. It’s a careful, honest investigation that says, here’s the evidence, and it’s not looking great for that particular dream right now.

Tom: So the authors are essentially the referees here, making sure the game is played fair. And their verdict is that, with this dataset and this protocol, the evidence for cross-subject vowel decoding is pretty thin. That’s a big deal for the field, because it sets a new standard for how these studies should be reported.

Jane: It really does. And it makes you wonder, if the results are this weak when you do everything right, what were all those earlier, more optimistic studies actually measuring? That’s the question we’re going to dig into next.

Summary of the Paper: Tom: So, Jane, we’ve set the stage. Now let’s get into the meat of what these researchers actually did. They took a public dataset, OpenNeuro ds006104, and they were incredibly meticulous about how they counted their trials. Can you break that down for our listeners?

Jane: I’d love to, Tom. The first thing they did was go through the raw event files. There were over twenty-one thousand rows of event markers. But not all of those were usable trials. They had to pair up a “marker” row with a “stimulus” row to make one single trial. So seven thousand six hundred eighty rows became three thousand eight hundred forty actual trials.

Tom: And then they had to decide which trials were even eligible. The dataset has active TMS conditions, where they’re stimulating the brain, and control conditions. They only used the control trials, which left them with one thousand two hundred eighty eligible trials.

Jane: Then they applied an artifact rejection rule. If any EEG channel had a peak-to-peak amplitude over four hundred microvolts, they tossed the trial. That removed one hundred eighty-six more, leaving them with one thousand ninety-four clean epochs from sixteen participants.

Tom: So they went from a mountain of raw data to a pretty focused pile of usable brain signals. And then they ran thirteen different machine learning models on it, all trying to classify which of five vowels — a, e, i, o, u — the person was hearing.

Jane: And here’s the kicker, Tom. The best model, a Random Forest, only hit twenty-one point four seven percent balanced accuracy. Now, pure chance would be twenty percent. So it’s above chance, but barely, and statistically, after correcting for testing thirteen models, it’s not significant at all.

Tom: That’s a huge reality check. I mean, a one point five percent improvement over chance sounds like the model is grasping at straws, not actually decoding vowels. And it wasn’t just one model. The deep learning models, which are all the rage, they performed right around chance too.

Jane: And the deep models had another problem. They were really unstable. If you ran the same architecture with a different random seed, you could get wildly different results. The paper showed that for some models, the accuracy could swing by over ten percentage points just based on the random starting point.

Tom: So not only are the results weak, but they’re also fragile. You could easily pick the one seed that gives you a good-looking number and report that, and the paper is calling that out as a major issue.

Jane: Exactly. They’re saying, if your model’s predictions change that much just by changing the random seed, then your model isn’t learning a stable, reliable pattern. It’s just fitting to noise.

Tom: And that’s the core of their negative finding. The evidence for cross-subject vowel decoding, when you audit everything carefully, is just not there in this dataset. Now, what does that mean for the future? Let’s talk about what they suggest we do differently.

Improvements Suggested by the Paper: Tom: So we’ve heard the bad news, Jane. The models don’t work that well. But this paper isn’t just a downer. It’s actually a roadmap for how to do better. What are the key improvements they’re pushing for?

Jane: The biggest one, Tom, is about transparency and provenance. They want every single trial in a study to be traceable. You should be able to look at a result, click on it, and see exactly which raw data file and which event row that prediction came from. They built a whole “prediction ledger” with over thirty-six thousand individual predictions, all mapped to their source.

Tom: That’s like a chain of custody for brain data. It means anyone can audit the results, not just trust the summary numbers. And they also want to see all the models, not just the best one. They found that one model, EEGNet-FBCSP, was actually just an alias for another model, EEGNet. It was the same code with a different name.

Jane: Right, so if you don’t check for that, you might think you’re testing fourteen models when you’re really testing thirteen. That inflates your model count and messes up your statistical corrections. Their solution is to have a clear registry that maps every displayed name to the actual executable code.

Tom: And then there’s the issue of seeds. They’re adamant that deep learning studies should report results from multiple random seeds, not just the one that worked best. They showed that a model could have a stable average score but be making completely different predictions trial-by-trial depending on the seed.

Jane: That’s a really subtle point. Two runs of the same model could have the same average accuracy, but they might be getting different trials right and wrong. That means the model isn’t finding a consistent pattern in the brain signals. It’s just stumbling onto different random solutions that happen to score the same.

Tom: So what’s the practical takeaway for someone building a brain-computer interface? Is this a dead end?

Jane: Not a dead end, but a reality check. The paper suggests we need to be more careful about what we claim. They also ran an analysis where they increased the number of training participants, and performance didn’t go up in a nice, steady line. That suggests that just adding more people to the training set isn’t a magic bullet.

Tom: So it’s not just a data quantity problem. There might be something fundamental about how variable brain signals are between people that makes this specific task, five-vowel classification, really hard.

Jane: Exactly. And their sensor-space analysis backs that up. They found that the differences between participants were over thirty-five times larger than the differences between vowels. So the signal you’re trying to find is tiny compared to the “noise” of just being a different person.

Tom: That’s a powerful way to think about it. The model is trying to find a needle in a haystack, and the haystack is made of other people’s unique brain patterns. So what’s the next step for the field?

Jane: The paper calls for a decisive next study that freezes all the rules before looking at the data, and then tests on a completely new, external dataset. That’s the gold standard. We need to see if these findings hold up when you have a truly untouched group of participants.

Conclusion: Tom: Well, Jane, we’ve covered a lot of ground on this paper, “Leakage-Audited Benchmarking Reveals Limited Evidence for Cross-Subject Auditory-Evoked EEG Vowel Perception Decoding.” Let’s wrap it up for our listeners.

Jane: Absolutely, Tom. The core message is that when you do everything by the book, when you audit for leakage, when you report all your seeds, and when you correct for multiple comparisons, the dream of a universal, cross-subject vowel decoder from EEG just doesn’t hold up on this dataset.

Tom: It’s a negative result, but it’s a really important one. It’s going to save other researchers a lot of time and false hope. They won’t have to rediscover this dead end themselves.

Jane: And it sets a new bar for how to report results. The idea of a full prediction ledger, where every single trial prediction is available for scrutiny, is going to become the standard for trustworthy benchmarking in this field.

Tom: So we’re saying goodbye to this paper, but we’re taking its lessons with us. It’s not the end of the road for speech decoding from brain waves, but it’s a clear sign that we need to be more rigorous and more humble about what we can achieve right now.

Jane: Well said, Tom. It’s a tough finding, but a fair one. And it gives us a solid foundation for the next paper we’re going to discuss, which I hear is a bit more optimistic. Let’s get ready for that one.

Tom: Sounds good, Jane. Thanks to everyone for tuning in, and we’ll see you on the next episode.

Northeastern University

eess.SP, cs.CL, cs.CV, cs.LG, cs.SD, q-bio.NC

Submitted: 2026-04-22

Updated: 2026-09-01

Comments: 19 pages, 7 figures; includes 11-page supplementary material. Associated code, prediction records, source data, and reproducibility materials: https://doi.org/10.5281/zenodo.21805983

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 74/100

Key concepts

Leakage-Audited Benchmarking
This refers to a thorough process where researchers check their own work to ensure no information from test participants accidentally enters the training data. It is crucial for ensuring that model performance is genuine and not due to memorizing specific participant details.
Cross-Subject Auditory-Evoked EEG Vowel Perception Decoding
This is the task of using brain wave data (EEG) to determine which vowel a person was hearing, specifically across different individuals. The paper investigates whether a computer can do this reliably without having seen that specific person before.
Random Seed Instability
Deep learning models showed instability where changing the random starting point could cause results to swing by over ten percentage points. This indicates the model is not learning a stable pattern but is instead fitting to noise, suggesting unreliable predictions.
Prediction Ledger
The authors suggest creating a traceable record for every single prediction made by a model, linking it back to the exact raw data file and event row. This provides a chain of custody for auditing results and ensuring transparency.

Terminology

Summary

Summary

This paper presents a leakage-audited, reproducible benchmark evaluating whether auditory-evoked EEG supports subject-independent five-vowel perception decoding, using the OpenNeuro ds006104 version 1.0.1 dataset. The authors reconstructed all Study 2 event tables and analyzed the consonant–vowel (CV) pair task (task-phonemes). One-to-one pairing of each TMS marker with its following stimulus converted 7,680 event rows into 3,840 independent CV trials. Excluding 2,560 active-condition trials left 1,280 eligible control-condition trials; a 400 µV maximum-channel peak-to-peak rule rejected 186 epochs and retained 1,094 epochs from 16 participants and 61 EEG channels.

Thirteen unique implementations were evaluated by leave-one-subject-out (LOSO) testing. Participant metrics were reconstructed from 36,102 exported trial predictions spanning 33 complete prediction replicas. Diagnostic analyses quantified five-seed deep-model stability, class-resolved errors, and participant-level sensor-space geometry; post hoc analyses examined sensitivity to participant retention. An exploratory MDM analysis comprised 9,616 genuine refits across training cohorts of 3–15 participants.

The main results show that Random Forest was numerically highest at 21.474% balanced accuracy (95% participant-bootstrap interval 19.526–23.482%; five-class chance 20%; one-sided Wilcoxon p = 0.090897; Bonferroni-13 p = 1.000000; 10,000-resample sign-flip p = 0.092791). No implementation survived multiplicity correction. Deep-architecture means were near chance, while technical-seed choices produced substantial within-participant ranges and low trial-label agreement for several architectures. Random Forest recall varied from 9.6% for /e/ to 32.8% for /i/. In a separate descriptive representation, participant-associated effects comprised 72.24% of the balanced standardized centroid sum of squares, compared with 2.04% for vowel-associated effects; between-participant same-vowel distances exceeded within-participant across-vowel distances for all 16 participants. Across 3–15 training participants, genuinely refitted MDM means ranged from 20.553% to 20.922% and did not increase monotonically.

The significance statement concludes: "Within this dataset and protocol, the evidence for cross-subject five-vowel decoding is limited. The benchmark establishes a reusable evidence chain linking source rows to independent trials, retained epochs, executable implementations, prediction replicas, participant-level metrics, multiplicity-adjusted inference, and explicitly bounded diagnostic analyses."

The paper emphasizes that the central finding is that "this control-condition CV dataset provides limited evidence for subject-independent five-vowel decoding under the evaluated LOSO protocol. The most favourable mean, obtained by Random Forest, exceeded nominal chance by only 1.474 percentage points, its participant-bootstrap interval included 20%, and both the Wilcoxon and sign-flip analyses were non-significant. Although MDM-EA and CNN-BiLSTM produced nominal one-sided p values below 0.05, neither survived correction across the 13 unique implementations."

The sensor-space analysis showed that "in the balanced standardized centroid representation, the participant fraction of variation was more than 35 times the vowel fraction, and the same ordering persisted without standardization. Moreover, every participant was closer, on average, to their own other-vowel centroids than to other participants representing the same vowel."

The paper also notes that "Deep learning did not provide a stable alternative. Averaging five seeds placed every architecture close to chance, yet seed-specific architecture means differed by as much as 5.26 percentage points and median within-participant ranges were 5.2–10.4 percentage points. Trial-level agreement exposed an additional form of instability: median agreement across seed pairs was only 5.7–24.6%, even though each pair classified the same 1,094 trials."

The exploratory training-cohort analysis "does not establish that adding participants will monotonically improve this endpoint. Across 9,616 genuine MDM refits, the group mean increased by at most 0.369 percentage points relative to n = 3, every pointwise paired interval included zero, and the full n = 15 endpoint returned to 20.554%."

The paper concludes: "Within these limits, the study establishes both a negative empirical boundary and a positive reproducibility standard. The negative boundary is that near-chance means, class-dependent errors, seed-sensitive predictions, and non-significant participant-level inference do not support a reliable cross-subject five-vowel decoder in this setting. The reproducibility standard is an end-to-end chain in which every event row maps to a trial or explicit exclusion, every retained epoch has a stable source identity, every model label maps to an executable implementation, every replica covers the same canonical trials, every participant metric is reconstructed from predictions, and exploratory diagnostics remain separated from the primary hypothesis family. These conclusions should not be generalized to imagined speech, overt speech, natural continuous listening, active-TMS effects, or online communication BCI."

Improvements for AI systems

Based on the methodology and findings of this paper, I can implement the following specific improvements to AI systems, particularly for EEG-based decoding and general machine-learning pipelines:

Improvement: Implement a validation system that strictly separates participants at every stage—feature scaling, normalization, and model fitting—using leave-one-subject-out (LOSO) folds. The system will automatically detect and flag any trial-level leakage (e.g., scaling parameters estimated from held-out data) and refuse to run if leakage is present.

What the improved system can do: Guarantee that reported performance metrics are not inflated by subject information leaking into training. It will output a leakage audit report alongside every model evaluation, making it impossible to accidentally report optimistic results.

Improvement: Build a model-identity management layer that maps display names to canonical executable implementations. The system will detect when two different labels (e.g., EEGNet and EEGNet-FBCSP) resolve to the same code and automatically de-duplicate them before any statistical comparison.

Improvement: For any deep architecture, the system will require and report results from all prespecified random seeds (e.g., 42–46), not just the best or average. It will compute within-participant seed ranges and trial-label agreement across seed pairs, and flag architectures where seed choice materially changes conclusions.

Improvement: Implement a statistical layer that uses participants (not trials, seeds, or prediction rows) as the independent inferential unit. It will apply Bonferroni correction across the de-duplicated model family and report both raw and adjusted p-values, along with participant-bootstrap confidence intervals.

Improvement: Add a diagnostic module that outputs row-normalized confusion matrices, per-class recall with participant-level bootstrap intervals, and predicted-class marginal distributions. The system will flag when a model's balanced accuracy is driven by a single class (e.g., 33% of predictions assigned to one vowel).

Improvement: For any model, the system will offer an exploratory module that genuinely refits the model on training cohorts of increasing size (e.g., 3, 5, 7,... participants) while keeping the test participant untouched. It will report mean performance, participant-paired changes, and within-cohort variability.

Improvement: Implement a mandatory prediction-export system where every trial prediction is stored with a canonical trial ID, held-out participant, true label, predicted label, model, and replica ID. All metrics are recomputed from this ledger, and any discrepancy between reported and recomputed values triggers an error.

Improvement: Before running any classifier, the system will compute a participant-vs-label variance decomposition (e.g., sum-of-squares fractions) and a paired distance contrast (within-participant across-class vs. between-participant same-class). It will flag datasets where participant-associated variance dominates label-associated variance by more than an order of magnitude.

Improvement: The system will automatically rerun the evaluation summary after excluding participants with low trial counts (e.g., <40 or <50 retained trials) and report the change in mean performance with paired-bootstrap intervals.

Improvement: For any event-table-based dataset, the system will explicitly track the counting unit (rows vs. trials vs. epochs) through every transformation—pairing, condition selection, artifact rejection—and output a flow diagram with exact counts at each stage.

In summary: The improved AI system will be a self-auditing, leakage-proof, statistically rigorous EEG-decoding benchmark that produces trustworthy, reproducible, and interpretable results. It will not merely report a number—it will tell you why that number is or is not meaningful, and it will refuse to let you fool yourself or others.

Sources

Related papers