Do Deep Ensembles Actually Capture Uncertainty in Graph Neural Networks?

summary

Video file (mp4)

The gist

The gist The results demonstrate that deep ensembles largely do not capture uncertainty in graph neural networks, as they suffer from epistemic collapse where independently trained models converge to

In short

Deep ensembles, a method for quantifying uncertainty by averaging models trained with different initializations, largely fail to capture meaningful uncertainty in Graph Neural Networks (GNNs). The study found that these ensembles suffer from 'epistemic collapse,' where independently trained models become too similar. This means the ensemble gains are mostly just stabilizing optimization noise, not providing better uncertainty estimates.

Key concepts

Deep Ensembles (DEs)
DEs involve training multiple GNN models from different random starting points and then averaging their predictions. The idea is that disagreement between these models reveals uncertainty. However, the paper shows this mechanism breaks down in GNNs because the models converge too closely together.
Epistemic Collapse
This occurs when independently trained GNNs consistently produce nearly identical predictions. Because deep ensembles rely on different models disagreeing to capture model uncertainty, this lack of diversity means the ensemble loses its primary advantage in estimating knowledge-based uncertainty.
Aleatoric vs. Epistemic Uncertainty
Aleatoric uncertainty reflects inherent noise in the data itself, which is irreducible. Epistemic uncertainty reflects a model's lack of knowledge about the data due to insufficient training or diversity. The study found that GNN ensembles are overwhelmingly dominated by aleatoric uncertainty, with very little useful epistemic component.

Terminology used across episodes

This episode discusses

The paper

Do Deep Ensembles Actually Capture Uncertainty in Graph Neural Networks? · Read on arXiv

Pedro C. Vieira, Pedro Ribeiro, Viacheslav Borovitskiy

University of Edinburgh · DCC/FCUP, University of Porto

While deep ensembles are widely considered to be the default method for uncertainty quantification in deep learning, their effectiveness for graph-structured data is often simply assumed based on successes in domains like computer vision. We investigate standard deep ensembles specifically for message-passing graph neural networks. Benchmarking across eight datasets representing varied tasks and complexities, we reveal that ensembles provide only marginal improvement over a single model. These marginal gains stem primarily from averaging out optimization noise in point predictions rather than from meaningfully better uncertainty estimates. Through an aleatoric-epistemic decomposition, we identify epistemic collapse: independently trained networks consistently converge to overly similar predictions. Because disagreement is the fundamental mechanism through which ensembles capture epistemic uncertainty, this lack of diversity neutralizes their key advantage. Towards explaining this phenomenon, we derive a bound on epistemic variance in terms of the graph topology and Dirichlet energy budget, linking epistemic collapse to the smoothing bias of message passing. Our results suggest that deep ensemble success does not seamlessly transfer to graph machine learning.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Do Deep Ensembles Actually Capture Uncertainty in Graph Neural Networks?".

Jane: The gist The results demonstrate that deep ensembles largely do not capture uncertainty in graph neural networks,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So, this paper, "Do Deep Ensembles Actually Capture Uncertainty in Graph Neural Networks?", it looks at deep ensembles—which are usually the go-to for uncertainty quantification in deep learning—and checks if they work well when you apply them to graph neural networks. Jane Basically, they assume that because they worked so well for things like computer vision, they should work fine on graphs too. Tom But this study looks at standard deep ensembles specifically on message-passing graph neural networks and finds that the improvements are actually pretty small compared to using just a single model. Jane They claim the marginal gains they see usually come from just stabilizing the noise in point predictions rather than getting better uncertainty estimates for what's going on in the data. Tom The big finding here is something called epistemic collapse, which means independently trained networks end up predicting nearly identical things to each other. Jane And since disagreement between models is how ensembles usually capture that model uncertainty, this lack of diversity basically cancels out their main advantage. Tom They even show that this collapse is so bad in classification tasks that the epistemic uncertainty becomes almost zero. Jane This suggests that what you get from random initializations isn't actually giving you different models, just slightly different versions of the same thing. Tom So, it’s a sober look at whether deep ensembles transfer their success seamlessly from classic deep learning settings to graphs.

Lu: From my side, I think this is interesting because it points toward how message passing might induce some form of convexity in the functional space, which restricts that functional diversity even in shallow networks nineteen.

Meng: From an engineering standpoint, if the disagreement is collapsing so fast, it means we can't rely on running many random initializations to get a diverse set of models. We need a different way to inject meaningful variation into the training process if we want these ensembles to actually do their job sixteen.

Lalam: If I could summarize this for you, it’s like having twenty identical robots trying different ways to build a bridge, but they all end up building the exact same bridge because the instructions are too restrictive <ref:2605.22593#pg5>.

Conclusion: Tom: So wrapping up this paper, "Do Deep Ensembles Actually Capture Uncertainty in Graph Neural Networks?", the authors Pedro C. Vieira, Pedro Ribeiro, and Viacheslav Borovitskiy found that deep ensembles mostly don't improve uncertainty quantification for graph neural networks. Jane They argue that the marginal gains they see are really just about reducing optimization noise in point predictions instead of getting better estimates for uncertainty itself. Tom The core issue they identify is epistemic collapse, where the independently trained models converge to overly similar predictions because there's no real disagreement left to capture model uncertainty. Jane This means that the mechanism deep ensembles use—having different models disagree—is neutralized because the individual graph neural networks don't end up being very different in their outputs. Tom It really suggests that for high-quality uncertainty quantification in graph learning, we need to look beyond just ensembling message-passing architectures. Jane A single "lucky" model can often do just as well as the whole ensemble, which is a big thing for practical application.

Lu: I see this as a strong push toward developing methods that deliberately encourage functional diversity in the GNNs themselves, rather than just relying on random initialization noise nineteen.

Meng: For me, it means our focus should shift from simple model aggregation to finding ways to make the underlying graph learning process itself more varied or robust against local optimization traps sixteen.

Lalam: If I were to try and improve culture based on this, I would say that we need more creative ways for the AI to explore its solution space, not just training it many times in the same way <ref:2605.22593#pg5>.

More episodes

← Home