An Axiomatic Assessment of Entropy- and Variance-based Uncertainty Quantification in Regression

arXiv:2504.18433 · cs.LG, stat.ML · Submitted 2025-04-25 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "An Axiomatic Assessment of Entropy- and Variance-based Uncertainty Quantification in Regression".

Jane: The gist: This work provides a formal way of representing uncertainty in continuous space using a general parametric formulation and proposes axioms to rigorously assess total, aleatoric,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So, wrapping up this discussion on "An Axiomatic Assessment of Entropy- and Variance-based Uncertainty Quantification in Regression". The core contribution here is providing that formal uncertainty representation framework and the set of axioms to rigorously assess total, aleatoric, and epistemic uncertainty measures.

Jane: Basically, it gives us a structured way to look at how we quantify uncertainty when dealing with continuous regression problems by defining what those measures should actually do.

Lu: The authors showed that when you compare variance-based measures against the axioms, they fulfill most of them, but also that they have limitations regarding epistemic uncertainty.

Meng: And the entropy-based measures have their own set of trade-offs, showing where they can be negative or scale dependent in their ordering.

Lalam: The paper establishes a formal foundation for uncertainty representation and quantification in the supervised regression setting by introducing this parametric framework and those axioms to guide the design of new measures.

Tom: It’s a principled methodology that helps us evaluate existing measures, highlighting both their strengths and where they fall short based on these formal constraints.

Jane: It gives us a clear roadmap for designing better uncertainty quantification methods by showing what properties an uncertainty measure should actually possess in this specific mathematical context.

Conclusion: Tom: So we're wrapping up this look at "An Axiomatic Assessment of Entropy- and Variance-based Uncertainty Quantification in Regression." Basically, the authors gave us a formal way to check if these different ways of measuring uncertainty—entropy versus variance—actually work together in a regression setting.

Jane: Right. They set up this whole framework with six rules, or axioms, to make sure whatever measure we use is actually behaving correctly across different kinds of data and scenarios.

Lu: It's really interesting how they force the measures to obey these structural rules, like translation invariance and scale-order equivariance in a continuous space. They’re essentially trying to build a mathematical house for uncertainty that doesn't collapse when you move things around or change the size of your data.

Meng: From an engineering standpoint, it tells us which measure is more reliable when we’re actually building something. It points out that while variance-based measures hit most of these rules, they still have a specific gap regarding epistemic uncertainty—that part about what we *don't* know about the model itself.

Lalam: I see a lot of potential here for how AI systems can get more reliable when making predictions. If we can rigorously assess total versus aleatoric versus epistemic uncertainty, those models become much better at telling us when they’re guessing based on data versus when they’re genuinely unsure about the underlying rules.

Tom: Exactly. It moves us past just picking a measure that looks good on paper and gives us a way to check its fundamental structure against these formal constraints.

Jane: So, the authors showed that one approach has some clear strengths and some known weaknesses, which is super important for anyone trying to choose a method for their own work.

Lu: It's not about finding the perfect measure yet; it's about having a language to properly critique them and see what they’re actually measuring in the continuous regression world.

Meng: And this framework could guide us toward creating new uncertainty measures that satisfy all these rules, which is a big step for practical application.

Tom: It sets up a really solid foundation for how we think about the uncertainty in predictive models moving forward. We need to see how this plays out in real-world deep learning applications next.

LMU Munich · Munich Center for Machine Learning (MCML) · DLR-German Aerospace Center · University of Tromsø · German Research Center for Artificial Intelligence (DFKI, DSA)

cs.LG, stat.ML

Submitted: 2025-04-25

Updated: 2026-07-03

Journal ref: Proceedings of the 42nd Conference on Uncertainty in Artificial Intelligence (UAI 2026), PMLR 337:880-899, 2026

License: http://creativecommons.org/licenses/by-nc-sa/4.0/

Importance score: 66/100

The gist: The gist: This work provides a formal way of representing uncertainty in continuous space using a general parametric formulation and proposes axioms to rigorously assess total, aleatoric, and

Key concepts

Parametric Framework: This is a way to mathematically define the uncertainty in a regression model. Instead of just predicting a single value, it assumes the uncertainty comes from an unknown parameter vector $\theta$. The actual predictive distribution $P(x)$ is then defined as a mixture over all possible parameter settings $\theta$, linking the input data $x$ to the underlying model parameters.
Total Uncertainty (TU): Total uncertainty represents the overall randomness in a prediction. It is quantified using entropy-based measures, which measure how much information is needed to describe the predictive distribution. It combines both aleatoric and epistemic sources of uncertainty into one comprehensive measure.
Axioms: These are six formal rules proposed to guide the design and evaluation of uncertainty measures in continuous regression. They include properties like translation invariance and scale-order equivariance, which help determine if a specific measure is mathematically sound or well-behaved across different scenarios.

Terminology

Epistemic Uncertainty — 58× in this paper · explained in 9 episodes
Axioms — 48× in this paper · explained in 2 episodes
Uncertainty Quantification — 21× in this paper · explained in 8 episodes
Aleatoric Uncertainty — 17× in this paper · explained in 4 episodes
Interpretability — 5× in this paper · explained in 15 episodes
Additive Decomposition — 4× in this paper · explained in 1 episode
Generalization — 4× in this paper · explained in 13 episodes
Active Learning — 2× in this paper · explained in 6 episodes
Core — 2× in this paper · explained in 2 episodes
Diffusion Model — 2× in this paper · explained in 8 episodes
Diffusion Models — 2× in this paper · explained in 6 episodes
Latent Representation — 2× in this paper · explained in 2 episodes
Normalizing Flow — 2× in this paper · explained in 3 episodes
Robustness — 2× in this paper · explained in 5 episodes
Verification — 2× in this paper · explained in 2 episodes
Wasserstein Distance — 2× in this paper · explained in 4 episodes
Benchmark — 1× in this paper · explained in 2 episodes
Benchmarking — 1× in this paper · explained in 3 episodes
Calibration — 1× in this paper · explained in 7 episodes
Domain Shift — 1× in this paper · explained in 4 episodes
Downstream Tasks — 1× in this paper · explained in 2 episodes
Generative Modeling — 1× in this paper · explained in 3 episodes
Inverse Problem — 1× in this paper · explained in 2 episodes
Kullback-Leibler (KL) Divergence — 1× in this paper · explained in 3 episodes
Munich Center for Machine Learning — 1× in this paper
Quantifying Uncertainty — 1× in this paper · explained in 5 episodes
Reinforcement Learning — 1× in this paper · explained in 4 episodes
Trace — 1× in this paper · explained in 2 episodes
Uncertainty Decomposition — 1× in this paper · explained in 2 episodes

Summary

The gist: This work provides a formal way of representing uncertainty in continuous space using a general parametric formulation and proposes axioms to rigorously assess total, aleatoric, and epistemic uncertainty measures in regression settings

Uncertainty Representation in Regression

The paper introduces a framework for representing uncertainty by proposing a general parametric family of predictive distributions P(· θ) where the distributional family itself is fixed and all uncertainty resides in the parameter vector θ ∈ Θ The dependence on the covariates x is captured by a covariatedependent second-order distribution Q(x) ∈ P(Θ), representing the (epistemic) state of knowledge about the first-order parameters at a given instance x ∈ X Formally, this involves defining a measurable map φ: Θ → P(Y): φ(θ) = Pθ, which transforms the parameter vector θ into a (specific) probability measure The predictive (mixture) distribution is then defined as P(x) = ZΘ Pθ Q(x)(dθ), assuming all first-order measures are dominated by the Lebesgue measure µ

Quantification Measures

The paper recalls two predominant sets of measures used in the literature to quantify total (TU), aleatoric (AU), and epistemic uncertainty (EU): entropy-based and variance-based measures

Entropy-based measures define TU(Q) = H(P), AU(Q) = EQ[H(Y Pϑ)], and EU(Q) = EQ[DKL(Pϑ∥P)]
Variance-based measures decompose total uncertainty into its aleatoric and epistemic components via the law of total variance: V(Y)z = EQ[V(Y ϑ)]z + VQ(E[Y ϑ])z

Axiomatic Assessment

The core contribution is a set of six axioms designed to accommodate the continuous regression setting, which allow for the rigorous assessment of uncertainty measures These axioms include translation invariance (A4) and scale-order equivariance (A5), which exploit the metric and algebraic structure of R d

The proposed axioms are:

(A0)

TU, AU, and EU should be non-negative

(A1 & A2)

EU(Q) = 0 if and only if Q = δθ, and EU(δθ) ≤ EU(Ql) ≤ EU(Qu), ∀Qφl ≤2cx Qu

(A3)

AU(δθl) ≤ AU(δθu), ∀Pθl ≤cx Pθu

(A4)

TU, AU, and EU should be translation invariant

(A5)

TU, AU, and EU should be scale-order equivariant

Comparison of Measures

The paper analyzes deep ensembles and deep evidential regression to compare the two measure types across different representation methods

(Deep Ensembles)

For a Gaussian ensemble, variance-based measures yield AU(Q) = 1/M Σ σ 2m, EU(Q) = 1/M Σ µ 2m - (1/M Σ µm 2), and TU(Q) = V(Y) = 1/M Σ σ 2m + µ 2m - (1/M Σ µm!2

For entropy-based measures, AU(Q) = 1/2M Σ log(2πeσ 2m), EU(Q) = 1/M Σ Z ∞ −∞ p(t θm) log p(t θm) / M, and TU(Q) = H(P)

(Deep Evidential Regression)

For the evidential regression setting with a Normal-Inverse-Gamma prior, variance-based measures yield AU(Q) = βα - 1, EU(Q) = βυ(α - 1), and TU(Q) = V(Y) = βα - 1 (1 + υ-1

For entropy-based measures, AU(Q) = 1/2 (log(2πe) + log(β) − ψ(α)), EU(Q) = Ht 2αγ,β(1 + υ) a - AU(Q), and TU(Q) is given by a complex expression involving the Beta function

Assessment of Properties

The paper demonstrates how the proposed axioms are used to evaluate existing measures, revealing their strengths and weaknesses

(Variance-based Measures)

Proposition 5.1 proves that the variance-based measures violate A1 because they cannot distinguish between two second-order distributions that share the same marginal distribution over the mean

Proposition 5.2 proves that the variance-based measures fulfill A0, A2, A3, A4, and A5

(Entropy-based Measures)

Proposition 5.3 shows that the entropy-based measure violates A0 for AU and TU because they can be negative in certain settings

Proposition 5.4 proves that the entropy-based measure fulfills A2, A4, and A5

Conclusion

The work establishes a formal foundation for uncertainty representation, quantification, and evaluation in the supervised regression setting by introducing a parametric framework and a set of axioms to guide the design of new measures The analysis exposes fundamental shortcomings in both entropy- and variance-based measures, providing practitioners with a principled methodology for evaluating existing uncertainty measures While the variance-based approach satisfies most axioms, it lacks a unique minimum for epistemic uncertainty, while entropy-based measures can yield negative values and exhibit scale dependence in their ordering The framework provides a principled foundation for both evaluating existing uncertainty measures and guiding the design of new ones

Limitations and Future Work

The limitations noted include the generality of restricting uncertainty only to parameters, which might be too restrictive in settings like Gaussian processes or diffusion models Furthermore, satisfying or violating the axioms does not directly guarantee good empirical performance in any particular downstream task Future work suggests searching for novel uncertainty measures that satisfy the proposed axioms in their entirety or extending the set of axioms to accommodate other desirable properties like robustness The framework's generalization to a more general class of predictive distributions is also a direction worth pursuing

Acknowledgements

C. Bülte, Y. Sale, G. Kutyniok, and E. Hüllermeier acknowledge support by the DAAD programme Konrad Zuse Schools of Excellence in Artificial Intelligence, sponsored by the Federal Ministry of Research, Technology and Space C. Bülte and G. Kutyniok acknowledge support by the German Research Foundation under the grant DFG-SPP-2298 E. Hüllermeier acknowledges support by the German Research Foundation under the grant GRK 3081 (project number 534429653) G. Kutyniok also acknowledges support by the gAIn project, which is funded by the Bavarian Ministry of Science and the Arts (StMWK Bayern) and the Saxon Ministry for Science, Culture and Tourism (SMWK Sachsen) Furthermore, G. Kutyniok is supported by LMUexcellent, funded by the Federal Ministry of Education and Research (BMBF) and the Free State of Bavaria under the Excellence Strategy of the Federal Government and the Länder as well as by the Hightech Agenda Bavaria <ref:2504.

Improvements for AI systems

Identify novel uncertainty measures using Axioms A4 and A5 to design regression uncertainty quantification systems. This framework allows for a principled baseline of intuitive properties against which any candidate measure should be evaluated.
Develop a variance-based uncertainty system that adheres strictly to the proposed axioms, ensuring The variance-based measures fulfill the majority of the proposed axioms (Proposition 5.2), leading to more reliable assessment than existing methods that violate specific axioms in ways that directly translate to undesirable and potentially misleading behavior.
Implement an entropy-based uncertainty system with explicit checks for negative values, understanding that AU (and therefore TU) can be assigned negative values (Proposition 5.3), which informs practitioners about the limitations of this measure in practice.
Create a deep ensemble model utilizing the Gaussian predictive distribution framework to calculate uncertainty using variance-based measures, specifically noting that the variance-based measure for EU does not change across σ21, σ22 (Figure 5).
Design a deep evidential regression system that uses the Normal-Inverse-Gamma prior structure to quantify uncertainty, allowing the system to leverage the closed-form solutions for both variance and entropy measures derived in Section B.3.

Abstract

Uncertainty quantification is crucial in machine learning, yet most (axiomatic) studies of uncertainty measures focus on classification, leaving a gap in regression settings with limited formal justification and evaluations. In this work, we provide a formal way of representing uncertainty in continuous space, using a general parametric formulation, allowing for tractable analysis and evaluation of uncertainty measures. Within this framework, we propose a set of axioms that enable rigorous assessment of total, aleatoric, and epistemic uncertainty measures. Together, this allows for a theoretical examination of uncertainty measures and their corresponding properties. As a specific example, we compare the widely used entropy- and variance-based measures with respect to established predictive models and analyze their limitations and challenges in uncertainty quantification. Our work provides a principled way to understand and develop uncertainty measures in supervised regression, offering theoretical insights and practical guidelines for reliable uncertainty assessment.