Evaluating utility of clinical prediction models: A comparison of decision curve analysis and cost curves

summary

Video file (mp4)

The gist

The gist The comparison between decision curve analysis and cost curves reveals that while both approaches assess model utility across different thresholds, they are closely related, with decision

In short

The paper compares Decision Curve Analysis (DCA) and cost curves to evaluate clinical prediction models across different decision thresholds. It shows DCA estimates expected utility using 'net benefit,' while cost curves, particularly Brier curves, measure expected loss. While both assess model utility, Brier curves are more generally applicable across various thresholds.

Key concepts

Decision Curve Analysis (DCA)
A visual approach used to assess a clinical prediction model's utility by considering the relative value of correct positive versus negative classifications. It estimates 'net benefit,' which reflects the expected utility, helping determine if a model is clinically useful at different thresholds.
Cost Curves
A general approach to assessing classifier performance across various operating conditions, including different class distributions and misclassification costs. The paper focuses on a specific type called the Brier curve, which measures loss based on probabilistic threshold choices.
Brier Curve
A specific cost curve where the loss is defined using a probabilistic threshold choice. The area under this curve, known as the Brier score, relates directly to clinical utility and combines refinement and calibration loss.
Net Benefit (NB)
A measure used in DCA that quantifies the expected utility of a prediction model. It captures the difference between the gain from correctly predicting positive cases and the loss from incorrectly predicting negative cases, helping practitioners choose optimal thresholds.

Terminology used across episodes

This episode discusses

The paper

Evaluating utility of clinical prediction models: A comparison of decision curve analysis and cost curves · Read on arXiv

Louise AC Millard, Peter A Flach

MRC Integrative Epidemiology Unit, University of Bristol · Population Health Sciences, University of Bristol · Intelligent Systems Laboratory, University of Bristol

Classification models typically predict a binary outcome and use one or more decision thresholds to produce a classification or inform a clinical decision. Decision curves and cost curves are model evaluation approaches that assess net benefit and loss across decision thresholds. In this paper we compare decision curves and cost curves, demonstrating that decision curves are closely related to a specific type of cost curve called a Brier curve. Users of both approaches set the classification threshold assuming calibrated model scores, using the relative clinical importance of correct positive and negative classifications. A given decision curve has a corresponding Brier curve such that reference lines common in each space can be included in either (e.g. the 'treat none' decision curve can be translated into an equivalent 'treat none' Brier curve). We suggest the upper envelope decision curve as a useful comparison to a model decision curve showing the maximum achievable net benefit through recalibration. We show that net benefit (used for decision curves) and Brier loss (used for Brier curves) will always choose the same model as optimal at any given threshold, but cannot be compared or aggregated across thresholds except under specific assumptions; doing so otherwise requires custom weightings.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Evaluating utility of clinical prediction models".

Jane: Here's the gist of the paper. The gist The comparison between decision curve analysis and cost curves reveals that while both approaches assess model utility across different thresholds, they are closely related, with decision curves being closely related to a specific type of cost curve called a Brier curve <ref:2509.24608#pg20>.

Tom: Model Evaluation Approaches Classification models typically predict a score and use a decision threshold to produce a classification <ref:2509.24608#pg2>.

Jane: Appropriate model evaluation should carefully consider the context in which a model will be used, including the relative value of correct classifications of positive versus negative examples, which affects the threshold that should be used <ref:2509.24608#pg2>.

Tom: Decision curve analysis (DCA) and cost curves are model evaluation approaches that assess the expected utility and expected loss of prediction models, respectively, across decision thresholds <ref:2509.24608#pg2>.

Title and authors: Tom: We just talked about what the paper is trying to do generally, focusing on comparing decision curves and cost curves for evaluating clinical prediction models. Now, let's look at who wrote this stuff and why it matters. The authors are Louise AC Millard and Peter A Flach from the University of Bristol.

Jane: They are academics in population health sciences, so you know they're coming at this from a clinical and epidemiological angle, which is important because they’re focused on how these models fit into real health contexts.

Lu: What I find significant is that their work directly addresses the question of which evaluation approach is superior when you're dealing with different operating conditions for a model.

Meng: For us in production, it means if we choose the wrong way to evaluate, we might deploy a model that looks good on paper but doesn't actually serve its intended clinical purpose under real-world constraints.

Lalam: The paper sets up this comparison because prediction tasks can be classification, ranking, or probability estimation one <ref:2509.24608#pg1>. When you use a threshold to turn that score into a classification, the choice of that threshold really depends on what you value—for example, how important it is to correctly identify a positive case versus avoiding false alarms for negative cases.

Tom: Exactly. The paper explains that appropriate model evaluation has to consider the context and the relative value of correct classifications of positive versus negative examples because that directly affects which threshold you should use one <ref:2509.24608#pg1,the relative value of correct classifications of positive versus negative examples>.

The paper's summary: Jane: So, what’s the main takeaway from this paper? Essentially, they show that decision curves and cost curves are closely related. They aren't completely different ideas; they point to the same underlying mechanism for assessing model utility across those thresholds.

Tom: That's a key finding. They demonstrate that decision curves are closely related to a specific type of cost curve called a Brier curve <ref:2509.24608#pg20>. This connection is what they spend the paper showing—it links the two approaches together.

Lu: The core mechanism they highlight is that both curves are derived assuming the model scores are calibrated and that you set your threshold based on those relative classification values <ref:2509.24608#pg2>. Also, they show that net benefit used for decision curve analysis and Brier loss used for Brier curves will always choose the same model as optimal at any given threshold <ref:2509.24608#pg2>.

Meng: That's a strong statement. It means if you use the net benefit measure from DCA and the Brier loss from a cost curve, they will point to the exact same best model for a specific operating condition <ref:2509.24608#pg2>.

Lalam: The paper goes on to say that across different thresholds, differences in Brier loss are comparable, whereas differences in net benefit just can't be compared directly because they represent different things—differences in expected utility <ref:2509.24608#pg2>.

The paper's improvements: Tom: The paper suggests a few ways we can use this comparison to improve our evaluation process. First, you can use decision curves to optimize model selection across multiple contexts because you can see how models perform relative to clinical utility when the value of correct positives versus negatives changes <ref:2509.24608#pg3>.

Jane: That’s really useful for us because it helps us choose a model that maximizes expected utility based on the specific priorities of the clinical scenario you're working in right now. It moves beyond just looking at one fixed threshold.

Lu: Then they suggest calculating an upper envelope decision curve. This is interesting because it shows, for a given model, how much clinical utility you could potentially improve just by recalibrating those scores <ref:2509.24608#pg2>.

Meng: For deployment, that’s helpful because it tells us if spending time and resources on score calibration actually yields a tangible gain in the expected benefit we care about, rather than just tweaking the model structure itself.

Lalam: And they also highlight how Brier curves are more generally applicable across a wider range of decision thresholds because that's when a lot of real-world scenarios don't stick to one specific threshold <ref:2509.24608#pg2>.

Conclusion: Tom: So, wrapping up the discussion on this paper, the main message is that Brier curves are a generally applicable alternative to decision curves because a unit difference in Brier loss corresponds to the same difference in model value across thresholds <ref:2509.24608#pg2>.

Jane: That's right. They suggest Brier curves might be easier to interpret across different thresholds because that unit of loss is consistent, which is a nice practical thing for us when we’re testing various operating conditions.

Lu: And they pointed out that the area under the Brier curve is called the Brier score, and it makes clear this relationship between that score and clinical utility <ref:2509.24608#pg2>.

Meng: For deployment, this means if you want to compare different models or different calibration strategies consistently across a range of potential thresholds, using the Brier curve might give you a more stable comparison metric.

Lalam: I just want to mention that one limitation they flag is that differences in net benefit across thresholds aren't directly comparable because they correspond to different differences in expected utility <ref:2509.24608#pg2>.

Tom: So, the paper "Evaluating utility of clinical prediction models: A comparison of decision curve analysis and cost curves" gives us tools to decide whether we should prioritize the detailed interpretation offered by decision curves or the broader applicability of Brier curves depending on our specific needs.

More episodes

← Home