A Three-Way Testing Framework for Quantifying Epistemic Calibration Uncertainty in SBI
stat.ME, stat.ML
Submitted: 2026-09-21
Updated: 2026-09-21
Code: https://github.com/Monoxido45/three_way_CP4SBI
License: http://creativecommons.org/licenses/by/4.0/
The gist: Current experimental scientists increasingly rely on simulation-based inference (SBI) to invert complex models with intractable likelihoods.
Terminology
Abstract
Current experimental scientists increasingly rely on simulation-based inference (SBI) to invert complex models with intractable likelihoods. A primary goal in these settings is to obtain credible regions with valid coverage. While recent model-agnostic conformal calibration methods have succeeded in constructing credible sets with prescribed local Bayesian coverage, their approximate nature introduces inherent epistemic uncertainty in the calibration process. In this work, we propose a novel tool for diagnosing calibration uncertainty. Our approach is based on a simple three-way hypothesis testing procedure. We demonstrate how this tool can be used to effectively assess necessary simulation budgets for calibration sets and analyze the epistemic uncertainty associated with cutoff estimation.
Sources
- Approximate Bayesian Computation with Deep Learning and Conformal prediction
- Conformal Calibration of Statistical Confidence Sets
- CP4SBI: Local Conformal Calibration of Credible Sets in Simulation-Based Inference
- Trustworthy scientific inference with generative models
- Approximating Likelihood Ratios with Calibrated Discriminative Classifiers
- Simulation-Based Inference: A Practical Guide
- Diffusion posterior sampling for simulation-based inference in tall data settings
- Variational Inference with Coverage Guarantees in Simulation-Based Inference
Related papers
- Doubly robust inference via calibration
- Bayesian Empirical Bayes: Simultaneous Inference from Probabilistic Symmetries
- Flexible Nonparametric Inference for Causal Effects under the Front-Door Model
- Deployment of AI-Assisted Interventions: Capacity Constraints and Noisy Compliance
- A Survey on Archetypal Analysis
- Dynamic Spatial Bayesian Machine Learning Model: Applications to Intergenerational Economic Mobility and Geographic Income Inequality in the United States