Conformal Prediction via Transported Beta Laws

This paper introduces a novel framework for analyzing split conformal prediction by focusing on the law of realized calibration-conditional coverage rather than just its marginal guarantee. It establishes that while standard conformal prediction provides a marginal coverage bound, the actual distribution of this coverage under non-i.i.d. settings can be characterized using optimal transport to compare it against an i.i.d. Beta reference law, thereby quantifying how different sources of non-i.i.d behavior deform this reference and translate into coverage degradation or bad-calibration probabilities across various settings like scale shifts, clustering, and mixing processes.

The Core Reference Object: The Beta Law

The paper establishes the continuous i.i.d. case as the universal benchmark for realized coverage under split conformal prediction (CP). For a fixed order statistic index k, the calibration-conditional coverage variable is defined as:

1. For 1 ≤ k ≤ n, define the conditional probability:

Cn,k:= P(S(k) ≤ S1,..., Sn).

2. In the continuous i.i.d. setting, this distribution has an exact form: Cn,k ∼ Beta(k, n + 1 − k).

3. The paper defines the i.i.d. reference law as βn,k:= Beta(k, n + 1 − k).

4. This beta law serves as the finite-sample reference object against which all non-i.i.d. deformations are measured using Wasserstein distances on [0, 1].

Quantifying Deviations via Optimal Transport

The framework utilizes optimal transport to compare the law of the realized coverage variable, νn,k, with the i.i.d. benchmark βn,k:

1. The central question is how much the law of the realized coverage departs from this beta reference and how that departure translates into coverage degradation.

2. This comparison is performed on [0, 1] using Wasserstein distances, specifically W1(νn,k, βn,k).

3. The identity map on [0, 1] being 1-Lipschitz allows the W1 comparison to immediately control the marginal coverage gap: Cov(k) − k/(n + 1) ≤ W1(νn,k, βn,k).

Characterizing Non-i.i.d. Mechanisms

The framework separates different sources of non-i.i.d. behavior based on how they deform the beta reference:

1. Test-side shift acts through a transport map on the coverage scale; this is explicitly shown in examples like the half-normal scale shift, where the realized coverage is transported via hr(u).

2. Calibration dependence changes the order-statistic law itself; this is characterized by analyzing how it deforms the underlying score distribution, often quantified using counting processes or Berry–Esseen approximations.

Coverage Guarantees and Bad Calibration Probabilities

The Wasserstein comparison directly yields bounds on coverage gaps and bad-calibration probabilities:

1. Theorem 11 establishes the coverage gap bound: Cov(k) − k/(n + 1) ≤ W1(νn,k, βn,k). If νn,k is in a p-Wasserstein beta neighborhood of radius ρ, then Cov(kγ) − γ ≤ ρ + 1/(n + 1).

2. Theorem 12 provides bounds on bad calibration events: P(Dn,k ≤ t) ≤ P(Bn,k ≤ t + ε) + W1/ε p. If a uniform transport bound is available, the comparison simplifies to P(Dn,k ≤ t) ≤ P(Bn,k ≤ t + ρ).

Application in Dependent Settings

The framework is applied across various dependent process settings:

1. In decoupled test scores (independent of calibration), the realized coverage becomes a transformation of the i.i.d. order statistic: Dn,k = h(U(k)), where h is the map relating calibration and test score distributions.

2. For stationary mixing processes, bounds are derived using a combination of direct decoupling arguments (for future test scores) and Berry–Esseen approximations for sample quantiles of strongly mixing sequences to control the deviation of the calibration order statistic from its i.i.d. beta reference law, leading to a final bound on the coverage gap: P(Tn+l ≤ T(kγ)) − γ ≤ ∆l + r 2πn/τγ - pγ(1 − γ) + O(n−1/2).

Conclusion

The study demonstrates that the beta law remains a useful finite-sample reference beyond the i.i.d. setting, allowing researchers to diagnose and compare different non-i.i.d.

Here are specific improvements for AI systems derived from the principles outlined in this paper, categorized by application area:

)1. Robustness and Uncertainty Quantification in Conformal Prediction (CP):

The core improvement is moving beyond marginal coverage guarantees to characterizing the distribution of realized coverage under non-i.i.d. data structures.

• AI Systems can be equipped with a Transported Beta Law Analyzer module that estimates the Wasserstein distance, specifically the W1 distance, between the empirically observed calibration-conditional coverage law and its i.i.d. Beta reference law, using optimal transport techniques (Proposition 7).

• This allows for precise quantification of how much a specific data shift (e.g., distribution shift in half-normal scale) or dependence structure (AR(1) process) deforms the conformal prediction guarantee on the coverage scale.

• AI systems can then dynamically adjust their prediction set construction parameters (like the threshold choice or the nominal level) based on this calculated Wasserstein radius, ensuring that the resulting prediction sets maintain a desired level of marginal coverage even when data violates i.i.d. assumptions.

)2. Adaptive Thresholding and Dynamic Calibration (Handling Distribution Shift):

The framework provides a mechanism to explicitly model and compensate for distribution shift in score distributions without relying solely on score-space distance metrics (as in previous work).

• For systems operating under known distribution shifts (e.g., when the test data generating process is known to be Gaussian with a different variance), the system can utilize the explicit transport map derived from Proposition 4 and Example 4.1.1 (half-normal scale shift).

• The AI can calculate a deterministic, non-linear transformation of the standard Beta order statistic based on the known scale ratio between calibration and test scores, effectively transporting the i.i.d. benchmark to match the shifted reality before calculating coverage statistics.

• This enables more accurate prediction set construction in regression tasks where error distributions change across different domains (e.g., medical imaging vs. standard datasets).

)3. Effective Sample Size and Dependence Modeling (Clustered and Temporal Data):

The paper provides concrete methods to account for non-i.i.d. structure through effective sample size concepts derived from clustering or mixing coefficients, rather than just treating dependence as an error term in a general way.

• For clustered data (Proposition 14), the AI can estimate the effective number of independent units by calculating the Wasserstein distance between the actual coverage law and a reference Beta law parameterized by this effective sample size, rather than using the total sample size.

• For time-series or sequential data (AR(1) example, Section 4.3), the system can explicitly model test-calibration dependence using Markov properties to quantify how much the future test score deviates from independence, and integrate this deviation into its uncertainty estimates (the term involving horizon length 'l' in Equation 1).

)4. Tail Risk Management for Calibration Events (Bad Calibration Probabilities):

The framework provides a sharp, transport-based bound for the probability of bad calibration events (where the realized threshold yields significantly low coverage).

• AI systems can utilize Theorem 12 to calculate tighter bounds on the probability of catastrophic calibration failures. Instead of relying on conservative Markov penalties (which can be loose), they can use the explicit monotone transport map to establish a sharper bound: if a uniform transport bound exists, the bad-calibration probability is directly controlled by shifting the i.i.d. reference law by that same distance, leading to much tighter risk assessment for deployment decisions.

)5. Generalization via Berry–Esseen Approximations (Asymptotic Control):

For large datasets where exact computation of the realized law is intractable, the framework provides a rigorous asymptotic tool based on the Berry–Esseen theorem.

• The AI can use Proposition 15 to derive an explicit, analytically derived upper bound on the Wasserstein distance between its empirical coverage and the i.i.d. reference as a function of sample size and known process mixing rates (e.g., for strongly mixing sequences).

• This allows for real-time monitoring of model performance under non-stationary conditions by tracking the decay rate of this approximation, providing a quantifiable measure of how far the current calibration regime is from the idealized i.i.d. benchmark.