Every pooling rule has its world: matching probability combination rules to situations and stakes

arXiv:2608.11275v1 [stat.ME]

---

Systems often need to combine two numerical assessments of the same yes/no question. The paper addresses a fundamental question: what formula is correct for combining probability or confidence reports, given what the numbers represent and how the sources are related?

The authors note that the correct formula for combining them depends on what the numbers represent and on how the sources are related. For reports 0.3 and 0.8, the formulas considered return values from 0.55 to 0.86, and four of them produce the same decision at threshold 1/2. Binary accuracy at that threshold cannot distinguish them.

The paper makes four main contributions:

1. For common probability and confidence-combination rules, it states what their inputs represent and derives generating conditions under which their output is the correct conditional probability.

2. It separates two experimental questions: controlled Monte Carlo worlds test whether each rule matches the probability implied by its stated assumptions; the same worlds measure the cost of a mismatch with logarithmic score and threshold decisions under different costs.

3. It shows that numerically different pooling rules can induce the same binary decision at threshold 1/2, so binary accuracy at one threshold can fail to distinguish materially different probability estimates.

4. For derivations that share uncertain premises, retaining the identities of those premises permits direct calculation of the proof-union probability, while reducing proofs to pairwise scalar probabilities loses information for three or more derivations.

The paper defines five input types:

• pi = P(Y = 1 Ei): the probability of the outcome after observing evidence Ei

• ri = P(Ai): the probability of one event that can make the outcome occur

• si ∈ [−1, 1]: a signed strength of supporting or opposing evidence

• mi ∈ [0, 1]: mass assigned to support or opposition, the rest to ignorance

• Si: the set of uncertain premises used by proof i

The rules considered include:

Average and weighted average (0.55 for inputs 0.3, 0.8): Input: probabilities under alternative cases. Target: the overall probability after averaging over the cases. Averaging is correct when the uncertainty is about which reading, passage, expert, or model applies.

Maximum (0.8), probabilistic sum (0.86), and bounded sum (1.0): Input: probabilities of two component events. Target: the probability that at least one occurs. "The dependence relation determines the union probability: the maximum applies when the events are maximally positively dependent (one contains the other), the probabilistic sum (called noisy-or in the Bayesian-network literature) when they are independent, and the bounded sum when the union has the largest probability compatible with the two reports (the upper Fréchet bound)."

Multiplicative pooling, upco, product of experts (0.63): Input: two calibrated posterior reports about one claim, computed from a common prior. Target: the posterior probability given both observations. The formula is correct under three assumptions: both reports are calibrated posterior probabilities, both use the same prior, and their observations are conditionally independent given either outcome. With prior 1/2, the posterior given both observations is upco(p1, p2). For a prior π different from 1/2, the correct combination divides the doubled prior odds back out once.

Geometric pooling and the weighted log-odds family (0.57 for equal weights): Input: two posterior reports about one claim. The rule is exact when both reported log-odds have the same multiplicative distortion, when both reports repeat the same evidence, or when two equal-sized samples are averaged.

The MYCIN certainty-factor rule (0.71 on the confidence scale): Input: signed evidence strengths for and against a claim. Target: the remaining support after the opposing evidence is taken into account. The revised EMYCIN rule uses: same sign positive: x + y − xy; same sign negative: x + y + xy; opposite signs: (x + y)/(1 − min(x, y)).

The ProbLog conflict rule (0.56) and the naive difference (0.50): Input: signed evidence strengths for and against a claim. Target: the probability that the supporting derivation holds and the opposing one does not. The ProbLog value is a(1 − b) under independence.

Cumulation: Input: two derivation probabilities and an estimated degree of independence. Target: the probability that at least one derivation holds. The formula is cumul(c1, c2; a) = max(c1, c2) + a[ps(c1, c2) − max(c1, c2)], where a ∈ [0, 1] estimates independence.

GK calculation: Input: the uncertain ground premises used by each retained proof, together with their probabilities. Target: the probability that at least one proof is available. For two retained proofs with shared event set S, the formula is gk(c1, c2; s) = c1 + c2 − c1c2/s, where s is the product over the shared events.

Dempster–Shafer combination (bel 0.74, BetP 0.83): Input: one source assigns mass to support and the rest to ignorance; the other assigns mass to opposition and the rest to ignorance. Target: lower and upper probabilities, and a decision probability (BetP, the pignistic probability).

Proposition 1: For all p1, p2 ∈ (0, 1): avg(p1, p2) > 1/2, geo(p1, p2) > 1/2, upco(p1, p2) > 1/2, and the signed strength output of the MYCIN combination is positive, each exactly when p1 + p2 > 1. All four formulas yield the same binary decisions at threshold 1/2 on the same numerical inputs. Their numerical outputs differ and can be distinguished by probability-sensitive scores or other thresholds.

The paper defines a world as a complete probability model for the two reports and the outcome. A rule is exact in a world when it returns the conditional probability of the outcome from the information supplied to the rule.

World 1 (Average): Two possible readings of an ambiguous query. With Z ∈ 1, 2 the intended reading, each report is calibrated conditional on its reading. The probability that the claim is true is the weighted average. The situation generalizes: when the two reports describe two mutually exclusive circumstances and circumstance i holds with probability wi, the correct combination is the weighted average.

World 2 (Multiplicative/upco): Two reports based on independent evidence. "Each round has P(Y = 1) = 1/2. Model 1 observes the image attached to the case and model 2 the accompanying text; the two observations are conditionally independent given the truth value... Under these conditions the posterior probability given both observations is exactly upco(p1, p2)." With prior 0.3, upco is systematically too high, but the odds pool with prior correction restores the exact posterior.

World 3 (Geometric): Doubled log-odds and averaged samples. Geometric pooling is therefore the exact rule for sources that double their log-odds; averaging member networks' logits, a standard ensemble variant, is equivalent to geometric pooling for binary predictions. Also, the geometric pool of the two distributions equals the posterior distribution of the averaged sample.

World 4 (Probabilistic sum): Two independent attempts at the same task. If the attempts come from two unrelated models whose failures are independent, the event fails only if both attempts fail: 1 − 0.70 · 0.20 = 0.86, the probabilistic sum.

World 5 (Maximum): Difficulty-nested attempts. Every task solved by the weaker attempt is also solved by the stronger one, so 'at least one passes' is the same event as 'the stronger attempt passes', with probability max(0.30, 0.80) = 0.80.

World 6 (MYCIN): Poisson evidence and item-level refutation. Proposition 2 shows that with supporting items arriving as a Poisson process of rate λa, each refuted with probability λb/λa, the probability that at least one supporting item survives unrefuted is 1 − e−(λa−λb) = (a − b)/(1 − b), the MYCIN conflict value.

World 7 (ProbLog): Independent supporting and opposing searches. The conflict case counts the rounds where the supporting search returns a passage and the contradicting search does not: a(1 − b), which is the ProbLog value.

World 8 (Cumulation): Two reasoning chains with unobserved overlap. "In the first [model], sharing is all or nothing: with probability 1 − a both chains use the same uniform variable, and with probability a they are independent... Here the probability that at least one chain holds is exactly cumul(c1, c2, a)." For item-level sharing, cumulation overestimates by up to 0.02.

World 10 (GK calculation): Two chains with identified shared premises. The GK value matches the simulated frequency for every k (largest absolute deviation 0.0007, within the Monte Carlo sampling error), where cumulation with the count-based parameter has an error of up to 0.021. For three proofs in a chain pattern, Folding the two-argument formula through the pairwise overlaps gives 0.8318, and ignoring the second overlap gives 0.9169, while the exact union probability is 0.8984.

World 9 (Dempster–Shafer): Opposing sources conditioned on non-conflict. Conditioning on non-conflict leaves three cases: only model 1 decisive (the claim is true), only model 2 decisive (the claim is false), and neither decisive. The construction also illustrates Zadeh's (1984) criticism: when m1 and m2 are both large, little mass remains after conditioning.

The paper uses two evaluation criteria:

Logarithmic score with Kelly betting: "Each round, before the outcome is known, two sources provide their reports and a pooling rule combines them into q. At even odds, the current wealth is split between the two outcomes: fraction q is allocated to the event and fraction 1 − q to its complement." The expected score is maximized exactly at q = p, and the shortfall is the binary Kullback–Leibler divergence in bits.

Threshold decisions and cost-loss model: "Review costs C; releasing a faulty answer costs L > C. We set L = 1 and write c = C/L... If the pooled probability of a fault is q, the expected cost of review is c and the expected cost of releasing is q, so the answer is sent for review exactly when q > c."

Key results from the betting worlds (oracle gaps in bits per round):

• World 1 (average exact): average gap 0, geometric 0.002, upco 0.085, MYCIN 0.092, noisy-or 0.310

• World 2 (upco exact): upco gap 0, MYCIN 0.003, geometric 0.070, average 0.089, noisy-or 0.276

• World 3 (geometric exact): geometric gap 0, average 0.015, MYCIN 0.111, upco 0.118, noisy-or 0.381

• World 4 (noisy-or exact): noisy-or gap 0, average 0.214, geometric 0.214, MYCIN 0.262, upco 0.269

• World 5 (maximum exact): max gap 0, average 0.063, geometric 0.063, noisy-or 0.087, MYCIN 0.111, upco 0.118

The paper notes: A gap of d bits per round compounds to a relative wealth factor of 2−dT over T rounds: for the world-2 average, at d = 0.089, the wealth ratio to the oracle is 2−0.89 ≈ 0.54 after ten rounds and about 1/478 after 100.

The paper establishes: The betting oracle gap is a weighted integral of the excess costs over all decision thresholds... betting oracle gap = ∫ extra cost-loss expense at threshold c · dc/(c(1−c) ln 2). The logarithmic weight 1/(c(1−c)) grows without bound near 0 and 1, assigning greater weight to decision errors at extreme thresholds. The Brier score assigns constant weight over thresholds, so its rule rankings can differ.

Varying the symmetric coefficient α from 0.5 to 2.5 in four settings:

1. Calibrated reports based on conditionally independent evidence: zero gap at α = 1 (upco)

2. Both reports based on the same observation: zero gap at α = 1/2 (geometric)

3. Underconfident reports stating half their justified log-odds: zero gap at α = 2

4. Half-shared evidence: best symmetric α is 0.6, with a gap of 0.044 bits per round remaining at every α

One common log-odds coefficient exactly corrects uniform distortion and complete duplication, but generally not partial sharing.

1. The appropriate combination rule depends on both what the input numbers represent and how their sources are related.

2. Using a rule outside the conditions for which it is correct can substantially increase both probability-sensitive loss and decision cost.

3. Different ways in which supporting and opposing evidence interact also lead to different combination rules.

4. When several proofs share uncertain premises, retaining the identities of those premises makes it possible to calculate directly the probability that at least one proof is available.

5. Weighted log-odds pooling can exactly correct some simple forms of systematic dependence or miscalibration.

To choose a combination rule, three questions should be answered: (1) What do the inputs represent? (2) How are the sources or events related? (3) If evidence supports opposite sides of a claim, how does that opposition operate?

The paper notes limitations: "Most of the analysis concerns two reports and binary questions; extending the results to larger collections of reports requires assumptions about their joint dependence. The experiments assume that the applicable generating mechanism and its parameters are known; the paper does not study how to infer them from data."

Based on the paper, here are the specific improvements I can make to AI systems:

Improvement: Instead of using a fixed aggregation method (e.g., always averaging or always using product-of-experts), I can dynamically select the appropriate combination rule based on:

• What the inputs represent (conditional probabilities, event probabilities, evidence strengths, or masses)

• How sources are related (independent, shared evidence, nested, or opposing)

• The stakes involved (cost of false positives vs. false negatives)

Capability: An AI system that combines multiple model predictions, expert judgments, or sensor readings can automatically choose between averaging, geometric pooling, upco, probabilistic sum, or maximum based on detected source relationships, rather than defaulting to one rule.

Improvement: When combining posterior probabilities from multiple sources, I can correct for non-uniform priors. The paper shows that upco is only exact with prior 1/2; for other priors, I must divide out the doubled prior odds.

Improvement: I can report not just the pooled probability but also the range of plausible values across different pooling rules, and identify when binary decisions at threshold 1/2 are robust vs. fragile.

Improvement: I can detect when multiple reports are based on overlapping evidence and apply the appropriate correction. The paper shows that geometric pooling (α=1/2) corrects complete duplication, while partial sharing requires intermediate coefficients.

Improvement: Instead of always using threshold 1/2, I can optimize decision thresholds based on the cost-loss ratio (c = C/L) and use the appropriate pooling rule for that threshold.

Improvement: When combining multiple logical derivations or reasoning chains, I can retain the identities of shared uncertain premises and compute the exact union probability, rather than approximating through pairwise scalar combinations.

Improvement: I can estimate the distortion factor in reported log-odds and apply the inverse coefficient to correct systematic miscalibration before pooling.

Improvement: I can evaluate pooling rules using logarithmic scoring (Kelly betting) rather than binary accuracy, which distinguishes rules that produce identical threshold decisions but different probability estimates.

Improvement: I can compute the integral of excess decision costs across all thresholds, weighted by 1/(c(1-c)), to identify where a pooling rule is most vulnerable.

Improvement: I can implement a meta-pooling system that classifies the evidence interaction type (independent, nested, opposing, shared-premise) and switches between rules accordingly.