When Offline Evaluation Misleads: A Diagnostic Protocol for Reward and Policy Selection in Delayed-Feedback Contextual Bandits

This paper presents a diagnostic protocol for selecting proxy rewards and contextual bandit policies in delayed-feedback settings, where the business north-star (e.g., a conversion) is observed only after a long delay and cannot train the bandit online. The authors train on a fast proxy reward and must decide both which proxy to use and whether a contextual bandit is worth its complexity over a fixed best message. They show that standard offline checks—a batch off-policy estimate, a marginal arm-discrimination test, and a confidence interval—can mislead systematically under delayed feedback.

The protocol screens a reward-and-policy candidate on two axes before trusting any reported lift: alignment (does optimizing the reward move the north-star?) and learnability (can the bandit identify the reward-optimal policy?). It is an ordered five-step screen: P1 (alignment, arm-level directionality), P2 (tool-fit, contextual vs. fixed/context-free), P3 (learnability, reward density and online replay), P4 (off-policy hygiene, correct intervals and 1/K overlap), and P5 (delay budgeting, time-to-significance).

The authors validate the protocol where the truth is known—on a controllable synthetic generator and on public benchmarks (Open Bandit Pipeline, UCI covertype, sklearn-digits)—and illustrate it on a deployed large-marketplace push-notification system (with five arms and one split, the evidence is directional rather than powered).

Two headline lessons recur:

• (N1) Reward density governs online learning efficiency. A single batch value can rank two rewards as tied when one in fact induces faster online learning. The fix is still offline: a step-by-step replay of the same log recovers the learning trajectory the single number hides. On synthetic ground truth, two rewards with the same optimal policy but different observation density (100% vs. 10%) have tied batch values (+0.0pp gap) but separate online (+50pp gap before convergence). This reproduces on UCI covertype and on OBP synthetic data.

• (N2) When the best single arm is unidentifiable from training data, the contextual policy's edge cannot be cleanly attributed to per-user personalization. A robustness reading is at least as consistent—the policy partly just avoids betting on the wrong arm. On a ground-truth generator with zero true personalization headroom, the doubly-robust achievable-ceiling check reads ≈0 (from −0.1pp at K=3 to −5.9pp at K=20), while with real conditional structure it is large (+72pp to +143pp). The deployment's near-zero reading is therefore the signature of absent personalization, not a noise artifact.

Additional findings include:

• The interval trap. With near-uniform logging and K arms, a deterministic policy matches only ≈1/K of logged traffic, so per-estimate variance dwarfs any plausible lift. A casually-computed bootstrap interval can make every candidate look significant when none is; the same data under a correct percentile interval shows the opposite. On the real Open Bandit Dataset, the Direct Method returns a 95% interval [0.00438, 0.00474] that is ≈20× tighter than the IPW family's (≈0.008) yet excludes the truth (0.00420), while IPW/SNIPW/DR are wide but cover it.

• Marginal versus conditional. A marginal arm discrimination test can declare a reward flat across arms precisely when the effect a contextual bandit exploits is conditional on the user—present, but invisible to a test that averages over contexts.

• The surrogate paradox (P1). A proxy can correlate positively with the north-star at the unit level yet be anti-aligned across arms. On synthetic data, a unit-level Spearman ρ=+0.77 passes a naive check, but the arm-level rank correlation is −1.0, and optimizing the proxy lands on the worst arm (−50% true north-star vs. random). On UCI covertype, a routine class-rebalancing step (up-weighting rare classes by (1/π k) γ) induces the same failure: as γ grows from 0 to 4, the proxy-optimal policy's true accuracy slides from +50% over the best fixed arm to −83%, while the unit-level correlation stays positive well past the point where the policy already hurts the north-star.

The deployment case study found that the deployed same-day reward was anti-aligned at the arm level (ρ≈−0.8), preferring arms that converted worst—a surrogate paradox caught by P1 before launch. The protocol-chosen reward was later deployed in a new test; the 1-day reward would have confidently promoted arms that sit mid-to-bottom on the 25-day horizon, and the trained policy achieved significant causal matching over random assignment in the one segment where arms genuinely differed (+43%, z=3.11). The business topline was null, attributed to arm homogeneity in the segment producing most conversions.

The authors are explicit about provenance: the individual steps are not new (alignment draws on surrogate validity and directionality; learnability on reward informativeness; tool-fit on the value-of-personalization literature). The only genuinely new machinery is narrow—replaying a randomized log to rank rewards (not policies) by the learning they induce, and an achievable-ceiling estimate as a fast personalization-headroom screen. Their contribution is the ordered composition of these checks for the delayed-feedback setting and the two findings N1 and N2.

Threats to validity include: single real setting; offline and counterfactual replay (single deterministic path, optimistic CIs); overlapping intervals on the deployment; reimplemented baselines; underpowered directionality on the deployment; N2's zero-headroom case being synthetic-only; delay being budgeted rather than modeled; uniform-random logging assumptions; no automated thresholds; and data confidentiality (all numbers relative).

Based on the paper, here are the specific improvements I can make to AI systems, and what the improved systems can do:

1. Add a reward density replay module to offline evaluation pipelines.

Instead of comparing candidate rewards by a single batch metric (e.g., AUC or average reward), the AI system will simulate a step-by-step replay of the logged data to estimate the learning trajectory (e.g., regret vs. number of interactions) for each reward. This lets the system rank rewards by how quickly and stably they enable online learning, not just by final performance. The improved system can then select a proxy reward that is 5× more sample-efficient, even when batch metrics show a tie.

2. Implement a surrogate paradox pre-check (P1) before any reward optimization.

The AI system will compute two correlations: (a) unit-level correlation between proxy and north-star, and (b) arm-level rank correlation between the proxy-optimal policy and the true north-star-optimal policy. If these disagree in sign, the system will flag the proxy as anti-aligned and refuse to use it for policy learning. This prevents deploying a proxy that appears positively correlated but actually drives the system to the worst-performing arm (e.g., catching a ρ=+0.77 unit-level but −1.0 arm-level case before launch).

3. Add a personalization headroom ceiling estimator (N2) to contextual bandit selection.

Before committing to a contextual policy, the AI system will compute an achievable-ceiling estimate (e.g., doubly-robust upper bound on lift from personalization) from the logged data. If this ceiling is near zero (e.g., <1pp), the system will default to a fixed best arm rather than a complex contextual model. This prevents overfitting to noise and avoids attributing spurious lift to user-specific features when the data shows no true conditional structure.

4. Replace naive bootstrap intervals with overlap-aware, IPW-family-corrected confidence intervals.

The AI system will automatically detect when logging is near-uniform and K is large (e.g., K>5), and then use IPW/SNIPW/DR intervals with a 1/K overlap correction, rather than Direct Method or naive percentile bootstrap. This ensures that reported significance is not inflated by variance underestimation—e.g., avoiding the 20× too-tight interval that excludes the true value, while still producing wide but honest intervals that cover it.

5. Add a delay budget scheduler (P5) for reward selection under delayed feedback.

The AI system will compute the expected time-to-significance for each candidate reward given its observation density and the delay distribution. It will then reject any reward that cannot reach statistical significance within the business’s decision deadline, even if it appears aligned. This prevents the system from committing to a proxy that will only yield confident signals after the north-star is already obsolete.

6. Implement a marginal-vs-conditional discrimination test for reward flatness.

When a marginal test (e.g., ANOVA across arms) says no arm differences, the AI system will automatically run a conditional test (e.g., per-user-segment or per-feature-bin) to check for hidden personalization headroom. If conditional differences exist, the system will proceed with contextual learning; if not, it will flag the reward as truly flat and avoid wasting compute on a bandit that cannot exploit user features.

7. Add a policy attribution guard for contextual edge claims.

When a contextual policy outperforms a fixed best arm, the AI system will decompose the lift into (a) personalization gain vs. (b) risk-avoidance gain (i.e., not betting on a bad arm). It will do this by comparing the contextual policy’s arm-selection distribution to the best single arm’s distribution under the same logged data. If most of the lift comes from avoiding the worst arm rather than from per-user targeting, the system will report the edge as robustness not personalization, preventing misleading business narratives.

8. Add a class-rebalancing surrogate paradox detector for supervised pre-training.

For any reward that is a reweighted or rebalanced version of a base label (e.g., up-weighting rare classes), the AI system will check whether the reweighting changes arm-level alignment with the true north-star. It will compute the arm-level rank correlation between the reweighted proxy and the north-star across γ values (reweighting strength) and stop at the first γ where the sign flips, even if unit-level correlation remains positive. This prevents a routine preprocessing step from silently turning a good proxy into a harmful one.

9. Build a five-step screen orchestrator for any delayed-feedback bandit deployment.

The AI system will run P1→P2→P3→P4→P5 in order before any live experiment: (P1) arm-level alignment, (P2) contextual-vs-fixed fit, (P3) reward density replay, (P4) off-policy interval hygiene, (P5) delay budget. It will output a go/no-go decision with a confidence label (e.g., directional vs. powered) and refuse to launch if any step fails. This makes the system safe for high-stakes, slow-feedback settings like marketing, healthcare, or finance, where a bad proxy can cause irreversible harm.

10. Add a synthetic ground-truth validation harness for any new reward or policy.

Before applying the protocol to real data, the AI system will generate a controllable synthetic environment with known north-star and personalization headroom, then run the same five-step screen on it. If the screen fails to detect a known surrogate paradox or overclaims personalization on zero-headroom data, the system will reject its own output and request human review. This builds self-verification into the AI’s decision pipeline, reducing silent failures.