Certifying What Helps Customer-Return Timing: A Screen-and-Confirm Test for Conditioning Signals, and Why Decay Is Nearly Enough

This paper introduces a screen-and-confirm protocol to certify whether a candidate signal improves a temporal point process (TPP) model's event-timing likelihood, and a model-free ceiling quantifying how little of customer-return timing is point-predictable. Using these tools, the authors certify that continuous-time decay is nearly sufficient for customer-return timing, and that conditioning signals (LTV, category, RFM, calendar, geography) are redundant or harmful on top of it.

The paper's contributions are: (C1) a screen-and-confirm certification method where a positive control plants a coupling of known strength and confirms the model recovers it, so a real-data null reads as no signal rather than weak method; (C2) a model-free ceiling showing that ≲ 5% of gap variance is explainable from any covariate, meaning returns are near-memoryless; and (C3) the certified result that on three public benchmarks (Amazon, Taobao, RetailRocket) and a real marketplace (Thumbtack), continuous-time decay is nearly sufficient, and conditioning is redundant or harmful on top of it.

Key results include: adding a decay head to a plain Transformer-Hawkes (THP → THP-D) yields large temporal-NLL gains on every dataset (Amazon 0.32 → −2.55, Δ −2.9; RetailRocket 0.52 → −3.53, Δ −4.0; Taobao −2.09 → −2.58, Δ −0.5; Thumbtack Δ −1.26). The gain is distributional—it sharpens the timing likelihood, not point error. Once decay is present, every conditioning signal is null or harmful: category worsens it (CatOnly-D +0.47; ExogCat-D +0.65), exogenous calendar/geography is null (ExogTHP-D +0.012), RFM collapses to ≈ 0 once target leak is removed, and proxy-LTV does not improve likelihood (LTV-CTPP-D +0.47). On public benchmarks, under decay every signal moves temporal NLL by ≲ 0.06.

The model-free ceiling shows no available covariate explains more than a single-digit percentage of gap variance on any of the four datasets: on Thumbtack the previous gap is the best single predictor at ≈ 2.2% (r = 0.147), category ≈ 1.7%, and season and DMA region explain effectively nothing (η2 ≈ 5–8 × 10−4); on public benchmarks the pattern repeats (previous gap 1.4–6.3%, event-type/mark η2 ≤ 1.1%). The best single covariate is always the previous gap—the inter-event clock itself.

The positive control validates the protocol: on synthetic data, a planted exogenous→gap coupling of strength β is recovered monotonically in both categorical and continuous encodings (ExogTHP improves by −0.059, −0.232, −0.824 at β = 0.5, 1, 2; decay variant ExogTHP-D recovers −0.042, −0.143, −0.337; continuous variant yields Δ time-NLL +0.000, −0.092, −0.258, −0.738 at β = 0, 0.5, 1, 2). β = 0 is correctly null in both. A real positive control on NYC green-taxi pickups shows the screen fires where a genuine exogenous signal exists: encoding hour-of-day as a continuous cyclic covariate lowers temporal NLL by Δ = −0.025 (1.756 → 1.731), replicated on a second month (Δ = −0.021).

The paper also documents honest-evaluation pitfalls: an NHP target-indexing bug produced a fake 7× MAE improvement; an RFM cadence feature leaked the target through a full-sequence normalizer; an S2P2 RMSE was read from an untrained auxiliary head; and the decay head's point-prediction read the hidden state after decaying it by the actual next inter-event interval, leaking the gap it predicts. Each was caught and corrected.

The mechanism for redundancy is dual: (A) sequence-derivable signals (RFM, category) are already captured by the decay backbone; (B) exogenous signals (season, region) shift demand composition but not per-customer timing of the next return. The paper concludes that for customer-return timing, the inter-event clock (decay) is the reliable lever, and before believing a feature helps, one should screen it against a positive control so that a null means no signal in the data, not a method too weak to find it.

Improvements to AI systems:

1. Add a screen-and-confirm certification module to any temporal point process (TPP) model. Before trusting a new covariate or conditioning signal, the system runs a positive control: it plants a synthetic coupling of known strength (e.g., exogenous→gap β) and verifies the model recovers it monotonically. If the control fails, the system rejects the real-data null as method too weak, not no signal. This prevents false negatives in feature engineering.

2. Integrate a model-free ceiling estimator into the training pipeline. The system computes the maximum explainable variance of inter-event gaps from all available covariates (e.g., via η2 or correlation with previous gap). If the ceiling is below a threshold (e.g., <5%), the system automatically down-weights or drops conditioning signals, avoiding overfitting to noise and reducing model complexity.

3. Replace or augment attention-based TPP heads with a continuous-time decay head. The system learns an explicit exponential-decay kernel for inter-event timing, which yields large likelihood gains (e.g., Δ temporal-NLL −2.9 on Amazon, −4.0 on RetailRocket) without needing external features. This makes the model more accurate for customer-return prediction and similar event-timing tasks.

4. Implement a leak-detection audit for target-derived features. The system checks whether any input feature (e.g., RFM cadence, normalized sequences, or hidden states) indirectly encodes the target inter-event interval. It flags and corrects leaks by comparing predictions against a held-out target or using a randomized ablation. This prevents inflated performance metrics (e.g., fake 7× MAE improvements).

5. Add a redundancy-check mechanism for sequence-derivable vs. exogenous signals. The system classifies each candidate covariate as either (a) already captured by the decay backbone (e.g., category, RFM) or (b) exogenous (e.g., season, geography). For (a), it verifies no additional likelihood gain; for (b), it tests if the signal shifts demand composition but not per-customer timing. If redundant or harmful, the system excludes it, simplifying the model and improving generalization.

6. Enable a null-aware feature evaluation mode. The system reports not just whether a feature improves likelihood, but whether the improvement exceeds what a positive control would produce at β=0. This ensures that any reported gain is statistically meaningful and not due to noise or leakage.

What the improved AI system can do:

• Certify feature usefulness for event-timing models with high confidence, avoiding both false positives (noise-driven gains) and false negatives (missed real signals due to weak methods).

• Predict customer-return timing with near-optimal accuracy using only the inter-event clock (decay), achieving state-of-the-art temporal NLL on public benchmarks and real marketplaces, while ignoring redundant or harmful covariates.

• Automatically detect and correct data leaks in training pipelines, preventing over-optimistic evaluation and ensuring deployed models perform as expected in production.

• Quantify the theoretical limit of predictability for any given dataset, allowing users to set realistic expectations (e.g., only 2% of return-timing variance is explainable) and avoid wasted effort on feature engineering.

• Generalize to other temporal event domains (e.g., user engagement, equipment failure, healthcare visits) by applying the same screen-and-confirm protocol and decay-head architecture, with minimal tuning.