RAISE: Diagnosing Acquisition Collapse in Costly LLM Signals

The paper makes a distinction that is easy to miss: detecting that an auxiliary, model-derived signal helps on average is not the same as learning to act on it per instance. A reward–SNR floor governs when the second is even possible. Even when the acquired signal is genuinely faithful and an in-sample oracle that picks the top-b examples by realized reward shows a sizable apparent gain, no deployable policy can learn when to acquire it. The apparent learnable structure is order statistics of noise, not exploitable signal.

The paper states a formal necessary condition (Proposition 1): Consider detecting, at significance α and power 1 − β, that the per-example effect Δi of a costly observation on the reward has positive mean, from N examples with per-example SNR ρ = μ/σ. Detection requires:

ρ ≥ ρ⋆(N) = (z1−α/2 + z1−β)/√N ≈ 2.8/√N

equivalently N ≥ Nmin(ρ) = (2.8/ρ)2, at α=0.05, 1−β=0.8 (so z1−α/2 + z1−β ≈ 1.96 + 0.84 = 2.8).

The paper is explicit that this is a necessary mean-detectability condition, not a sufficient policy-learning/regret bound, and not an impossibility theorem. A policy that learns when to acquire must at minimum detect that the conditioned-on reward has signal; if even the average effect is below the floor, a heterogeneous policy conditioned on the same noisy reward is a fortiori undetectable.

Acquisition collapse at every granularity. Across per-impression, cluster (K=4−64), hand-defined regime, and uplift-tree granularities on three datasets, learned routing never beats random. A matched-moment i.i.d.-noise placebo reproduces ≥ 100% of the oracle's apparent gain (MIND +0.0518 = +0.0518). The oracle is order statistics of noise, not exploitable structure.

Dataset placement against the floor:

• MIND: N=1263, ρ=0.048, ρ⋆=0.0788, Nmin=3403, ρ⋆/ρ=1.64 (2.7× short of Nmin), effect ≈ 0 (positive)

• Amazon-Beauty: N=650, ρ=0.014, ρ⋆=0.1098, Nmin=40000, ρ⋆/ρ=7.85 (62× short), effect ≈ 0 (negative)

• REES46: N=498, ρ=0.138, ρ⋆=0.1255, Nmin=412, ρ⋆/ρ=0.91 (powered), effect significantly negative

The REES46 case matters for honesty: the acquisition collapse is not merely a power excuse, because the one dataset with enough power shows the acquired signal is net-negative, not net-positive.

Positive control. A synthetic signal injected at controllable SNR is recovered by the same deployable pipeline once cluster-SNR exceeds ≈ 0.20 (MIND) / 0.35 (Amazon); the real data sit at 0.075 / 0.056, an order of magnitude below. This establishes the floor is a true low-SNR limit rather than a broken/underpowered pipeline.

HTE-SNR sufficiency check. Correlation, out-of-fold R2, and a heterogeneity-SNR against a permutation null are all inside the null band on both datasets (MIND corr = −0.012, R2 = −0.010; Amazon corr = +0.001, R2 = −0.005): there is no learnable heterogeneity to exploit.

The paper instantiates the abstract costly observation in a concrete method: a frozen LLM decomposes a user history into K=3 ranked, confidence-scored, evidence-grounded intent hypotheses, embedded and fused as an input-embedding branch of a recommender. Each hypothesis hk carries (i) a calibratable confidence γk ∈ [0,1] and (ii) a set of evidence indices Ek ⊆ 1,...,n citing the history events that support it. The primary coordinate is a confidence-weighted best-facet match:

fBmax(c) = max k∈ 1,...,K γk · cos(c, ek)

The max-over-facets aggregation lets a candidate match any of the user's disjoint interests rather than a blended average, which a single summary cannot express.

• Faithfulness (grounded): On MIND, paired difference in cosine similarity between a hypothesis and its cited vs. non-cited history events is +0.0705 (95% CI [+0.068, +0.073]). On REES46 (87.9% single-category windows), the difference is ≈ 0, as expected.

• Distinctiveness: 1 − cos over hypothesis pairs is 0.204 on MIND versus 0.104 on REES46 (2× separation on multi-intent histories).

• Calibration: Raw top-1 confidence is over-confident (ECE 0.142, Brier 0.166 on REES46). Cross-fit isotonic regression reduces ECE to 0.031 (−78%).

Redundancy gradient: On MIND, the +SHE lift in NDCG@10 descends monotonically as the base gets stronger: L0 (subcategory popularity) +0.0161*, L1 (ID pair) +0.0146*, L2 (ID+text) +0.0100*, L3 (strong mean-pooled text) +0.0094 (ns). The sparse slice starts 3× higher at the weak end.

Controlled 2×2 test (MIND): Holding split, slate, labels and the late-fusion ranker fixed, varying only (unordered mean-pool vs. ordered GRU) × (no LLM vs. +SHE):

• Mean-pool base: 0.3701; Ordered GRU base: 0.3992

• +SHE over mean-pool: +0.0109 [−0.0046, +0.0277], p=0.165

• +SHE over ordered GRU: +0.0114 [+0.0030, +0.0209], p=0.005

• Redundancy gap (interaction): −0.0005 [−0.0164, +0.0150], p=0.919

The reading is careful: ordered access does not globally absorb the LLM signal. What is real is a regime split: the gap is positive on sparse/short histories (+0.033/+0.022, absorption) and reverses on long/multi-intent histories (−0.006/−0.005, complementarity).

Robustness (B1–B5): (B1) 5-seed sweep: every seed has GRU>mean-pool and significant SHE gain over GRU (+0.011 to +0.023). (B2) Degradation sweep: SHE gain stable and reliably significant only over ordered backbones. (B3) Residualization retains 101% of the gain (raw +0.0114 → resid +0.0115); sequence state explains SHE features with R2 ≈ 0.00–0.01. (B4) Redundancy probe gives R2 ≤ 0.010 on all slices. (B5) SASRec replicates the pattern: SHE gain over ordered SASRec +0.0179 [+0.0076, +0.0281], same regime split.

The paper's actionable recommendation: Do not learn per-instance acquisition from noisy offline rewards. Use pre-specified, design-time regime gates. Route SHE to (a) cold-start / underdetermined histories and (b) long multi-intent histories. The four-step recipe: (1) estimate reward SNR ρ and check against floor Nmin = (2.8/ρ)2 — if N < Nmin, do not attempt a learned per-instance router; (2) pre-specify a small number of gates from cheap, label-free slice features (history length, distinct-category count, sparsity), not from the reward; (3) enable the costly signal only inside the two value regimes; (4) validate the gate at the pooled-regime level with an out-of-fold 95% CI, never per instance.

On MIND (N=1263, over the strong content baseline), the gate calls the model on 86.4% of windows and matches spending everywhere: gate NDCG@10 = 0.4554 vs. spend-everywhere 0.4552 (difference +0.0001, 95% CI [−0.0038, +0.0042]). The gate's only realized benefit is a 14% reduction in expensive calls at no accuracy loss.

(1) The motivating production observation is observed, not controlled. (2) The detectability floor is a necessary mean-detection condition, not a policy-learning/regret bound. (3) Amazon-Beauty uses a local LSA space (embedding-API ACL), internally consistent but not identical to the ada-002 space used elsewhere. (4) SASRec is an additional ordered-backbone check at N ≈ 1.4k, not a claim of a uniformly stronger backbone. (5) All significance is at the sample sizes reached; several nulls are power-limited and labeled as such.

All offline results (R1–R13) are regenerated by a single command (scripts/reproduce.sh) from committed feature tables (CPU, no LLM calls); only from-scratch hypothesis generation needs an LLM. A 58-claim ledger (results/paper claims.csv) maps every claim to a script, a persisted CSV, and a figure.

Based on this paper, here are the specific improvements I can make to AI systems:

Improvement: Before training any model that decides when to acquire costly auxiliary signals (e.g., LLM-generated features, external API calls, human annotations), I will compute the per-example signal-to-noise ratio (ρ = μ/σ) of the reward conditioned on that signal, and compare against the floor Nmin = (2.8/ρ)2.

What the improved system can do: Automatically refuse to learn a per-instance acquisition policy when N < Nmin, preventing wasted compute and false confidence. For example, if I have 1,000 training examples and the auxiliary signal's effect on reward has ρ = 0.05, I will not attempt learned routing—I will use a fixed rule instead.

Improvement: Instead of training a classifier to predict when to use a costly signal (which the paper shows collapses to random), I will hard-code 2–4 gates based on cheap, label-free slice features (history length, distinct-category count, sparsity).

Improvement: When evaluating whether an auxiliary signal provides exploitable structure, I will generate a matched-moment i.i.d. noise placebo (same mean, variance, and sample size) and run the identical pipeline. If the placebo reproduces ≥100% of the apparent gain, I will conclude the gain is order statistics of noise.

Improvement: Before claiming any heterogeneous treatment effect (HTE) from an auxiliary signal, I will compute out-of-fold R2 and correlation against a permutation null. If these fall inside the null band (e.g., corr ≈ 0, R2 ≤ 0), I will conclude there is no learnable heterogeneity.

Improvement: When using LLM-derived features, I will decompose inputs into K=3 ranked hypotheses, each with (i) a calibrated confidence score (via cross-fit isotonic regression, reducing ECE from 0.142 to 0.031) and (ii) explicit evidence indices. I will aggregate via max-over-facets (fBmax) rather than mean-pooling.

Improvement: I will treat the auxiliary signal's value as a function of the base model's strength, not a constant. I will test over at least 4 backbone levels (weak popularity → strong sequence model) and only deploy the signal where the lift is significant.

Improvement: Before deploying any auxiliary feature, I will regress it out of the base model's sequence state. If the residualized feature retains ≥95% of its original gain (as in the paper: +0.0114 → +0.0115), I will conclude it is complementary, not redundant.

Improvement: When adding an auxiliary signal, I will test it against both an unordered mean-pool base and an ordered GRU base, holding everything else fixed (split, slate, labels, ranker). I will report the interaction term's confidence interval.

Improvement: Every null result will be labeled with its achieved power given the sample size and measured SNR. If N < Nmin, I will mark the result as power-limited null rather than evidence of no effect.

Improvement: I will maintain a machine-readable ledger mapping every quantitative claim to a specific script, persisted CSV, and figure, regenerable by a single command with no external API calls.