An Identifiability Theory of Masked Prediction: Mode Blindness and Mask Schedules

arXiv: 2608.01383v2 [cs.LG], 7 Aug 2026

The paper studies a foundational identifiability question for masked prediction: When does near-optimal masked prediction identify the underlying joint data law? Masked prediction learns representations by fitting a schedule-weighted family of conditional laws, where the mask is drawn from a prescribed schedule. Different schedules underlie BERT-style pretraining, masked autoencoding, absorbing-state discrete diffusion, and masked latent prediction. Although these methods differ architecturally, they all optimize a schedule-weighted collection of conditional distributions rather than the joint law itself.

The authors motivate the problem with a thought experiment: "Suppose the data consist of two well-separated regimes, such as code and prose, and we change only their mixing proportions while leaving the law within each regime unchanged. Once the visible context is long enough to reveal the regime, the conditional laws become almost independent of the global mixture. A model can therefore predict masked blocks nearly optimally while assigning substantially different weights to the two regimes. We call this phenomenon mode blindness. Crucially, whether it occurs is determined not by the model class, but by the mask schedule."

The central theoretical object is the ε-identifiability modulus (Definition 2.2):

Fix a mask schedule µ and a distance d continuous on ∆(X) × ∆(X). For a data law p ∈ ∆(X) and ε ≥ 0, define the ε-identifiability modulus Aµd (p, ε):= sup d(p, q): q ∈ Q(p), Dµ (p∥q) ≤ ε.

The modulus measures the largest distributional error among model laws whose masked discrepancy from p is at most ε. It separates the two directions of an identifiability claim: A lower bound on Aµd requires only a single explicit model q ∈ Q(p) inside the discrepancy budget, whereas an upper bound must control every admissible model in that budget. At ε = 0, the modulus measures exact objective-level non-identifiability; for ε > 0, it quantifies the stability of joint-law recovery.

The model class is restricted to admissible models Q(p):= q ∈ ∆(X): supp(p) ⊆ supp(q), i.e., support-covering models.

The fixed-mask log discrepancy is defined (Definition 2.1) as:

DK (p∥q):= EXK ∁ ∼pK ∁ [DKL (pK (· XK ∁) ∥ qK (· XK ∁))].

The schedule-averaged discrepancy is Dµ (p∥q):= EK∼µ DK (p∥q). The paper notes the excess-risk interpretation: Dµ (p∥q) is the population excess masked-prediction risk of q relative to p.

The paper considers data with two well-separated global modes, partitioning the sample space into X = X+ ⊔ X− with p(Xτ) > 0 for both modes. The true mode weight is w:= p(X+) ∈ (0, 1). For each mode τ, the conditional mode law is pτ (·):= p(· Xτ).

The mode-reweighted law is defined as qλ:= λ p+ + (1 − λ) p− for λ ∈ (0, 1). Since p+ and p− are supported on disjoint regions, qλ preserves within-mode conditional laws while setting the global mode mass to qλ (X+) = λ.

Proposition 3.1 (Mode-weight information decomposition) states that for every mask K with visible set K ∁:

DK (p∥qλ) = EXK ∁ ∼pK ∁ kl βK ∁ (XK ∁) ∥ βλ,K ∁ (XK ∁), DK (p∥qλ) = kl(w∥λ) − DKL (pK ∁ ∥qλ,K ∁).

The interpretation: "the fixed-mask discrepancy equals the total mode-weight information kl(w∥λ) minus the portion already revealed by the visible marginal. A large context reveals nearly all of it, leaving little discrepancy; an empty visible context reveals none, so the full mask recovers D[N] (p∥qλ) = kl(w∥λ)."

Assumption 4.1 (Large-context mode pinning) requires that for visible sets V of size V ≥ spin, the typical set of visible contexts EVτ:= xV ∈ supp(pτV): p(X−τ XV = xV) ≤ e−κV captures overwhelming mode mass, satisfying pτV (EVτ) ≥ 1 − e−c1 V. The combined rate is c:= min c1, κ.

The paper links this to low-temperature spin systems: "In the Curie–Weiss model below the critical temperature, the Gibbs measure concentrates near two magnetization basins, and as the visible subset of spins grows, standard large-deviation estimates reveal the sign of the magnetization with exponentially small error."

The first main result formalizes the thought experiment:

"Suppose Asm. 4.1 holds with combined rate c = min c1, κ. Consider the mode-reweighting family qλ ∈ Q(p): λ ∈ Ibl, where qλ is defined in Eq. (12) and the mode-weight interval is Ibl:= [c0 /2, 1 − c0 /2]. There exists a constant C ≤ 6c−2 0, depending only on c0, such that every mask schedule µ with vmin (µ) ≥ spin satisfies supλ∈Ibl Dµ (p∥qλ) ≤ C EK∼µ e−cK ∁ ≤ Ce−c vmin (µ)."

Consequently, the modulus is macroscopic already at exponentially small tolerance:

AµTV (p, ε) ≥ (1 − c0)/2, for every ε ≥ C EK∼µ e−cK ∁.

The paper states: Read together, Eqs. (20) and (21) state a specific form of approximate population non-identifiability: a constant total-variation gap survives at a tolerance that is exponentially small in the visible-context size.

The same witness family forces approximate-tensorization constants to grow exponentially:

"Suppose Asm. 4.1 holds with combined rate c = min c1, κ. Let µ be a mask schedule with vmin (µ) ≥ spin. Then there exists λ ∈ Ibl such that C̄AT (qλ, µ) ≥ c20 (1−c0)2 c vmin (µ) e and C̄AT (p, µ) ≥ c40 (1−c0)2 c vmin (µ) e." (with appropriate constants 12 and 32 in denominators)

The paper notes: the failure of rapid-mixing recovery reflects the objective itself rather than a limitation of existing analyses.

The second main result shows how the mask schedule may resolve the obstruction:

Fix a compact I ⊂ (0, 1) containing w, and let cI, CI be the constants of Lemma 4.1. For every mask schedule µ and every λ ∈ I: cI kl(w∥λ) EK∼µ mmsep (Z XK ∁) ≤ Dµ (p∥qλ) ≤ CI kl(w∥λ) EK∼µ mmsep (Z XK ∁).

Under both structural assumptions, for every visibility budget s with spin ≤ s ≤ sov:

"Dµ (p∥qλ) ≥ cI u0 e−Ls πs (µ) kl(w∥λ), Dµ (p∥qλ) ≤ 45 CI πs (µ) + EK∼µ e−cK ∁ 1 K ∁ > s kl(w∥λ)."

The interpretation: the discrepancy of a reweighted law is the mode-weight information kl(w∥λ) times the schedule-averaged residual MMSE. The low-visibility mass πs (µ) carries the retained sensitivity, while the high-visibility tail is exponentially suppressed by pinning.

"If πs (µ) > 0, then Dµ (p∥qλ) ≤ ε =⇒ w − λ ≤ eLs/2 √(ε/(2cI u0 πs (µ)))."

This is a stability guarantee along the reweighting family: near-optimality at ε forces w − λ = O(√(ε/πs (µ))). The converse direction is also provided.

The directional limitation vanishes for the full mask:

"If π0 (µ) > 0, then every q ∈ Q(p) satisfies Dµ (p∥q) ≥ π0 (µ) DKL (p∥q), and consequently AµTV (p, ε) ≤ √(ε/(2 π0 (µ)))."

Unlike the directional guarantee, Prop. 4.1 controls every admissible model: full-mask mass alone converts a discrepancy budget into a joint-law bound.

The mode-blindness direction extends to M ≥ 2 well-separated regions:

"Suppose the M-mode pinning assumption above holds with combined rate c = min c1, κ, and let Λ:= λ ∈ ∆M −1: λj ≥ c0 /2 ∀j. There is a constant C ≤ 12c−2 0, depending only on c0, such that every mask schedule µ with vmin (µ) ≥ spin satisfies supλ∈Λ Dµ (p∥qλ) ≤ C EK∼µ e−cK ∁ ≤ Ce−c vmin (µ)."

The modulus lower bound becomes AµTV (p, ε) ≥ 1/M − c0 /2 ≥ 1/(2M).

The paper computes the low-visibility mass πs (µ) for several masking paradigms:

• Untruncated masked-diffusion schedules: πs (µdiff) ∼ (s + 1) g(1)/N, i.e., Θ(N −1) for fixed s.

• Schedules clipped away from full masking: πs (µ) = e−Θ(N).

• Fixed-ratio masking: πs (µ) = 0 for every s < N − ⌊t0 N ⌋.

• Structured object and block masking: πs (µ) = 0 for every s < αN.

The paper concludes: Across these paradigms, the asymptotic behavior of the low-visibility mass πs (µ) is determined by how much schedule mass lies near full masking.

The paper constructs an explicit law satisfying both Asm. 4.1 and Asm. 4.2 simultaneously, with all constants in closed form. The law has A = −1, +1, N odd, and a parameter θ ∈ (0, 1). The partition is Xτ:= x ∈ X: τ m(x) > 0 where m(x) is the empirical magnetization. Each mode law is a product law conditioned on its region. Proposition F.1 verifies both assumptions with explicit constants: c0 = 1/2, κ = L0 θ/2, c1 = θ2 /16, spin = 16 ln 2/θ2, L = L0 + 1, sov = ⌊αN ⌋ where α = θ/(2(1 + θ)).

The paper reports exact enumerations on the closed-form law:

• Mode blindness (Fig. 2a): For the reweighting witness with w − λ = 1/4, "the masked discrepancy collapses by 171 orders of magnitude as the visible context grows, while DTV (p, qλ) stays at 0.25, the certified gap of Eq. (21); the measured rate matches the r⋆ (θ) = 0.5108 of Remark F.1 to 0.27% at N = 1023."

• Sensitivity (Fig. 2b): across all 49 evaluated cells, the normalized ratio of Lemma 4.1 lies in [4.000, 4.408], on the parameter-free Fisher limit 1/(w(1 − w)) = 4.

• Boosted recovery (Fig. 2c): "mixing near-complete masks of mass πs (µ) = 10−2 into a fixed-ratio base schedule lifts the witness discrepancy by 113 orders of magnitude (Thm. 4.2), and at ε = 10−4 the recovery radius shrinks at log–log slope −0.49, matching the πs (µ)−1/2 scaling of Cor. 4.3."

The paper trains a small model on data from Law P under two-point schedules:

"Under the blind schedule the five seeds do not converge on w: they end at ŵ ∈ 0.0002, 0.0003, 0.0022, 0.6129, 0.6921, three runs assigning the + mode near-zero mass and two settling at interior weights, for a mean error of 0.36. Any positive full-mask mass removes the failure, and the error then falls as π0 (µ)−0.44±0.09, an interval that contains the −1/2 of Cor. 4.3, reaching 9.5 × 10−4 at π0 (µ) = 10−1, a factor of 380 below the blind baseline."

The paper measures residual mode uncertainty on code–prose and German–English corpora:

"The estimate is bounded below at short contexts and suppressed by at least 91× over the measurable range, matching the required shape; its functional form is a power law near V −1 rather than an exponential (R2 ≥ 0.97 against ≤ 0.95; fitted exponents −0.94 to −1.15)."

The paper notes: at mmsep ∝ V −1, a fixed-ratio schedule with K ∁ = Θ(N) retains the same Θ(N −1) mode-weight sensitivity as untruncated diffusion's low-visibility channel, so blindness is milder than the pinned worst case.

The paper situates itself among several lines of work: masked objectives as conditional estimation (pseudolikelihood, BERT, MAE, discrete diffusion), recovery guarantees via approximate tensorization in rapid-mixing regimes, metastability of low-temperature spin systems, empirical inconsistency of masked conditionals, and population-level analyses of pretraining objectives (contrastive learning, next-token prediction).

The paper concludes:

"This work establishes population-level identifiability limits of masked prediction through an ε-identifiability modulus that puts the failure of joint-law recovery on the same footing as its success. For data with well-separated modes, schedules dominated by large visible contexts are mode blind: because these contexts resolve the underlying mode, the objective reduces to within-mode conditional matching and becomes exponentially insensitive to the global mode weights, a failure intrinsic to the objective. An exact information decomposition separates what remains: low-visibility mass restores mode-weight sensitivity through residual mode uncertainty, while full-mask mass anchors the joint law over all admissible models. Our opening question thus has a schedule-dependent answer: on the same data at the same near-optimal loss, the joint law's global structure is recoverable or invisible according to these two masses alone."

The paper also notes: The mask schedule is therefore a determinant of what pretraining identifies, not merely of how efficiently it trains.

Based on the paper, here are the specific improvements I can make to AI systems:

Improvement: Add a pre-training diagnostic that computes the schedule's low-visibility mass πs(µ) and full-mask mass π0(µ) before training begins.

What the improved system can do:

• Detect when a mask schedule is mode-blind (e.g., fixed-ratio masking with πs(µ) = 0 for all s = o(N)) and warn the practitioner that global mode weights (e.g., code vs. prose proportions, language mixing ratios) will be unrecoverable from the objective.

• Automatically recommend a minimum full-mask mass π0(µ) > 0 or low-visibility mass to restore identifiability, with the trade-off quantified: the recovery radius scales as √(ε/π0(µ)) and the sample cost scales as 1/π0(µ).

These improvements are directly implementable from the paper's theorems (Thm. 4.1, Thm. 4.2, Prop. 4.1, Cor. 4.3) and the exact information decomposition (Prop. 3.1), with all constants computable from the data law and schedule.

Masked prediction learns representations by fitting a schedule-weighted family of conditional laws, but it remains unclear when near-optimal conditional prediction pins down the underlying joint law. We study this question for data with two well-separated global modes, outside the reach of rapid-mixing recovery guarantees, and show that the answer is decided by the mask schedule alone. Under large-context mode pinning, reweighting the two modes can move the joint law by a constant in total variation while perturbing the masked objective exponentially little in the visible-context size: mask schedules dominated by large contexts are provably blind to the global mode weights. To quantify this, we introduce an epsilon-identifiability modulus, the largest distributional error consistent with a given excess risk, and prove that it remains macroscopic at an excess risk that is exponentially small. An exact information decomposition pinpoints what restores identifiability: mode-weight sensitivity is governed by the residual mode uncertainty given the visible context. Consequently, low-visibility masks recover this sensitivity, and positive full-mask mass anchors the joint law over all admissible models with no assumption on the data law. Empirically, we test our theory at three levels: enumeration on computable laws verifies the predicted rates, gradient training reproduces both the mode blindness and the recovery, and measurements on real corpora place natural text between the two certified regimes.