The JEPA Predictor: A Transferable Operator for Occluded Feature Completion

Joint-Embedding Predictive Architectures (JEPAs) introduce a trainable predictor head that maps visible-context features and target-position tokens to predicted target features in an encoder’s representation space, and this work demonstrates that this predictor functions as a portable operator for occluded feature completion across different image encoder families. This finding is significant because it shows that a frozen, pretraining predictor can recover accuracy lost to occlusion when paired with any frozen image encoder through a single linear projection, providing a generic completion module without requiring retraining of the backbone or the predictor itself.

How it works

The core protocol involves wiring a frozen JEPA predictor to any frozen image encoder through a single closed-form linear projection between feature spaces, with no backbone retraining. Specifically, for non-JEPA hosts, a linear projection matrix W is fit in closed form on 500 ImageNet-1k images to minimize the per-token Mean Squared Error (MSE) between host features and donor encoder features at matched patch positions. At test time, the host encodes visible patches, W projects these into the donor space, and the frozen predictor returns predicted features at masked positions. This process is summarized in Algorithm 1: Fpartial ← f(x) ⊙ (1 − up(M)) followed by H ← W Fpartial and Fpred ← π H, pos(M).

Key Experimental Setup

The protocol was benchmarked across four non-JEPA hosts (CLIP, DINOv3, DINOv2, MAE), two JEPA donors (I-JEPA and V-JEPA 2), three mask fractions (K ∈ 0.25, 0.56, 0.77), and two datasets: ImageNet-9 and Stanford Dogs. The evaluation metric is the matched probe accuracy, where a linear probe is trained on mean-pooled completed features at the same K used at test time to read off the classification of the completed sequence. The results show that across both datasets, the lift over each host’s masked-encoder baseline grows monotonically with K in every host–donor pair.

Mechanism: Cost-Benefit Balance

The lift is explained by a single cost-benefit account: Cost on visible patches, benefit on masked patches. The linear projection W incurs a cost on all visible patches, as its ridge fit explains only a fraction of donor-feature variance (measured by the lossiness metric R2W). Conversely, the predictor provides a growing benefit on masked patches. The net lift becomes positive when masked-position benefit overtakes visible-position projection cost. Furthermore, the mechanism is identifiable: the projection pays a fixed cost on visible patches and the predictor provides a growing benefit on masked patches; the benefit dominates the heavy-occlusion regime.

Performance and Generalization

The cross-encoder transfer protocol yields significant results. At heavy mask, 7 of the 8 host–donor pairs show a positive lift with Welch-t 95% CI strictly above zero on both datasets. The lift is sharpest on CLIP, which shows the largest recovery, reaching +36.18pp on Stanford Dogs at K=0.77 with the I-JEPA donor. This suggests that the host–predictor complementarity dictates performance, as evidenced by the ordering of hosts based on bridge quality (R2W). The within-JEPA specialization (using a JEPA encoder paired with its own frozen predictor) maintains high accuracy, exceeding the strongest encoder-only baseline by +9.51pp at K=0.77 on ImageNet-9.

Diagnostic Implications and Limitations

The study identifies the regime where the linear bridge breaks down as low K on fine-grained classification, where the projection cost exceeds the benefit, leading to negative lifts in Stanford Dogs. A methodological consequence is noted regarding V-JEPA 2: an L2-only protocol would incorrectly flag its predictor outputs as uninformative because they live in a different L2 frame than the encoder’s. The protocol also demonstrates that contiguous occlusion (object-on-object occlusion, bounding-box crops) provides the benefit, while lift collapses on random Bernoulli masking. Finally, the work notes that broader transfer to non-natural-image domains is left for future research.

Conclusion

The frozen JEPA predictor acts as a transferable operator because it learns a function of feature-position and context that any encoder approximately respects. The protocol provides a deployable K-agnostic probe that recovers the predictor’s benefit across occlusion levels, confirming that the transfer is a property of the predictor’s completion-operator, not of any one feature geometry. This result holds for both native and transplanted predictors.

Here are specific improvements to AI systems derived from the findings in this scientific paper, categorized by application domain:

)Specific Improvements for AI Systems

The core innovation is a transferable operator (the frozen JEPA predictor) that enables feature completion across different encoder families (CLIP, DINOv3, MAE). This allows deploying powerful, pre-trained completion knowledge without retraining the host model. The key mechanism is a fixed cost on visible patches and a growing benefit on masked patches.

Here are specific improvements:

1. GenAI/Image Inpainting Systems:

2. Domain-Specific Inpainting for High-Fidelity Reconstruction:

3. Cross-Modal Feature Completion in Video Understanding:

4. Robustness to Occlusion in Autonomous Driving Perception Models:

5. Efficient Deployment of Large Foundation Models for Visual Tasks (The Portable Operator):

)What the Improved AI System Can Do (Specific Applications)

Based on the paper's results, an improved AI system leveraging this protocol can achieve the following capabilities:

1. Predictive Image Restoration with Encoder-Agnostic Knowledge:

2. Zero-Shot Feature Completion for Fine-Grained Classification from Partial Inputs:

3. Unified Feature Extrapolation Across Different Pretraining Architectures (e.g., using CLIP knowledge to complete DINOv2 features):

4. Real-Time Occlusion Handling in Autonomous Systems by Utilizing Frozen Prediction Heads:

5. Efficient Deployment of Large Vision Models with Minimal Inference Overhead:

)Detailed Capability Breakdown and Mechanism Mapping

Application Area Specific System Improvement Mechanism Leveraged from Paper Expected Performance Gain

:---:---:---:---

1. GenAI/Image Inpainting Systems (e.g., Stable Diffusion inpainting) Implement a Portable Completion Module that allows an image encoder (like CLIP or DINOv2) to complete missing regions using a frozen, pre-trained JEPA predictor, without needing to fine-tune the host model for every new mask fraction. The protocol uses a single linear projection bridge (W) and the frozen donor predictor (I-JEPA/V-JEPA 2) to inject contextually relevant features at masked positions. Significantly higher fidelity reconstruction compared to standard pixel-space completion or nearest-neighbor methods, especially under heavy occlusion, by leveraging the predictor's learned prior.

2. Zero-Shot Feature Completion for Fine-Grained Classification (e.g., Wildlife/Medical Diagnosis) Create a Transferable Classifier that can classify fine-grained entities (like Stanford Dogs breeds or subtle medical anomalies) using an encoder not specifically trained for that task, by simply bolting on a JEPA predictor and fitting the bridge W. The results show the lift over hosts like CLIP is maximized when the predictor's benefit dominates the projection cost. This allows non-JEPA encoders to achieve high accuracy if paired with a strong donor predictor. Recovering significant accuracy gains (e.g., +36pp on Stanford Dogs at heavy occlusion) where standard encoder baselines fail, specifically for fine-grained tasks where subtle features are critical.

3. Cross-Modal Feature Completion in Video Understanding (e.g., Action Recognition) Use the cross-encoder transfer protocol to complete occluded frames in video sequences by projecting the host's frame features into the donor's feature space and using the predictor to infer masked action tokens. The mechanism is proven transferable across encoder families (CLIP, DINOv3, MAE) via a single linear bridge W, demonstrating that the predictor-completion operator is independent of the host geometry. Improved temporal consistency and motion prediction in videos with temporary occlusions by leveraging a predictor trained on context/target relationships.

4. Robustness to Occlusion in Autonomous Driving Perception Models (e.g., Object Detection/Segmentation) Deploy a frozen JEPA predictor onto an existing perception backbone (like DINOv3 or MAE) to handle sudden, localized occlusions in sensor data (e.g., temporary obstructions on a vehicle). The protocol identifies that the benefit grows monotonically with mask fraction K, meaning the system performs better as occlusion becomes heavier. The mechanism is identifiable: benefit dominates the heavy-occlusion regime. Enhanced safety by maintaining high classification accuracy when sensor inputs are partially obscured, specifically for contiguous occlusion scenarios (bounding-box crops).

5. Efficient Deployment of Large Foundation Models (The Portable Operator) Develop a standardized deployment pipeline where a pre-trained JEPA predictor can be plugged into any frozen encoder host via a one-time linear fit (W), eliminating the need to retrain either the encoder or the predictor for new deployment scenarios. The protocol requires no retraining of either model, only fitting the bridge W in closed form on 500 images, making it highly deployable and low-cost post-training. Drastic reduction in MLOps costs associated with fine-tuning foundation models for every new deployment task or data distribution.