Context-Adaptive Inference: A Unified Statistical and Foundation-Model View
Yue Yao, Caleb N. Ellington, Jingyun Jia, Baiheng Chen, Dong Liu, Rikhil Rao, Jiaqi Wang, Samuel Wales-McGrath, Yixin Yang, Zhiyuan Li, Eric P. Xing, Ben Lengerich
stat.ML, cs.LG, stat.ME
Submitted: 2026-07-25
Comments: 90 pages, 13 figures. Manuscript source and living version: https://github.com/AdaptInfer/context-review
Code: https://github.com/AdaptInfer/context-review
License: http://creativecommons.org/licenses/by/4.0/
The gist: Modern predictive systems are expected to adapt their behavior to the specific situation they are facing.
Terminology
Abstract
Modern predictive systems are expected to adapt their behavior to the specific situation they are facing. A clinical model should not treat every patient the same; a retrieval-augmented model should change its answer when given different evidence; a mixture-of-experts model should route different inputs to different experts. We call this capability context-adaptive inference: before predicting, the system uses information about the current context to specialize its parameters or computation for that instance. This article provides a unified view of context-adaptive inference across three traditions that are usually treated separately: (i) explicit adaptation in statistics (e.g. varying-coefficient models, local regression, hierarchical sharing), (ii) rapid task-specific adaptation in meta-learning and transfer, and (iii) implicit adaptation in large foundation models via prompting, retrieval, and expert routing. We formalize these approaches under a common objective: to map context c to adapted parameters theta(c), then to predict via f(x; theta(c)). Under squared loss, linear prediction heads, and fixed features, we prove that explicit parameter adaptation and implicit routing are mathematically equivalent to kernel ridge regression on joint features of inputs and context. Building on this bridge, we propose practical design principles and evaluation metrics including adaptation-efficiency, routing stability, and context-specific robustness to guide when to specialize, how to constrain that specialization, and how to audit context-adaptive models in deployment. Finally, we identify open problems in identifiability, robustness under distribution shift, and efficient large-scale adaptation, outlining design principles for methods that are scalable, reliable, and transparent in real-world settings.
Sources
- A Survey of Deep Meta-Learning
- LoRA: Low-Rank Adaptation of Large Language Models
- Foundational Models Defining a New Era in Vision: A Survey and Outlook
- A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPT
- Matching Networks for One Shot Learning
- Prototypical Networks for Few-shot Learning
- Online Learning: A Comprehensive Survey
- Contextures: Representations from Contexts
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
- Learning Transferable Visual Models From Natural Language Supervision
- On the Opportunities and Risks of Foundation Models
- Scaling Laws for Neural Language Models
- Training Compute-Optimal Large Language Models
- Graph Estimation From Multi-attribute Data
- Learning Sample-Specific Models with Low-Rank Personalized Regression
- Time-Varying Coefficient Model Estimation Through Radial Basis Functions
- Fast spatio-temporally varying coefficient modeling with reluctant interaction selection
- Spatially Varying Coefficient Models for Estimating Heterogeneous Mixture Effects
- Tree Boosted Varying Coefficient Models
- A tree-based varying coefficient model
Related papers
- Behavior of prediction performance metrics with rare events
- Optimal Estimation of Generic Dynamics by Path-Dependent Neural Jump ODEs
- A Posterior-Dynamics Framework for Imaging Inverse Problems with Pretrained Diffusion Priors
- Efficient quadratic entropy with distance sketches
- Embedding-Bias in Conditional Independence Testing
- From Log-Odds to Shapley Values: An Explanatory Geometry for the Weighted Naive Bayes Classifier