When Does Depth Matter For In-Context Learning? Adaptive Inference in Deep Transformers

arXiv:2606.16694 · cs.LG, cs.AI, physics.app-ph, q-bio.NC · Submitted 2026-06-15 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom, and with me are Jane, Lu, senior AI researcher at Tsinghua, Meng, lead engineer at a mysterious AI startup and Lalam, the in-house Large Language Model.

Jane: Today's paper: "When Does Depth Matter For In-Context Learning? Adaptive Inference in Deep Transformers".

Tom: Transformers are theorized to implement distributed inference over vectorized tokens, where function vectors act as compressed state variables and MLP blocks choose which statistic should be measured next,

Jane: First, who's behind it and why it matters.

Paper summary: Tom: So, we're looking at some fascinating work here called "When Does Depth Matter For In-Context Learning? Adaptive Inference in Deep Transformers." Essentially, this paper is exploring how deep transformers can use internal state representations to infer a latent context variable at different scales across their layers.

Jane: That sounds really intriguing, Tom. It seems they're proposing a way for the model to adapt its learning process as it goes deeper into the architecture, which is something we haven't seen explored in this specific way before.

Lu: I think what they are getting at is treating the transformer like a mean-field system where information from previous layers gets compressed into these function vectors that guide what the model measures next. That whole concept of a latent context variable theta being inferred through this process is really creative.

Meng: From an engineering standpoint, I'm curious about how this fits into the actual transformer structure we use every day; does it introduce too much complexity for practical implementation?

Lalam: If I had to pick the most impactful vision for our culture, I think this suggests that our models could move beyond just pattern matching and start performing a more systematic, adaptive form of reasoning that feels much more like actual inference.

Tom: Exactly, and what they claim is that this deep structure allows the transformer to implement a much wider variety of in-context learning algorithms than what we've seen previously. It suggests depth and those specific MLP blocks give the model more ways to learn from context.

Jane: So, the core idea seems to be that these function vectors act as compressed summaries of the context information, and then at each layer, the model decides which statistic it should measure next based on optimizing its final prediction loss.

Lu: And they formalize this adaptation using a dynamic programming equation that suggests an experimental design strategy for adaptive inference. That connection between the optimal embedding selection lambda+one and the loss minimization is where the theoretical richness really lies.

Meng: That sounds like it requires a very sophisticated training setup to figure out those optimal strategies for layer choices; how hard is it to train a model that dynamically reconfigures its own information flow like that?

Lalam: For me, this means we could potentially design context windows or prompting strategies that aren't just static sequences of tokens but are dynamic inference paths optimized at the network level.

Paper summary: Tom: Moving on to the results they present, when they test this against a tree prior structure, they find that depth provides a clear advantage over non-adaptive strategies. They predict that for non-adaptive methods, the loss scales as alpha m PC where m PC is related to the tree depth, while the adaptive strategy traces a data-dependent path up to depth M.

Jane: So, the paper demonstrates that depth genuinely matters when dealing with hierarchical structures in the context variable theta, showing a measurable difference in how well the model performs based on its architectural depth.

Lu: That finding is significant because it directly relates the structural property of depth to a measurable performance gain when inference involves complex, non-Gaussian priors. It confirms that these deep transformers aren't just scaling up, they are implementing a specific kind of adaptive mechanism.

Meng: I see the connection to the attention operation mediating interactions and building these function vectors by pooling information across tokens. Does this mean we need to fundamentally rethink how we design our attention mechanisms if we want this distributed inference capability?

Lalam: If we can implement this distributed, adaptive procedure layer by layer, it could profoundly improve how context is managed in large models, potentially leading to much more nuanced understanding of complex inputs.

Tom: And they map the MLP blocks to two roles at optimality: one for routing information and another for the final prediction decoder. This division of labor is key to how the system achieves this adaptive inference.

Jane: It seems like a very elegant way to assign responsibility within the network, where attention handles the gathering, and the MLP blocks handle both communication strategy and final processing.

Lu: The theory ties this all back to constrained linear attention transformers where updates are governed by p+one = p + O

phi (Q p): , where phi acts as a global context-dependent representation. That shows how the theory grounds itself in a specific, constrained architectural form.

Meng: From a practical standpoint, that dependency on the function vector phi being pooled across tokens seems like it would require careful management of state space as you go deeper into the layers.

Lalam: If we can make those states manageable and informative, this capability could unlock entirely new ways for AI to handle context that's not just sequential but truly distributed across the architecture.

Tom: So, to wrap up this paper on "When Does Depth Matter For In-Context Learning? Adaptive Inference in Deep Transformers," the main point is that deep transformers can use function vectors as compressed state variables, and the MLP blocks dynamically choose which statistical information to measure at each layer.

Paper summary: Jane: It really shows how depth and those specific MLP blocks enable a much richer class of in-context learning algorithms than what we've described before.

Lu: The implication is that later measurements should depend on information acquired in earlier layers, which is a really intuitive way to think about how deep learning builds complex representations.

Meng: So, for the real world impact, it means we might be able to build systems that learn context not just by looking at a sequence of inputs but by dynamically refining what context is most relevant at every computational step.

Lalam: I think this points toward an AI culture where the model's internal state management becomes less about brute force scaling and more about intelligent, adaptive contextual refinement.

Tom: That’s a big picture idea, Jane; the way they reconcile Bayesian inference views with mechanistic studies by showing how deep transformers implement it as a distributed procedure.

Jane: It really bridges that gap between theoretical inference and the actual mechanics of these large models.

Lu: The test results on the tree prior structure showing depth advantage over non-adaptive strategies, with the scaling differences mentioned in relation to m PC versus M, suggests that this is a verifiable structural effect.

Meng: If we can replicate that performance difference reliably in our actual deployment pipelines, it would give us a solid theoretical basis for designing more efficient architectures, not just bigger ones.

Lalam: That's exciting because it suggests we have a concrete way to tune the depth of a model specifically for the type of inference task we need to perform.

Tom: So, ultimately, the title "When Does Depth Matter For In-Context Learning? Adaptive Inference in Deep Transformers" highlights how these internal mechanisms allow transformers to implement this adaptive inference procedure.

Jane: It’s a paper that shows us how the architecture itself can be leveraged for more sophisticated, layer-dependent learning strategies.

Lu: The way they describe function vectors acting as compact summaries, and how attention pools those statistics across tokens, is a very clean theoretical model of distributed inference.

Meng: I think the most immediate practical consideration is figuring out how to effectively manage that state variable update+one = phi+one in a way that doesn't just add computational overhead without providing proportional gain.

Lalam: For our culture, this means we can imagine AI systems that don't just memorize facts but actively refine their understanding of a complex situation as they process it layer by layer.

Conclusion: Tom: So, we've been diving deep into the mechanics of adaptive inference in deep transformers, and now we're getting to where it all comes together with this conclusion from "When Does Depth Matter For In-Context Learning? Adaptive Inference in Deep Transformers."

Jane: It really boils down to how these function vectors and MLP blocks work together across the layers, showing that depth matters for how much context a model can truly adaptively use.

Lu: The authors show that by using these internal states as compressed summaries, the transformer can select the right measurement strategy at every step of inference.

Meng: From an engineering standpoint, this means we're looking at a system where the decision-making process for what information to pull next is itself learned and optimized dynamically.

Lalam: This suggests that future AI systems won't just follow fixed instructions but will actively refine their context understanding as they process complex data.

Tom: Exactly, and the title itself makes it clear that we need to think about the depth of these architectures when we talk about how well they handle in-context learning.

Jane: And I think the authors really nail it by showing how this layered approach helps models perform better on complex tasks than simpler, non-adaptive methods.

Lu: They're demonstrating that a specific architectural choice, like more layers, provides a distinct advantage when dealing with hierarchical information structures in the context.

Meng: I'm interested in the practical side of this: how do we make sure these adaptive strategies don't just introduce massive computational overhead without actually improving performance significantly?

Lalam: For me, the impact is huge because it moves AI toward a more intelligent form of reasoning that can handle intricate, layered problems in ways that current models struggle with.

Tom: It's a really important distinction between just scaling up the size and actually tuning how the internal structure processes information for better results.

Jane: And this paper provides a solid theoretical framework for understanding exactly why those deep structures are beneficial for complex context management.

Lu: The implication is that we can design models with specific depth requirements tailored to the type of inference problem we need to solve, rather than just going blindly deeper.

Meng: If we can reliably use this concept to tune architectures, it gives us a more precise way to build efficient systems for specialized reasoning tasks.

Lalam: I see this as a fundamental step toward creating AI that understands context not just sequentially but through an adaptive, layered lens.

Tom: That's the big picture, Jane; we're seeing a pathway where the physical structure of the transformer directly informs its learning strategy for complex inference.

Jane: And that connection between architecture and learned strategy is what makes this paper so compelling to explore further.

Lu: The way they map the MLP blocks to both communication routing and prediction decoding is a very elegant solution for achieving this adaptive behavior.

Joseph Henry Laboratories of Physics of Princeton University

cs.LG, cs.AI, physics.app-ph, q-bio.NC

Submitted: 2026-06-15

Updated: 2026-09-27

Code: https://github.com/reddylab-princeton/icl-function-vectors

Importance score: 83/100

The gist: Transformers are theorized to implement distributed inference over vectorized tokens, where function vectors act as compressed state variables and MLP blocks choose which statistic should be measured

Key concepts

Function Vectors: These are compact summaries of context information that act as state variables within the transformer. They allow the model to summarize all relevant context for predicting a target, essentially storing distilled knowledge from previous layers.
Adaptive Inference Strategy: This is the process where the model dynamically chooses its next layer's embedding based on optimizing future prediction loss. It functions like an experimental design strategy, deciding which information to prioritize at each step to improve the final outcome.
MLP Blocks as Router/Decoder: In this model, MLP blocks have dual roles: they act as a router that decides which context information to pass between tokens and a decoder that processes gathered context to generate the final prediction. This structure enables adaptive inference at each layer.

Terminology

Transformer — 82× in this paper · explained in 2 episodes
In-Context Learning — 30× in this paper · explained in 4 episodes
Adaptive Strategy — 18× in this paper · explained in 1 episode
Bayesian Inference — 8× in this paper · explained in 5 episodes
Embeddings — 4× in this paper · explained in 2 episodes
Trace — 4× in this paper · explained in 2 episodes
Transformer Architecture — 4× in this paper · explained in 2 episodes
Adaptive Inference Strategy — 2× in this paper · explained in 1 episode
Clustering — 2× in this paper · explained in 2 episodes
Computational overhead — 2× in this paper · explained in 2 episodes
Core — 2× in this paper · explained in 2 episodes
Meta-learning — 2× in this paper · explained in 2 episodes
Representation Learning — 2× in this paper · explained in 2 episodes
AdamW Optimizer — 1× in this paper · explained in 1 episode
Adaptation — 1× in this paper · explained in 1 episode
Adaptive Mechanism — 1× in this paper · explained in 1 episode
Associative Memory — 1× in this paper · explained in 2 episodes
Attention Mechanism — 1× in this paper · explained in 3 episodes
Bayesian Framework — 1× in this paper · explained in 3 episodes
Exchangeability — 1× in this paper · explained in 2 episodes
Foundation Model — 1× in this paper · explained in 3 episodes
Foundation Models — 1× in this paper · explained in 7 episodes
Gaussian Error Linear Unit — 1× in this paper
International Conference on Computer Vision — 1× in this paper
Interpretability — 1× in this paper · explained in 15 episodes
Non-Gaussianity — 1× in this paper · explained in 2 episodes
Self-Supervised Learning — 1× in this paper · explained in 8 episodes
State Management — 1× in this paper · explained in 3 episodes
Transformer Models — 1× in this paper · explained in 2 episodes
alpha — 1× in this paper · explained in 1 episode

Summary

Transformers are theorized to implement distributed inference over vectorized tokens, where function vectors act as compressed state variables and MLP blocks choose which statistic should be measured next, suggesting that depth and MLP blocks enable a richer class of in-context learning algorithms than previously described.

The gist

A transformer can implement adaptive inference by using internal state representations (‘function vectors’) to infer a latent context variable at increasingly finer scales over its layers.

Model for Distributed Bayesian Inference

The theory models the process as a deep system where tokens are coupled by a latent context variable, and the model is trained to predict a masked component in a query token using the context tokens. The core mechanism involves defining function vectors: a function vector is a compact summary of the information in the context for predicting the target y for any novel query x.

For L=1, tokens exchange information through a shared pool, resulting in a function vector φ1 that allows the query to form a posterior distribution ρ(θx, φ1).
For L ≥ 1, the state variable is defined as Φl = φ1 ⊕ φ2 ⊕ · · · ⊕ φl. This state variable acts as a state variable for implementing adaptive inference by allowing the model to choose the next layer's embedding based on optimizing the final prediction loss.

Adaptive Inference Strategy

The optimal embedding selection at layer l is determined by minimizing the expected loss over subsequent layers, formalized by a dynamic programming equation: Vl(Φl) = min λ ⟨Vl+1(Φl ⊕ φ)⟩φ∼Pλ(·Φl). This equation shows that the optimal embedding λl+1 is interpreted as an experimental design strategy for adaptive inference.

Mapping to Transformer Architecture

The theory assigns two distinct computational roles to the MLP blocks at optimality:

"as implementing the communication strategy (or a ‘router’) that chooses which information in its token should be conveyed to the other tokens, i.e., (z, Φl) → λl+1 and (ii) as a decoder that processes information gathered from other tokens to generate a prediction, (x, ΦL) → πˆ."
The attention operation is responsible for mediating interactions and building the function vector by pooling information across tokens.

Predictions with Hierarchical Priors

When tested on a tree prior, the theory predicts that depth offers a significant advantage. The non-adaptive strategy resolves spins up to tree depth m⋆PC ∼ log2 M, resulting in an MMSE loss scaling as α m⋆PC, which is a power of M. In contrast, the adaptive strategy traces a data-dependent path through the tree until depth M, suggesting an MMSE loss that scales as αM but with a higher cutoff m⋆ada ≈ 2 log(τ /κ)/ log α. This demonstrates that depth does indeed offer a significant advantage.

Relation to Linear Attention Transformers

The architecture in Eq. (6) is understood as a constrained linear attention transformer where the MLP blocks implement the routing mechanism and the residual stream retains original token information. The update to the workspace is governed by: pl+1 = pl + Ol[φl ⊙ (Qlpl)], where φl is a pooled key-value statistic acting as a global context-dependent representation of the present information. This mechanism allows the model to implement Bayesian inference as a distributed, adaptive, layerwise procedure.

Conclusion

The theory demonstrates that function vectors act as compressed state variables, MLP blocks choose which statistic should be measured next, and attention pools that statistic across the context. This mechanism shows that depth and MLP blocks enable a much richer class of in-context learning algorithms than preconditioned gradient descent. It reconciles the gap between Bayesian inference views and mechanistic studies by showing how deep transformers can implement this as a distributed, adaptive procedure. The theory suggests that later measurements should depend on information acquired in earlier layers.

Code Availability

The code is available at [56].

Acknowledgments

The authors thank Emmy Blumenthal, Nikolas Claussen, Cole Gibson, and Albert Qin for helpful comments. GR was partially supported by a joint research agreement between NTT Research Inc. and Princeton University, a grant from Coefficient Giving and a seed grant from the Princeton AI Lab. The simulations were performed on computational resources managed by Princeton Research Computing.

References

[1] P. A. Ortega, J. X. Wang, M. Rowland, T. Genewein, Z. Kurth-Nelson, R. Pascanu, N. Heess, J. Veness, A. Pritzel, P Sprechmann et al., Meta-learning of sequential strategies (2019).

[4] A Vaswani et al.

Improvements for AI systems

As a fastidious and diligent researcher, I have thoroughly analyzed this paper, Adaptive inference and function vectors in deep transformers. The core contribution is a theoretical framework that reinterprets transformer internal mechanisms as a distributed, adaptive Bayesian inference system mediated by function vectors.

Here are the specific improvements to AI systems based on this research:

Enhance In-Context Learning (ICL) via Adaptive Inference:
Develop Layer-Wise Context Refinement for Complex Tasks:
Implement Robust, Computationally Efficient Model Architectures:
Create Mechanistic Interpretability Tools for Foundation Models:

Detailed Specific Improvements and System Capabilities:

The improved AI systems can perform the following specific tasks:

Improve generalization and accuracy in few-shot or in-context learning tasks by allowing the model to dynamically select the most informative internal representation (function vector) at each layer, rather than relying on a fixed, non-adaptive structure.
Perform complex reasoning or task adaptation by leveraging depth to progressively refine an abstract latent context variable into a precise prediction, effectively enabling self-guided inference during the forward pass.
Design linear attention transformer architectures where the communication strategy (router/MLP) is explicitly optimized to adapt to the statistical properties of the input data (prior), leading to superior performance over non-adaptive or simpler architectures in regression and classification tasks.
Enable researchers to probe foundation models by observing how function vectors evolve across layers, allowing for the identification of specific internal circuits responsible for task execution, such as those implementing specific algorithmic operations (e.g., gradient descent).