Language Models Struggle to Use Representations Learned In-Context

summary

Video file (mp4)

The gist

The gist Open-weights language models struggle to deploy in-context representations learned from context to complete simple downstream tasks like next-token prediction and adaptive world modeling,

In short

The study tested if large language models (LLMs) can use representations learned from context to solve simple tasks like predicting the next word or modeling an adaptive world. Findings show that these in-context representations are largely inert, meaning they cannot be flexibly deployed to perform downstream tasks, even when they encode new information.

Key concepts

In-Context Representation Learning
This is the process where a model learns useful patterns directly from the text provided in a prompt without any formal training on that specific task. The researchers tested if these learned patterns could be reliably used later for different goals, like predicting the next word or understanding a new world.
Adaptive World Modeling (AWM)
This is a novel task designed to test if an LLM can use in-context semantics to solve problems that require understanding and manipulating a dynamic environment. The models were shown to struggle when trying to apply novel, context-defined rules or topologies they learned from the prompt.
Inert Representations
This refers to representations (the internal data structures the model creates from context) that are present in the model's hidden states but cannot be effectively used by the model to solve a specific problem. The study found that even when these representations contain new meanings, they remain largely unused for tasks like next-token prediction.
Flexible Deployment
This is the ability of a model to take information learned in one context (like a prompt) and apply it reliably to solve a completely different, subsequent task. The paper concludes that current LLMs struggle with this flexibility, indicating they need better training methods for such adaptability.

Terminology used across episodes

This episode discusses

The paper

Language Models Struggle to Use Representations Learned In-Context · Read on arXiv

Michael A. Lepori, Tal Linzen, Ann Yuan, Katja Filippova

Brown University · Google Research · New York University Google DeepMind

Though large language models (LLMs) have enabled great success across a wide variety of tasks, they still appear to fall short of one of the loftier goals of artificial intelligence research: creating an artificial system that can adapt its behavior to radically new contexts upon deployment. One important step towards this goal is to create systems that can induce rich representations of data that are seen in-context, and then flexibly deploy these representations to accomplish goals. Recently, Park et al. (2024) demonstrated that current LLMs are indeed capable of inducing such representation from context (i.e., in-context representation learning). The present study investigates whether LLMs can use these representations to complete simple downstream tasks. We first assess whether open-weights LLMs can use in-context representations for next-token prediction, and then probe models using a novel task, adaptive world modeling. In both tasks, we find evidence that open-weights LLMs struggle to deploy representations of novel semantics that are defined in-context, even if they encode these semantics in their latent representations. Furthermore, we assess closed-source, state-of-the-art reasoning models on the adaptive world modeling task, demonstrating that even the most performant LLMs cannot reliably leverage novel patterns presented in-context. Overall, this work seeks to inspire novel methods for encouraging models to not only encode information presented in-context, but to do so in a manner that supports flexible deployment of this information.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Language Models Struggle to Use Representations Learned In-Context".

Jane: The gist Open-weights language models struggle to deploy in-context representations learned from context to complete simple downstream tasks like next-token prediction and adaptive world modeling,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So, we’re looking at this paper called Language Models Struggle to Use Representations Learned In-Context. The main idea here is that while language models can learn representations from context—like seeing a pattern in a prompt—they still struggle to use those learned representations when they have to do actual tasks later on.

Jane: Right, so the big question is, if an AI learns a new way to understand something just because you showed it in the prompt, can it actually use that understanding when you ask it something totally different? That’s what this paper is really digging into.

Lu: Exactly. The authors are testing whether these in-context representations are flexible enough to handle different tasks after they've been learned from the context, which is a big step toward building systems that can adapt to new situations upon deployment.

Meng: From my side, I’m thinking about deployment—if a model learns something new in context but then fails on a simple downstream task like predicting the next word, it just means it’s not robust enough for real-world use where things get messy.

Lalam: I see what they're saying is that even when the latent representations encode completely novel semantics from the context, they often end up being inert when a different instruction comes along.

Tom: That’s a strong claim. They first test this on next-token prediction and then move into this more complex idea called adaptive world modeling to see if those same representations can be deployed flexibly.

Jane: So, the paper sets up two main tests: one for simple prediction and another for a task that requires building an internal model of how things connect in a space. This is where they really try to push the limits of in-context learning.

Lu: They use prompts with two parts—one part defines the context semantics, like specific words or structures, and the other part asks for something based on those semantics, and they find that even when representations reflect novel topology information, it’s hard to actually use that topology later on >

Meng: And they found that open-weights models struggle with this deployment. That means if you take a model like a smaller version of something publicly available, it doesn't automatically get the benefit of the context learning for practical applications >

Lalam: They show evidence that in both tasks, these in-context representations are largely inert, even though they seem to encode the new semantics perfectly within their latent states >

Tom: That’s a key finding because it challenges the idea that simply encoding information in a hidden state is enough for flexible deployment. The abstract specifically mentions this difficulty with open-weights LMs trying to complete simple downstream tasks like next-token prediction and adaptive world modeling >

Jane: It suggests that the representations learned in context aren't just stored; they aren't easily accessible or usable when the model needs to apply them to a new problem, which is a problem for building truly adaptable AI systems >

Paper summary: Lu: The authors point out that even when they look at closed-source, state-of-the-art reasoning models on the adaptive world modeling task, they still cannot reliably use the novel patterns presented in context >

Meng: So even those top performers aren't automatically solving this deployment hurdle. That tells me we need to think about how to make these learned representations more usable for actual agents >

Lalam: The overall conclusion of this paper is that in-context semantics encoded in hidden representations are often inert, or unable to be used for either next-token prediction or adaptive world modeling >

Tom: So the title itself really sums up the issue: Language Models Struggle to Use Representations Learned In-Context. It’s about the gap between learning something new and actually being able to use it when things get complicated >

Jane: When we think about this, it means that just seeing a pattern in a prompt isn't enough for an AI to gain a flexible, useful understanding that it can carry forward into novel situations >

Lu: The limitation they highlight is that in-context semantics encoded in hidden representations may not be flexibly deployable, but the paper doesn't offer a solution directly; instead, they suggest targeted training regimens or mechanistic interventions could make these semantics available to earlier model layers >

Meng: That sounds like heavy training work. I’m wondering how much effort it would take to retrofit existing models so that their in-context learning isn't just theoretical but practical for deployment >

Lalam: Also, they mention that it’s difficult to understand the relationship between the externalized chains-of-thought produced by reasoning models and their hidden token representations, which adds another layer of complexity when we try to make these things work >

Tom: So if we look at what this means for AI development right now, it suggests that simply scaling up context length or prompt complexity isn't the answer; we need a fundamental change in how those learned internal representations are structured >

Jane: It pushes the research toward architectures and training strategies that support flexible deployment of representations learned in-context, which is what they hope to inspire for building truly adaptable artificial agents >

Lu: The paper builds on previous work that looked at whether world models afford other manipulations, suggesting that the current view isn't quite complete >

Meng: For production, this means we can’t just rely on a model "remembering" context; we need mechanisms to actively manipulate those learned structures for different purposes >

Lalam: The core finding remains that representations learned in context are often inert when faced with novel tasks, which is a cautionary note for how we view the power of in-context learning >

Tom: So, this paper shows us that while in-context learning is impressive, we still have a long way to go before language models can reliably use those learned concepts for complex reasoning outside of the exact context they were given >

Conclusion: Tom: So, we’ve seen how these language models learn stuff just by looking at context, but now we're talking about whether they can actually use that stuff later on in different situations.

Jane: Exactly. This paper is titled "Language Models Struggle to Use Representations Learned In-Context." The authors are showing that this isn't just a theoretical problem; it happens when you ask the model to do real tasks after it’s learned from a prompt.

Lu: What they're doing is testing if these in-context representations are flexible enough for things like next-token prediction and adaptive world modeling. They’re seeing if the patterns they learn in context actually stick around when the task changes.

Meng: And what they found is pretty definitive—open-weights models struggle to deploy those learned semantics, even when they've encoded the right information in their hidden states for things like grid topology. That’s a problem for production deployment.

Lalam: I agree with Meng. I see it too; the representations are often inert when faced with novel examples, whether that’s a simple next-token prediction or trying to model an adaptive world. They aren't reliably using the context they saw before.

Tom: That means we can’t just assume that if a model learns something new in a prompt, it automatically gains the ability to use it for anything else. It stops being like a flexible system and more like a static data store.

Jane: Right. The authors are suggesting that this gap between learning and using is real, even for models that are already pretty high performing. They're pointing toward needing training regimens or architectural changes to make these learned representations actually available for deployment in novel contexts.

Lu: And they highlight that we still don’t fully understand the connection between those externalized reasoning chains and the hidden token representations, which makes it hard to fix this right now. It points toward a bigger need for new ways of structuring these internal models.

Meng: So, if this is true, then simply increasing context length isn't the answer; we need something that lets the model manipulate those learned concepts rather than just storing them passively. That’s what I’m thinking for shipping models at scale.

Lalam: I’m quite sure about that conclusion. The core finding is that in-context semantics are often inert, and they aren't useful for either next-token prediction or adaptive world modeling without some kind of mechanism to activate them properly.

Tom: This paper isn't just about a model failing a test; it’s a warning about the current way we think models should learn and adapt. It suggests that in-context learning is impressive, but deployment requires more than just memorizing patterns.

Jane: It really does. And this leads us nicely into what might actually fix this, like those targeted training regimens the authors mention next.

More episodes

← Home