Learning, Fast and Slow: Towards LLMs That Adapt Continually

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: Today's paper: "Learning, Fast and Slow: Towards LLMs That Adapt Continually".

Jane: The gist Large language models (LLMs) can be adapted continually by introducing a fast-slow learning framework that separates task-specific information into fast weights and persistent general behaviors into slow weights…

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So, we're looking at this paper today, titled "Learning, Fast and Slow: Towards LLMs That Adapt Continually." It basically tackles the problem that when you train an AI model to do a new task using reinforcement learning, those parameters get permanently changed to absorb that task-specific information.

Jane: That's the core issue we've been talking about—parameter updates force them to absorb things, which leads to forgetting what they knew before and losing their general ability to learn new things. So this paper proposes separating those weights into two parts: slow weights and fast weights.

Lu: Exactly. They treat the model parameters as "slow" weights that are expensive to update, but they treat the context or prompts as "fast" weights that you can change cheaply and frequently without permanently altering the main model structure.

Meng: So if I understand correctly, this means we keep the core brain of the AI stable and general, but we have a separate mechanism to handle all the quick task-specific tweaks? That sounds like it could be useful for deployment where we don't want constant retraining of massive models.

Lalam: It seems they're suggesting that instead of just updating everything, we can distribute the learning signal across both types of weights so the model stays flexible.

The paper's summary: Tom: The paper introduces a fast-slow learning framework called Fast-Slow Training or FST, which jointly optimizes those slow weights through reinforcement learning and fast weights through something called reflective prompt evolution. It’s like they have two different training loops running at the same time.

Jane: They set up a system where the slow parametric component, which is the model weights denoted by theta, represents that expensive part of the model that usually encodes long-lived behavior and persists across tasks.

Lu: And then there's this fast textual component represented by phi, which are prompts and context. These can be changed cheaply and frequently to capture task-level adaptation without permanently modifying the model itself.

Meng: So the training mechanism involves cycles of slow-weight updates where they use reinforcement learning, while simultaneously optimizing the fast weights using this reflective optimization process on a pool of prompts. That sounds like a lot of coordination for what is supposed to be efficient.

Lalam: It’s structured in interleaved updates where the slow loop updates theta from just the scalar reward, and then the fast loop updates phi using that rollout's full text, including thoughts and errors. This allows both components to co-evolve during training.

The paper's improvements: Tom: Now for what they claim is better. They show that this approach is more sample efficient than just training with slow weights alone across reasoning tasks, reaching performance with substantially fewer samples and consistently hitting a higher performance ceiling.

Jane: That's a big deal because it means you get better results without needing massive amounts of data to force the slow weights to learn everything they need. They reach RL’s peak accuracy with fewer training samples.

Lu: And they also found that this method induces smaller slow-weight displacement, which means the model stays much closer to its original base policy, with up to seventy percent lower KL divergence compared to parameter-only trained models at the same reward level.

Meng: Smaller displacement sounds good for stability on production. Less drift away from the general knowledge base is definitely something you want when you're trying to maintain a stable model in use.

Lalam: The paper also shows that this fast-slow training preserves plasticity, meaning it adapts more effectively to subsequent tasks than models trained with only parameters, which prevents catastrophic forgetting.

Conclusion: Tom: So to wrap up on "Learning, Fast and Slow: Towards LLMs That Adapt Continually," FST suggests that post-training should be viewed as optimizing these two different adaptation channels together to get rapid task-specific gains while keeping the model general.

Jane: It essentially gives us a path toward methods that are more data efficient, less destructive, and better suited for continual learning because it lets the fast textual weights handle the immediate changes while the slow parametric weights maintain generality.

Lu: The mechanism is interesting because they show that the fast channel can extract task structure from just a handful of rollouts and inject it immediately through phi without needing many slow weight updates to start moving signal.

Meng: From an engineering standpoint, I'm interested in how this distributes the adaptation. They found that sometimes the fast weights dominate on some tasks, like HoVer-hard and CodeIO, but the slow weights carry most of the gain on math problems like Polaris.

Lalam: Overall, Fast-Slow Training is a way to get rapid task-specific improvements while preserving what we already have in terms of generality and plasticity. It's a way to handle adaptation that I think makes a lot more sense for real-world use cases.