2608.07169-Agent Memory Distillation: Empowering Small LLM Agents with Hierarchical Teacher Memory

Introduction to the show: ident: Paper Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "Agent Memory Distillation: Empowering Small LLM Agents with Hierarchical Teacher Memory".

Jane: The paper was written by Taeil Kim, Kangsan Kim and Sung Ju Hwang from KAIST and DeepAuto.ai.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Paper summary: Tom: Welcome back to the channel, everyone. Today we're unpacking a paper that takes a well-known idea in eye—getting a big model to teach a small model—and applies it to something called agent memory. The authors are from KAIST and DeepAuto.ai, and the core question is pretty simple.

Jane: And the answer they found is that you can't just hand a small model a big model's memory and expect it to work. The small model often can't understand or use what it's given. So they built a system that organizes that transferred memory into three layers, matching how the small model actually thinks.

Lu: Right, so instead of one big blob of advice, you get high-level strategy, concrete examples of how to do each sub-step, and then specific tips for individual tools. That hierarchy is the key move here.

Meng: And it's training-free, which is a big deal. You don't have to fine-tune the small model at all. You just inject the right memory at the right time during a task, and the small model gets better almost immediately.

Tom: We're looking at real numbers here, too. On the AppWorld benchmark, some of these 4-billion-parameter models go from around fifteen percent accuracy up to nearly fifty percent. That's a massive jump for doing zero training.

Jane: And on BFCL, a function-calling benchmark, the small students actually end up outperforming the teacher that taught them. That's not something you see every day in distillation papers.

Lalam: What I find exciting is the broader implication. We keep building bigger and bigger models, but this suggests a cheaper path: keep the big model in the lab, distill its experience into something small enough to run on a phone or a laptop, and get most of the capability.

Tom: So the plan for this episode is to walk through the paper page by page, starting from the problem they identify, then how the memory system is built, and finally all the experiments that back it up.

Jane: And we've got Lu, Meng, and Lalam joining us today to help break it all down. Let's start at page one, where they set up the problem.

Page 1: Tom: So page one opens with a pretty relatable struggle. Small language models trying to build their own memory from experience just don't have enough successes to learn from. If a model fails at most tasks, its memory bank is mostly failures.

Lu: That's the self-evolution problem. Big models can run task after task, succeed often, and build up a rich set of successful examples. A small model might complete only one in ten tasks, so even if it remembers everything, there's almost nothing useful to remember.

Jane: The paper shows a figure that makes this concrete. The student's own memory gives it almost no lift because it's starved for successful trajectories. Then they show what happens if you naively hand over the teacher's memory.

Meng: And that's the second surprise. Just giving the small model the teacher's high-quality memory barely helps either. The example they give is a teacher memory that says "log in before playing music," but the small model doesn't know how to log in in the first place.

Tom: So you're stuck between two failures. Too little experience on one side, and a capability gap on the other. The teacher's advice assumes knowledge the student doesn't have.

Lu: Exactly. This is actually a known problem in knowledge distillation, where a huge teacher can be so far ahead of a small student that the student can't follow the teacher's reasoning. The paper is bringing that same insight into the agent memory world.

Jane: And their solution is to break the teacher's knowledge into three memory types at different levels of abstraction. Workflow memory for the overall plan, subtask memory for concrete execution examples, and function memory for tool-level details.

Meng: That way the student gets the big picture, but also the specific step-by-step patterns it can actually copy. It's not just advice anymore—it's a template it can fill in.

Tom: The teaser numbers on this page are already strong. One student model jumps from about fifteen percent to over forty-nine percent on AppWorld, and another model actually surpasses the teacher on two of the three benchmarks.

Lalam: This is the exciting part for me. If this works, small models don't need to reinvent the wheel. They just need the right kind of scaffolding from a bigger model, and they can punch way above their weight class.

Jane: So the question becomes, how do they actually build these three memory types from teacher trajectories? That's what page two starts to lay out.

Page 2: Jane: Page two dives into related work, and it's really setting up why this paper is different from what came before. The existing memory systems like Reflexion, ExpeL, and MemP were mostly tested on large proprietary models, not small ones.

Lu: Right, and that's a crucial gap. Those systems work when the model is already smart enough to use its own memories well. The paper's point is that small models are exactly the ones that need memory the most, but they're the ones for whom these systems fail.

Meng: There's also a line about how small models have weaker in-context learning and instruction following. So even if you show them a memory entry, they might misapply it or just get confused by the extra text in their context window.

Tom: That explains why the naive transfer fails. It's not that the teacher's memory is bad, it's that the student can't digest it. And this connects to a classic result in distillation about the teacher-student gap.

Lu: The paper cites that work explicitly. If the teacher is too strong relative to the student, the student's performance actually degrades. The knowledge just doesn't transfer cleanly.

Jane: And then they mention the few previous attempts at agent distillation. Some use training, which is expensive and doesn't explore memory transfer. Others reuse teacher-generated plans but don't let the student draw on the teacher's memory directly.

Meng: So the paper is positioning itself as the first systematic study of teacher-to-student memory transfer, with a design that explicitly accounts for the capability gap.

Lalam: I like that they're not just saying "bigger teacher, better student." They're saying the format of the knowledge matters as much as the quality. High-level strategy in prose, concrete examples in code, and tool-specific fixes at the moment of failure.

Tom: And that sets up the core contribution. The next pages will explain exactly how they construct and inject this hierarchical memory. Let's get into the method.

Page 3: Tom: Page three formalizes the whole setup. We've got an agent solving multi-turn tool-use tasks, calling functions from a predefined set, and getting observations back. There's a teacher model and a student model, and the goal is to maximize the student's success using teacher-generated memory.

Lu: They define the memory store as three banks: workflow, subtask, and function. And the key detail is that all of it is built from successful teacher trajectories only. No failures pollute the memory.

Jane: That's a deliberate choice. Since the teacher succeeds far more often than the student, you can afford to be picky and only learn from the good runs.

Meng: Then they explain workflow memory. For each successful trajectory, the teacher writes a natural language insight that captures the overall strategy. But they abstract away concrete values—things like specific IDs, emails, and file paths get replaced with typed placeholders.

Tom: So instead of "log in with username john@example.com," it's "log in with <EMAIL>." That keeps the memory general enough to apply to new tasks with different details.

Lu: And each workflow entry is paired with a natural language query describing the task. When the student gets a new task, it uses that task instruction to retrieve the most similar workflow memory.

Jane: Then there's subtask memory, which is where the granularity gets interesting. The teacher takes each successful trajectory and splits it into coherent segments—like "authenticate to Venmo" or "sum transactions."

Meng: Each segment gets a label, a short description, and the concrete execution example, meaning the actual tool calls with their observations. These are encoded into vectors and stored in the subtask memory bank.

Tom: So you're building a library of reusable sub-procedures. The idea being that a new task might not match a whole old task, but it will probably share a subtask like "log in" or "paginate through results."

Lu: And the last piece on this page is function memory, which captures individual tool invocations. Each function name gets its own records, storing a concrete example of how the teacher called it, plus the API docs when available.

Jane: That's the reactive layer. When the student gets an error from a tool call, it can look up that specific function and see exactly how the teacher did it correctly.

Page 4: Jane: Page four gets into the mechanics of how these memories are actually injected into the student's context. There are two modes: proactive and reactive.

Tom: Proactive means before the student starts working on a task. The workflow memory is retrieved using the task instruction as the query, and the top match is prepended to the system prompt. So the student gets the high-level plan up front.

Lu: And subtask memory is also proactive, but it's smarter. The student first decomposes the task into an ordered list of up to six subtask labels. Each label is then used to retrieve the best matching segment from the subtask memory bank.

Meng: Deduplication matters here. If two subtask labels retrieve the same segment, you don't want to inject it twice and waste context. So they make sure each segment appears only once.

Tom: Function memory is the reactive one. It only kicks in when a tool call returns an error. The failing function's name is used to look up candidate records, and the top ones are appended to the error message as a hint.

Jane: That's clever because you're not bloating the context during normal execution. The student only gets the extra guidance at the moment it's actually stuck.

Lu: And it makes sense from a small model's perspective. If you pre-load too much text, the model might lose track of the actual task. By keeping the context lean until needed, you reduce the cognitive load.

Meng: The retrieval is all embedding-based cosine similarity, with a threshold to discard low-confidence matches. And in the main experiments, they use top-1 for each memory type—one workflow entry, one subtask segment per decomposed label, and one function record per failing call.

Tom: So the entire memory system is designed around the idea of giving the student exactly what it needs, at the right granularity, at the right time. No more, no less.

Jane: And with that, we move to page five, where they lay out the experimental setup and benchmarks.

Page 5: Tom: Page five gets into the experimental setup, and this is where the paper's claims get tested. They use GPT-5-mini as the teacher, and four different student models ranging from 4B to 8B parameters.

Jane: The students are Qwen3-4B, Qwen3-8B, Gemma4-E4B, and Llama3 point 1-8B. And they evaluate on three benchmarks: AppWorld, BFCL V3, and ToolSandbox.

Lu: AppWorld is the heavy one—multi-app tasks involving email, messaging, and payment services through Python API calls. Success is measured by database-state tests, so it's a strict pass/fail.

Meng: BFCL is focused on function calling, where the agent has to invoke the right functions with accurate arguments across multiple turns. It's about precision, not just getting the job done.

Tom: And ToolSandbox adds conversational complexity. The tools depend on shared world state, and there's an LLM-simulated user driving the dialogue. It's a tougher, more realistic setting.

Jane: Interesting detail—they repeated each experiment twice and reported the average, which is a solid robustness check. And the baselines include three existing memory frameworks adapted to the teacher-to-student transfer setting.

Lu: ReasoningBank, MemP, and SASM. These are meant to represent the current state of the art in agent memory, and none of them were designed for cross-model transfer.

Meng: Also on page five they set the retrieval count k=1 for all memory types, which I think is going to matter later. And memory entries are encoded using OpenAI's text-embedding-3-small model.

Tom: I like that they're keeping the setup simple. One teacher, a fixed set of students, three benchmarks, and a clean comparison against existing memory methods.

Jane: Now we're ready for the main results, which we saw teasers of earlier. Page six brings the full table.

Page 6: Jane: Page six has the main results table, and it's quite a picture. Across all four student models and all three benchmarks, AMD beats the zero-shot baseline and all three memory baselines.

Tom: The gains are dramatic on AppWorld—an average of 27 point 2 percentage points. Qwen3-4B goes from 14 point 88 to 49 point 40 percent, and Gemma4-E4B goes from 24 point 4 to 54 point 17 percent.

Lu: What's notable is that the baselines are unstable. ReasoningBank actually hurts Qwen3-4B on AppWorld, dropping it from 14 point 88 to 10 point 71 percent. MemP and SASM show similar inconsistency across models.

Meng: That supports their argument that flat or badly structured teacher memory can introduce noise that small models can't handle. It's not just about having good memories—it's about presenting them in the right form.

Tom: And then there's the headline result that some students match or even surpass the teacher. Gemma4-E4B reaches 54 point 17 percent on AppWorld, while the teacher GPT-5-mini only gets 50 percent.

Jane: Same on BFCL. Three of the four students surpass the teacher's 36 point 5 percent. Qwen3-8B gets 45 point 5 percent, and Gemma4-E4B gets 46 percent. That's a huge margin.

Lu: The paper's explanation is that the student isn't just copying the teacher's trajectories. It's re-instantiating the distilled decision-making patterns under its own inductive biases. So the student can end up better than the source.

Meng: There's also a nice interaction efficiency result. Students without memory use many more turns—Qwen3-4B uses about 24 turns on AppWorld, while the teacher uses 10. With AMD, the student drops to about 15 turns.

Tom: So the memory is making the student not only more accurate, but also more efficient. It's learning from the teacher not just what to do, but how to do it with fewer steps.

Jane: That's a compelling combination. Higher accuracy and fewer wasted actions. Next we should look at the ablations to understand which memory type is doing the heavy lifting.

Page 7: Tom: The ablations on page seven break down how much each memory type contributes. They start with workflow memory alone, then add function or subtask, and finally combine all three.

Jane: And the result is pretty clear: subtask memory gives the biggest boost. On AppWorld, adding subtask memory on top of workflow takes Qwen3-4B from 22 percent up to 47 percent. That's a 25-point jump.

Lu: The intuition there is that high-level plans tell you what to do, but subtask segments show you how to actually do each step. For a small model, having concrete code to follow is far more actionable than abstract prose.

Meng: Function memory adds smaller gains on top, and in one case—Llama3 point 1-8B on AppWorld—it actually hurts slightly. The paper connects that to the model's weaker instruction-following capacity. More context during error recovery can push it off track.

Tom: And the "student memory" variant confirms the core premise. If you build the same memories from the student's own trajectories instead of the teacher's, you get results close to zero-shot. The student's experience is just too sparse and unreliable.

Jane: Page seven also looks at the teacher's effect. For the stronger Qwen3-8B student, teacher accuracy predicts student performance. The best teacher, GPT-5 point 5, gives the best student result. But for the weaker Qwen3-4B, that ordering breaks down.

Lu: That's a fascinating wrinkle. GPT-5-mini, which has only 50 percent teacher accuracy, transfers better to Qwen3-4B than DeepSeek V4 Pro with 81 point 55 percent accuracy. So it's not just about teacher strength—it's about compatibility.

Meng: And the student size analysis shows gains peak at 4B. A 1 point 7B model is too weak to use the memories effectively, while 8B and 14B models are strong enough to approach the teacher's own level.

Tom: So 4B is the sweet spot: capable enough to leverage the memory, but with enough room to improve. That tells you something about where this technique would be most useful in the real world.

Jane: Exactly. If you're deploying a model that's already quite capable, memory distillation gives you less headroom. It's the mid-sized models that get the biggest bang.

Page 8: Tom: Page eight has two more analyses that are really interesting. First, they look at how many memories to retrieve. And the finding is that k=1 is already optimal.

Jane: That's surprising. You'd think more memories would help, but accuracy drops as you increase the retrieval count. For subtask memory, Qwen3-4B falls from 49 point 4 percent at k=1 down to 33 point 34 percent at k=5.

Lu: The likely reason is context overload. Small models have limited capacity to handle extra text. Lower-ranked memories are more likely to be irrelevant, and injecting them distracts the model from the actual task.

Meng: So the design principle is precision over breadth. Give the student only the single best example, not a pile of loosely related ones.

Tom: Then they ablate the memory representation itself. Workflow memory works best as natural language text, while subtask and function memories work best as code. Replacing all three with text only drops performance to 26 point 19 percent on AppWorld.

Jane: That makes sense. A high-level strategy is naturally prose—"authenticate first, then paginate through results." But a concrete API call pattern needs to be shown in actual code for the student to follow reliably.

Lu: The qualitative case studies on this page really drive the point home. There's a Venmo task where the student without memory processes all-time payment requests instead of just this month's. Workflow memory fixes the date filter.

Meng: Then the date-parsing code triggers a type error that the student can't fix on its own. Subtask memory provides the working parse pattern.

Tom: And finally, the balance retrieval fails because the student reads the wrong dictionary key. Function memory supplies the correct multi-key parsing pattern, and the task completes.

Jane: It's a beautiful demonstration of the cascade. Each memory type fixes a different layer of the problem, and you need all three to succeed on the full task.

Lalam: That case study is the clearest explanation of why this design works. It's not one big memory—it's three small, targeted memories that cover planning, execution, and error recovery.

Jane: And the paper also includes robustness checks showing the gains hold even when memory is built from a completely different set of tasks than the evaluation tasks. So it's not just memorizing the test set.

Conclusion: Tom: So we've reached the end of the paper, and it's time to wrap up what we've learned. Agent Memory Distillation is a training-free way to transfer a teacher's experience to a small student agent, and it works by splitting that experience into three complementary memory types.

Jane: The key insight is that raw memory transfer doesn't work—the capability gap between teacher and student gets in the way. But by organizing the knowledge hierarchically, into workflow plans, subtask examples, and function-level fixes, the student can actually absorb and use it.

Lu: The numbers back it up. Consistent gains across three benchmarks, with the biggest wins on AppWorld, and several students matching or exceeding the teacher's own accuracy. And it works across four different student models.

Meng: The ablation story is also clean. Subtask memory is the most important piece, but you need all three for the best results. And keeping retrieval to top-1 is crucial for small models that can't handle context overload.

Lalam: The broader message is hopeful. We don't necessarily need bigger and bigger models for every task. If we can distill the experience of a large model into a small one without any training, that opens up cheap, deployable agents on modest hardware.

Tom: Of course, there are limitations they acknowledge honestly. The benchmarks are all text-based tool use. They haven't tested multimodal environments or open-ended coding, where the action space is much less structured.

Jane: And the memory is frozen after construction. It can't adapt to distribution shifts or incorporate the student's own test-time successes and failures. That's a clear direction for future work.

Tom: Alright, that was a rich discussion. We covered the problem, the method, the experiments, and the limitations. A solid piece of work from the KAIST team.

Jane: Thanks to Lu, Meng, and Lalam for joining us. And thanks to everyone listening. We'll be back with the next paper soon.