LLM Agents Factory: Retrieval of Domain-Specific LLM Agents

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "LLM Agents Factory: Retrieval of Domain-Specific LLM Agents".

Jane: The paper was written by Vitalii Belov, Artyom Sosedka, Andrey Sakhovskiy, Elizaveta Kovtun, Artyom Boyarskikh et al. from Sber AI and Moscow Institute of Physics and Technology and National University of Science and Technology MISIS and Skolkovo Institute of Science and Technology and Artificial Intelligence Research Institute.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Title: Tom: Welcome back to the show, everyone. Today we're digging into a paper that's been making the rounds — it's called "LLM Agents Factory: Retrieval of Domain-Specific LLM Agents." And Jane, I have to say, the title alone got me excited because it promises to solve a problem I've been complaining about for months.

Jane: Oh, tell me about it, Tom. The problem here is that when you ask a large language model to act as an agent — you know, to plan, to use tools, to solve a task — the standard approach is to generate the agent's "persona" on the fly. You ask the model, "Hey, what kind of expert should handle this?" and it makes one up in real time.

Tom: And that's expensive, right? Every single user request triggers a whole agent-creation process. The authors of this paper point out that this dynamic generation is slow, it's unstable — the same request might give you a completely different agent each time — and it burns tokens like crazy.

Jane: Exactly. So what they did is they flipped the whole thing. Instead of generating an agent from scratch every time, they built a giant library of pre-made agents — over twenty thousand of them — and then they just retrieve the right one. Like picking a tool off a shelf instead of forging a new hammer every time you need to hang a picture.

Tom: That's a great analogy, Jane. And the shelf here is grounded in Wikipedia categories. They took six hundred ninety-one high-level Wikipedia domains, crossed them with forty different roles like planner, verifier, tutor, and generated profiles for each combination. So you get a "tutor for mathematics" or a "verifier for history" — all pre-made, all validated.

Jane: And the retrieval part is clever too. They use a sentence embedding model to match your query to the best agent profile. It's lightweight, it's fast, and it's deterministic — same query, same agent, every time.

Tom: Which is huge for industry, by the way. If you're building a product, you need reproducibility. You can't have your AI assistant acting like a different personality every time a user asks the same question.

Jane: Right. And the authors show that this retrieval-based approach actually beats the dynamic generation baseline on accuracy while using way fewer tokens. We're talking about a threefold reduction in token consumption and faster inference.

Tom: I love that they're not just saying "retrieval is good" — they're showing it with numbers on MMLU, BIG-bench, and BIG-bench Hard. And they even distilled the whole thing into a smaller model that can generate agents directly, which is another path entirely.

Jane: So the big idea here is: treat agent construction as a search problem, not a generation problem. And that's a shift that could really change how we build AI systems in production.

Tom: And I can't wait to dig into how they built that agent base — the filtration pipeline, the deduplication, all of it. That's coming up next.

Summary: Jane: So we're back with "LLM Agents Factory: Retrieval of Domain-Specific LLM Agents," and Tom, we teased the big idea — retrieval over generation. But let's get into the actual construction pipeline, because that's where the magic happens.

Tom: Yeah, and it's a proper engineering effort. They start with a grid of six hundred ninety-one Wikipedia domains times forty roles. For each cell, they prompt a big teacher model — GPT-OSS 120B — to generate up to five different agent profiles. Each profile has a persona, a description, and a list of tools.

Jane: And they don't just accept everything the teacher spits out. There's a three-stage filtration pipeline. First, they check syntax — the profile has to be valid JSON, and the tools have to come from a whitelist of ten executable functions.

Tom: Right, because there's no point having an agent that claims it can call a tool that doesn't exist. Then they do semantic deduplication — they embed all the profiles and use FAISS to find near-duplicates, merging them with a union-find algorithm. That removes about eighteen percent of the candidates.

Jane: And they also add a few hand-crafted "meta agents" — generic routers and triage agents — to catch queries that fall outside the domain grid. It's a safety net for out-of-domain requests.

Tom: The result is a curated base of over twenty thousand agents, each one grounded in a real Wikipedia category. That grounding is important because it means the agent's domain knowledge is tied to something structured and human-curated, not just whatever the model hallucinated.

Jane: And then the retrieval part — they serialize each agent into a flat text representation, embed it with a Sentence-BERT model, and index it with FAISS. When a query comes in, they embed it the same way and pull the top-K most similar agents.

Lu: Can I jump in here? I think the grounding in Wikipedia categories is actually the most underrated part of this paper. Most agent frameworks just let the model invent a persona out of thin air. Here, every agent is anchored to an ontological cell — a real category in human knowledge. That makes the whole system interpretable in a way that dynamic generation just isn't.

Meng: And from a practical standpoint, that grounding also means you can audit the system. If something goes wrong, you can look at which agent was retrieved and why. You can't do that if the agent was generated by a stochastic process at runtime.

Tom: Exactly. And the authors even show that retrieval-based agents beat the non-agent baseline on MMLU and BIG-bench — so it's not just cheaper, it's actually better.

Jane: But there's a trade-off, and we'll get into that when we talk about the distillation approach and the multi-agent setups. That's next.

Improvements: Tom: So we've covered the agent base and the retrieval mechanism. Now let's talk about what this paper actually improves over the state of the art. Jane, what's the biggest win here?

Jane: The biggest win is efficiency without sacrificing accuracy. They compare against AutoGen, which is a popular framework that generates agents dynamically at runtime. And on all three benchmarks — MMLU, BIG-bench, and BBH — the retrieval-based approach, which they call ALR, matches or beats AutoGen's accuracy while using about three times fewer tokens and running faster.

Meng: And that's not just a nice-to-have. In production, token cost and latency are the two things that actually determine whether you can deploy a system. If you're serving millions of requests a day, a threefold reduction in tokens is a massive cost saving.

Lu: But I want to push back on something. The paper also explores distillation — they call it ALR-Distill. They fine-tune a smaller model, Qwen3-4B, to generate agent profiles directly from queries, using synthetic training data created by the teacher model. And that approach performs on par with retrieval while doubling token consumption.

Tom: So the trade-off is interesting. Retrieval is cheaper and faster, but distillation gives you a self-contained model that doesn't need a retrieval index at runtime. For some deployments, that might be worth the extra tokens.

Jane: And then there's the multi-agent setups. They tested a few configurations where you retrieve multiple agents and let the solver pick, or where you combine retrieval with a generated agent. On the harder BBH benchmark, the multi-agent approaches actually win — they get up to sixty-nine point six percent accuracy compared to sixty-eight point five percent for single-agent retrieval.

Meng: But the multi-agent setups cost more — more tokens, more latency. So the paper is really showing a spectrum of options: cheap and fast single-agent retrieval, more expensive but more accurate multi-agent, and the distillation middle ground.

Lu: And that's the real contribution, I think. It's not just "retrieval beats generation." It's a systematic study of the trade-offs, with a concrete framework you can actually use. The authors release the code and the agent base on Hugging Face, so anyone can reproduce their results or build on top of it.

Tom: And the improvements go beyond just the benchmarks. The paper argues that retrieval-based construction improves interpretability — you can see exactly which agent was chosen and why. It improves reproducibility — same query, same agent, every time. And it reduces the token overhead from those long meta-prompts that dynamic generation requires.

Jane: So the improvements are threefold: cost, control, and consistency. And those are exactly the things that matter when you're building real systems, not just running experiments.

Meng: I'd add one more: the grounding in Wikipedia categories means the agents are aligned with human knowledge structures. That's a step toward making AI systems more trustworthy, because you can trace their behavior back to a defined domain.

Lu: And I think that's the foundation for something bigger — we'll talk about where this could go in our final segment.

Conclusion: Tom: Alright, we're wrapping up our discussion of "LLM Agents Factory: Retrieval of Domain-Specific LLM Agents." Jane, what's the one thing you want listeners to remember?

Jane: The core insight is simple: don't generate agents on the fly when you can retrieve them from a well-organized library. The authors built a repository of over twenty thousand domain-grounded agents, and they showed that retrieval beats dynamic generation on accuracy while being cheaper and faster.

Tom: And the implications are pretty big. For industry, this means AI systems can be more predictable, more auditable, and more cost-effective. For research, it opens up questions about how to build and maintain these agent libraries — how to keep them fresh, how to expand them to new domains.

Lu: I think the most exciting direction is extending this beyond Wikipedia categories. The authors mention that the same pipeline could be instantiated over medical, legal, or financial ontologies. Imagine a hospital system where every agent is grounded in a validated medical taxonomy — that would be a huge step toward trustworthy AI in healthcare.

Meng: And from an engineering perspective, I love that they released the code and the agent base. That means we can actually build on this, adapt it to our own domains, and measure the trade-offs in our own systems.

Jane: There are limitations, of course. The agent base is grounded in Wikipedia, which doesn't cover every niche domain. And the synthetic training data inherits biases from the teacher model. But the framework itself is solid, and the authors are clear about what's left to do.

Tom: Future work includes metadata-aware retrieval — searching by domain or role explicitly — and hierarchical orchestration that only invokes multi-agent reasoning when the task is complex enough to justify the cost. That's a smart way to balance quality and efficiency.

Lu: And I'd add that the distillation approach could be refined to produce more concise profiles, reducing the token overhead even further. There's a lot of room to optimize.

Tom: So we're saying goodbye to "LLM Agents Factory" — a paper that treats agent construction as a search problem and shows that sometimes the best way to build is to look it up. Thanks for listening, and we'll see you for the next paper.

Jane: Take care, everyone.