Faster, Cheaper, More Accurate: Specialised Knowledge Tracing Models Outperform LLMs

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "Faster, Cheaper, More Accurate: Specialised Knowledge Tracing Models Outperform LLMs".

Jane: The paper was written by Prarthana Bhattacharyya, Joshua Mitton, Ralph Abboud and Simon Woodhead from Eedi and Learning Engineering Virtual Institute.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Jane: We also have Lu with us today — senior AI researcher at Tsinghua.

Tom: We also have Meng with us today — lead engineer at a mysterious AI startup.

Jane: We also have Lalam with us today — the in-house Large Language Model.

Tom: Alright, let's get started.

Title: Tom: Welcome back to the show, everyone! Today we’re looking at a paper that’s going to ruffle some feathers in the AI world. It’s called “Faster, Cheaper, More Accurate: Specialised Knowledge Tracing Models Outperform LLMs.” Jane, I have to say, that title is basically a dare to every big tech company out there.

Jane: It really is, Tom. And I love it. The paper is from researchers at Eedi, which is an online learning platform, and they’ve basically pitted the big general-purpose models like GPT-4o-mini and Gemini against these tiny, specialized models built just for one task: predicting how a student will do on a future math question.

Tom: Right, and the title tells you exactly how it ends. But the scale of the win is what got me. I mean, we’re talking about models with under a million parameters beating models with billions of parameters. That’s like a bicycle beating a Formula One car in a city race.

Jane: That’s a great analogy, because the bicycle is built for the city streets, right? The LLMs are built for everything, but these knowledge tracing models are built for one specific road. They’re looking at a student’s history of answers and trying to figure out if they’ll get the next one right.

Tom: And they’re not just winning by a little. The best specialized model hit about seventy-two point eight percent accuracy, while the best LLM they tested, Gemini, only got to sixty-six point five percent. And some of the LLMs were actually worse than just guessing the average score on every question. That’s a brutal baseline to miss.

Jane: It’s a really important baseline, though. If a model can’t beat "always guess the most common answer," it’s not learning anything about the student. And a few of these huge LLMs couldn’t do it. They just don’t have the temporal understanding—they don’t track the student’s learning journey the way a specialized model is forced to.

Tom: So, the title is basically a spoiler, but the details are in the numbers. And I have to say, as someone who loves efficiency, seeing a zero point seven three million parameter model beat a seven billion parameter model is deeply satisfying.

Jane: It’s the whole argument for specialization in a nutshell. And it sets up the big question we’re going to dig into: if they’re more accurate, are they also actually practical to run? Because that’s where I think the real story is.

Tom: Oh, absolutely. The accuracy gap is one thing, but the cost and speed gap is where this paper really gets wild. Stick around, because we’re about to talk about latency and what it actually costs to serve a hundred thousand students.

Summary: Tom: So, Jane, we’ve established that the specialized models win on accuracy. But the paper’s summary really hammers home the other two words in the title: faster and cheaper. And honestly, the numbers there are almost comical.

Jane: They really are. Let’s talk about speed first. The knowledge tracing models—DKT, SAKT, and their new one, LLM KT—they all predict a student’s next answer in under a quarter of a second. That’s per student, for their whole history. Meanwhile, GPT-4o-mini takes about three seconds, which is slow but manageable.

Tom: But then you get to the open-source models they hosted themselves. Qwen2 point 5-7B took over three thousand two hundred seconds per student. That’s almost an hour. For one student’s prediction. You can’t run an adaptive learning platform if the model takes an hour to decide what question to show next.

Jane: Exactly. And that’s where the "cheaper" part comes in. They calculated the annual cost to serve one hundred thousand students, each getting forty predictions. The specialized models cost less than two dollars a year. Two dollars, Tom.

Tom: And the LLMs? GPT-4o-mini came in around,300 a year. But the self-hosted ones were brutal. Qwen2 point 5-7B would cost almost,000 a year. Llama-1B, even the fine-tuned one, was nearly,000. So we’re talking about a six hundred to twelve thousand times cost difference.

Jane: And here’s the kicker—they’re paying more for worse results. The fine-tuned Llama-1B got seventy-one percent accuracy, which is actually decent, but it costs,000 and takes half an hour per student. The specialized SAKT model got seventy-two point seven percent accuracy, costs under a dollar, and runs in a tenth of a second.

Tom: It’s not even a trade-off. It’s just a strictly better option. And I think that’s the point the authors are making. They’re not saying LLMs are useless—they’re saying that for this specific task, using a general-purpose model is like using a sledgehammer to hang a picture frame.

Jane: Right. And the sledgehammer is also on fire and costs a fortune. But I do want to bring in Lu and Meng here, because I think they’ll have different takes on why this gap exists and whether it can be closed. Lu, you work with these models every day—does this surprise you?

Lu: Honestly, Jane, it doesn’t surprise me at all. The LLMs are solving a much harder problem. They’re trying to be a general reasoner. The KT models are solving a narrow temporal prediction problem, and they can encode the structure of that problem directly into the architecture. The LLM has to rediscover that structure from text prompts, which is incredibly inefficient.

Meng: And from a deployment standpoint, it’s not just the inference cost. It’s the memory footprint, the GPU requirements, the cold-start latency. Running a 7B model on CPU is painful. The specialized models are so small you can run them on basically anything. That’s a massive operational advantage.

Tom: Great points, both of you. So we’ve got accuracy, speed, and cost all favoring the specialists. But the paper doesn’t stop at just comparing. They also built their own model, and that’s where things get interesting. Let’s talk about that next.

Improvements: Tom: So, Jane, the paper doesn’t just say "our old models are better." They actually built a new model, and it’s the best of both worlds. It’s called LLM KT, and it’s a clever hybrid that uses LLM embeddings but keeps the temporal modeling in a tiny transformer.

Jane: Right, and this is the part I found really smart. Instead of feeding the LLM the whole student history and asking it to reason, they use a small embedding model—Qwen three 0 point 6B—to convert the question text, the construct, the misconceptions, all of that into fixed vectors. They cache those embeddings offline.

Tom: So the LLM does the heavy lifting of understanding the question content, but it happens once, ahead of time. Then at runtime, the tiny KT model just looks up those pre-computed vectors and does its temporal thing. No LLM inference in the loop at all.

Jane: Exactly. And the result is that LLM KT gets the best accuracy in the whole paper—seventy-two point eight percent, just barely edging out SAKT at seventy-two point seven percent. But it also keeps that sub-zero point two five second latency and the sub- annual cost. They got the semantic richness of an LLM without paying the runtime price.

Meng: That’s a really elegant engineering solution. You’re essentially using the LLM as a feature extractor, not as a predictor. And because the embeddings are cached, you could even swap out the embedding model later if a better one comes along, without retraining the temporal model.

Lu: And I think this points to the future. We’re going to see more of this pattern—using LLMs to generate rich representations, but then using small, specialized models to do the actual prediction. It’s a division of labor. The LLM handles language understanding; the specialized model handles the student’s learning state.

Tom: It also solves a practical problem they mentioned in the paper. The zero-shot Llama-1B model kept failing to follow the prompt format. It would output random text instead of just "Yes" or "No." By taking the language generation out of the loop entirely, you eliminate that failure mode completely.

Jane: That’s a great point. The fine-tuned Llama did better—seventy-one percent accuracy—but it still required prompt engineering and careful output parsing. The LLM KT model just sidesteps all of that. It’s a much more robust system for production.

Meng: And the cost comparison really shows it. The fine-tuned Llama cost,000 a year to serve. The LLM KT model costs.73. For the same scale of deployment, you’re getting better accuracy for a fraction of the cost. That’s the kind of improvement that actually makes a product viable.

Tom: So the improvement isn’t just a better model—it’s a better architecture for the problem. And it makes me wonder, Lu, where does this leave the big LLMs? Are they just going to be relegated to embedding generators?

Lu: Not relegated, but specialized. They’re still the best at understanding language. But for prediction tasks with temporal structure, we’re going to see more and more of these hybrid systems. The LLM provides the semantic understanding; the small model provides the temporal reasoning. It’s a partnership, not a competition.

Jane: And that’s a much more nuanced takeaway than "LLMs are bad." They’re not bad—they’re just not the right tool for every job. And this paper proves that with numbers, not just opinions. Let’s bring in Lalam to get a bigger picture on what this means for education technology.

Conclusion: Tom: Alright, we’ve covered the accuracy, the speed, the cost, and the clever hybrid architecture. Let’s wrap this up. Jane, what’s the big picture takeaway from "Faster, Cheaper, More Accurate: Specialised Knowledge Tracing Models Outperform LLMs"?

Jane: The big picture is that specialization wins for specific tasks. The paper shows that if you want to predict student performance, you should use a model built for that. The LLMs, even the best ones, just can’t match the temporal modeling that KT models do natively.

Tom: And the practical implications are huge. For an EdTech platform trying to serve millions of students, the cost difference between and,000 a year is the difference between a viable product and a money pit. It’s not just about accuracy—it’s about whether you can actually deploy it.

Meng: And from an engineering standpoint, the LLM KT model is the kind of solution I love. It uses the best of both worlds—LLM embeddings for semantic understanding, small transformer for temporal prediction—and it does it in a way that’s actually deployable on commodity hardware.

Lu: I’d add that this is a template for other domains too. Anywhere you have temporal sequences with rich textual context—health records, financial transactions, even recommendation systems—this hybrid pattern could apply. Use the LLM to understand the content, use a small model to track the state.

Lalam: And I think the cultural impact is about access. When models are this cheap and fast, they can be deployed in resource-constrained environments—schools in developing regions, rural areas with limited connectivity. The barrier to personalized learning drops dramatically. That’s the real win.

Tom: That’s a beautiful way to put it, Lalam. The paper isn’t just about winning a benchmark—it’s about making adaptive learning accessible to everyone. And that’s a mission we can all get behind.

Jane: Absolutely. So, to sum up: specialized KT models beat LLMs on accuracy, speed, and cost. The new LLM KT model shows you can have the best of both worlds by using LLM embeddings offline. And the implications for education are profound.

Tom: And with that, we’re going to say goodbye to this paper. It’s been a great discussion, and I think we’ve all learned something about the value of specialization. Thanks to Lu, Meng, and Lalam for joining us.

Jane: And thanks to all our listeners. We’ll be back soon with another paper, and we’ll see if it can live up to the standard this one set. Until then, keep learning, keep questioning, and remember—sometimes the smallest model is the smartest choice.

Tom: See you next time, everyone!