Benchmarking Testing in Automated Theorem Proving

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "Benchmarking Testing in Automated Theorem Proving".

Jane: The paper was written by Jongyoon Kim, Hojae Han and Seung-won Hwang from Seoul National University and Electronics and Telecommunications Research Institute.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Title: Tom: Welcome back to the show, everybody. Today we're looking at a paper that just hit arXiv called "Benchmarking Testing in Automated Theorem Proving." Jane, I have to say, the title alone got me curious.

Jane: Same here, Tom. When I first read it, I thought, okay, we already have benchmarks for theorem proving. But the word "testing" is the twist. In software, you don't just check if code compiles — you run tests to see if it actually does what it's supposed to do. This paper says, why don't we do that for math proofs?

Tom: Right. And that's a bigger deal than it sounds. For a long time, we've been checking whether an AI-generated proof compiles in Lean. If it compiles, we call it correct. But this paper points out that compiling just means the logic is internally valid. It doesn't mean the theorem actually says what the problem asked for.

Jane: Exactly. They give a great example. You ask the model to prove that addition is commutative, a plus b equals b plus a. The model instead writes a plus b equals a plus b. That's a tautology. It compiles fine. It's a true statement. But it's completely useless for the original problem.

Tom: And that's where "testing" comes in. In software, you write a function, and then you have other code that calls that function. If your function is wrong, the code that depends on it breaks. The paper applies that same logic to math theorems.

Jane: So instead of just checking the theorem in isolation, they look at all the other theorems in the repository that use this one. If the generated theorem is semantically wrong, those dependent theorems won't compile anymore.

Tom: They call it Testing Accuracy, or TA. And the numbers are pretty striking. The best model they tested, Claude-Sonnet-four point five, gets about eighty percent compilation accuracy. But its testing accuracy drops to under thirty-nine percent.

Jane: That's a huge gap. And it tells us something important — our current evaluation methods are giving us false confidence. We think these models are doing great, but they're actually missing the meaning of the problems.

Tom: I love that they're borrowing from software engineering. Integration testing has been standard practice for decades. Why shouldn't formal math verification work the same way?

Jane: Because it's harder to set up. You need a repository with real dependencies, not just isolated problems. And that's exactly what they built. We'll get into how they did it in a bit, but first — let's just appreciate the shift in thinking.

Tom: Absolutely. This could change how we evaluate every theorem-proving model from here on out.

Jane: And it might explain why some models look great on paper but fail in real mathematical work.

Tom: Stay with us — next we'll talk about the benchmark itself and how they built it.

Summary: Tom: So we're back with "Benchmarking Testing in Automated Theorem Proving." Jane, let's get into the actual benchmark. They call it T2.

Jane: Right. T2 stands for Theorem Testing. And the key thing is, they didn't create new math problems from scratch. They went to real-world Lean four repositories — actual math libraries used by researchers.

Tom: That's a big deal. Most benchmarks use isolated olympiad-style problems. This one uses real research-level math with real dependencies.

Jane: Exactly. They built a dependency graph. Each theorem is a node, and the edges show which theorems use which. Then they pick a target theorem — that's the one the AI has to generate. The theorems that depend on it become the test suite.

Tom: So if the AI generates a wrong version of the target, all those dependent theorems fail to compile. It's like breaking a chain.

Jane: And they made sure each target has at least two successor theorems. On average, there are forty-one successors per problem. That's a lot of tests.

Tom: They also created a harder subset called T2 Hard. Those are problems where the model has to generate both the proposition and the proof — not just fill in a proof for a given statement.

Jane: And the results on that hard set are brutal. The best model gets under five percent testing accuracy. GPT-five-nano compiles at over seventy-five percent but only gets one point five percent on testing.

Tom: That's a fifty-fold gap. It really shows that compilation is not a reliable measure of semantic correctness.

Jane: And they tested eighteen models — closed-source, open-source, and specialized math models. The specialized ones, like DeepSeek-Prover and Goedel-Prover, they actually overfit to compilation.

Tom: Meaning they're really good at producing syntactically valid proofs, but they don't understand the meaning any better than general models.

Jane: Right. The paper says it bluntly — domain-specific fine-tuning improves syntactic fluency without improving semantic correctness.

Tom: That's a hard pill to swallow for the theorem-proving community.

Jane: It is. But it's also a wake-up call. We need to evaluate these models the way we evaluate software — by testing them in context, not just checking if they compile.

Tom: And the fact that they built this benchmark automatically, without human annotation, is huge. It means it can scale.

Jane: They used a strong LLM to generate natural language descriptions of the theorems. They manually verified a sample and found no errors.

Tom: So the whole pipeline is automated. Dependency graph, context extraction, natural language annotation — all of it.

Jane: Which means we can apply this to any Lean repository out there. The method is general.

Tom: And that's the exciting part. This isn't just a one-off benchmark. It's a framework.

Jane: Next, we'll talk about what the paper suggests we do differently — and what that means for the future of AI math.

Improvements: Tom: Welcome back. We're still on "Benchmarking Testing in Automated Theorem Proving." Jane, the paper doesn't just diagnose the problem — it suggests concrete improvements. Let's talk about those.

Jane: The biggest one is using successor theorems as context. When you give the model the theorems that depend on the target, it performs better on testing accuracy.

Tom: That makes sense. If you know what other theorems need to use your result, you can shape your statement to fit.

Jane: But here's the interesting part — providing the natural language proof alone doesn't help much. And providing successor theorems without the NL proof can actually hurt.

Tom: So it's the combination that matters.

Jane: Exactly. The NL proof gives the reasoning context, and the successor theorems give the semantic constraints. Together, they help the model understand what the theorem needs to say.

Tom: They also tried few-shot prompting — giving the model examples of solved problems. And that didn't help either.

Jane: Which tells us the difficulty isn't about understanding the task format. The models know what to do. They just can't do it well enough.

Tom: And iterative refinement with compiler feedback? They tried that too. The gains were minimal.

Jane: Right. Hilbert, which is a method that uses compiler errors to refine proofs, only improved from three point two to five point zero percent on the hard set. Still very low.

Tom: So the paper is saying — no amount of prompt engineering or feedback loops fixes the fundamental gap.

Jane: The fundamental gap is between syntactic validity and semantic correctness. Models can produce proofs that type-check, but they can't reliably produce theorems that mean what the problem asks.

Tom: And that's a deeper problem. It's not about better prompting. It's about better training or better architectures.

Jane: The paper also shows that BLEU score, which measures lexical similarity to a reference proof, is completely unreliable. Even at high BLEU scores, over seventy percent of samples are semantically incorrect.

Tom: So we can't use that as a proxy either.

Jane: No. The only reliable signal is whether the dependent theorems compile.

Tom: Which brings us back to testing. The more successor theorems you have, the stricter the evaluation.

Jane: And they show that. With only two successors, models get around seventy percent testing accuracy. With five or more, it drops to near zero.

Tom: So each additional test catches more semantic errors.

Jane: It's like adding more unit tests to your code. Each one catches a different bug.

Tom: The paper's message is clear — we need to move from checking whether proofs are valid to checking whether they're useful.

Jane: And useful means they work in the context of real mathematics, not just in isolation.

Tom: That's a paradigm shift.

Jane: It is. And it's one that could make AI theorem proving actually trustworthy.

Tom: Let's wrap this up in our final segment.

Conclusion: Tom: And we're back for the final stretch on "Benchmarking Testing in Automated Theorem Proving." Jane, let's pull it all together.

Jane: So the paper gives us two things. First, a new metric called Testing Accuracy that checks semantic correctness by seeing if dependent theorems compile. Second, a benchmark called T2 built from real Lean repositories with over two thousand two hundred problems.

Tom: And the key finding is that compilation accuracy massively overestimates model capability.

Jane: The best model gets eighty percent compilation but under thirty-nine percent testing accuracy. On the hard set, it's even worse — under five percent.

Tom: So what does this mean for the field?

Jane: It means we need to change how we evaluate theorem provers. If we keep using compilation as the gold standard, we'll keep getting models that look great but don't actually understand math.

Tom: And that's not just an academic problem. These models could be used to verify software, check financial systems, even validate safety-critical code.

Jane: Right. If a model can prove a theorem that doesn't say what it's supposed to say, that's a failure with real consequences.

Tom: The paper also shows that specialized math models don't do better on semantic correctness — they just overfit to syntax.

Jane: Which suggests that the next big breakthrough won't come from more training data or better prompting. It'll come from training models to reason about meaning, not just form.

Tom: And the benchmark itself is fully automatic. No human annotation needed. So it can scale to any Lean repository.

Jane: That's the exciting part. This isn't a one-time evaluation. It's a framework that can grow with the field.

Tom: There are limitations, of course. It only works for Lean four and it doesn't apply to standalone theorems with no dependents.

Jane: But for the vast majority of real-world math, where theorems build on each other, this is the right approach.

Tom: So what's the takeaway for our listeners?

Jane: The takeaway is that we need to test AI math the way we test software. Not just check if it compiles, but check if it works in context.

Tom: And that's a standard we should demand.

Jane: Absolutely. This paper gives us the tools to do that.

Tom: Alright, that's a wrap on "Benchmarking Testing in Automated Theorem Proving." Great discussion, Jane.

Jane: Great discussion, Tom. Thanks to everyone listening. We'll be back with the next paper soon.

Tom: Until then, keep questioning your metrics.