ADVENT: LLM-Driven Automatic Predicate Invention for ILP

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: Today's paper: "ADVENT: LLM-Driven Automatic Predicate Invention for ILP".

Jane: The gist The ADVENT mechanism pairs LLM abductive generation with Prolog deductive verification to create an iterative loop for predicate invention in Inductive Logic Programming,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So we’re talking about ADVENT today, this paper is about predicate invention in Inductive Logic Programming and how they fixed that bottleneck.

Jane: Yeah, the main problem with predicate invention is that existing methods rely too much on human expertise and end up creating predicates that are kind of opaque, which makes it hard to use them in new areas or reuse them elsewhere.

Tom: Exactly. So this paper introduces ADVENT, which is an LLM-driven mechanism for predicate invention in ILP.

Lu: The core idea is pairing the LLM’s ability to abduce with Prolog’s deductive verification, creating a loop where the concrete execution results guide the LLM to pick better candidates for new predicates.

Jane: It sounds like they are trying to make this process much more automated than just relying on domain experts.

Meng: I wonder how automated that actually is, though. We're talking about inventing new rules here, right?

Tom: They’re using Large Language Models to spot implicit patterns in structured relational data and invent these auxiliary predicates with actual names and definitions that make sense.

Lalam: And those invented predicates and learned rules get put into a knowledge pool for reuse across different tasks.

Jane: That sounds like it could really help with cross-task reuse, which is a big deal in machine learning.

Lu: They show experiments on nine poker-hand concepts across seven different LLMs, and the results are pretty strong.

Tom: The paper says that LLM-driven PI gets a fifty-eight percent success rate where standard ILP alone fails completely.

Meng: Fifty-eight percent against what baseline? That sounds like a lot of improvement for something so fundamental to learning rules.

Jane: And they add formal verification on top of that, and the success rate goes up to eighty percent.

Lalam: So, the deductive verifier is really helping guide how the LLM refines those candidates.

Tom: That's what I was hoping for with formal verification—getting concrete execution results to guide the refinement process rather than just letting the LLM guess.

Paper summary: Lu: The paper shows that this mechanism actually produces rules that are human-interpretable, which is important because we don't want black boxes.

Jane: And they mention something really interesting about that knowledge pool.

Lalam: They found that the knowledge pool can yield gains of up to thirty-one percentage points in performance for new targets.

Tom: That’s a pretty big jump from where ILP alone was failing, and it’s happening while also making rules interpretable.

Meng: From a production standpoint, having reusable components is good because it means you don't have to invent everything from scratch every time you tackle a new problem.

Lu: The framework itself is structured in four stages for each learning task.

Tom: It starts with the Representation Check, which is basically a quick test to see if the existing background knowledge already has enough power before starting anything else.

Jane: If that check fails, then you enter this iterative invention loop where the LLM generates predicates and Prolog verifies them, right?

Lu: That’s Stage two of the pipeline: when the initial representation check fails, it signals that we need to enrich our hypothesis space by automatically synthesizing new auxiliary predicates while Prolog guides the refinement.

Tom: So Stage three is a Post-invention Representation Check, which is basically re-testing if adding those new predicates actually helped before we start the main rule induction phase.

Jane: And then stage four is the actual ILP Rule Induction, where all those invented predicates are injected into the original background knowledge to create a richer search space for finding rules.

Meng: What about performance at scale? I mean, this whole process with the LLM and Prolog verification sounds complex.

Lalam: The paper mentions that formal verification often outperforms LLM self-critique in five out of seven models they tested, which suggests that having grounded execution results gives more reliable signals than just the AI looking at its own output.

Tom: I agree with Lalam there. It’s about getting concrete data back from the system to guide the invention process, not just having the LLM guess what works by itself.

Paper summary: Jane: So for someone who doesn't read this paper, it boils down to an AI system that learns better relational rules by letting an LLM propose new ones and then using a formal logic checker to make sure those proposals actually work on the data.

Lu: That’s the high-level concept. But one thing they point out is the limitation around that initial check.

Tom: Yeah, there is a limitation mentioned regarding how much of the initial background knowledge needs to be there before it signals that invention is necessary.

Jane: So, if your starting knowledge isn't rich enough, you have to invent stuff right away?

Lu: Exactly. The paper highlights that this process can be computationally intensive because of the iterative loop and the verification steps, which might make it challenging for real-world use cases with massive datasets.

Meng: That’s what I worry about when deploying anything like this. If the cost to run that entire PI loop is too high, it’s just theoretical.

Lalam: They did mention that future work will focus on developing selective retrieval mechanisms to filter that knowledge pool based on relevance, which should help manage the complexity of reusing all those invented predicates.

Tom: It sounds like they are moving toward a more practical system by focusing on filtering what’s in the pool rather than just letting the LLM look at everything every time.

Jane: So, ADVENT is presented as a promising direction for automating predicate invention and allowing this knowledge reuse across different tasks, even if it still has some scaling hurdles to clear.

Lu: That seems like a fair assessment of where they are now with this mechanism.

Meng: It’s interesting because it connects the high-level generative power of an LLM with the rigor of formal verification, which is exactly what we need when building reliable systems in production.

Lalam: So, ADVENT pairs those two things up to tackle predicate invention in ILP better than before.

Conclusion: Tom: So we've been diving into ADVENT, this paper is about how they tackled predicate invention in Inductive Logic Programming, and the authors are Tingting Yu and Pei-Cing Huang.

Jane: Yeah, so what they did was create this whole system that pairs a Large Language Model with Prolog verification to invent new rules automatically.

Tom: Right, it’s not just letting the AI guess; they’re setting up an iterative loop where the execution results from Prolog guide the LLM to refine those candidate predicates.

Lu: The core idea is using that feedback—the concrete execution results—to let the LLM identify implicit patterns in structured data and invent auxiliary predicates with actual names and definitions.

Meng: I'm curious, what does this actually change for us when we're trying to deploy these logic systems? Is this something we can just drop into our existing pipelines?

Lalam: The paper shows that these invented rules and learned concepts accumulate in a knowledge pool, which means they can be reused across different learning tasks.

Jane: So the main point is that this mechanism isn't just about inventing one rule; it’s about building a reusable library of logic components for future problems.

Tom: Exactly. And the experimental results are what really grab my attention—they tested nine poker-hand concepts using seven different Large Language Models.

Jane: They found that LLM-driven predicate invention hit a success rate of fifty-eight percent in cases where standard ILP methods completely failed to find any solution.

Tom: And they showed that adding formal verification on top of that bumps it up to eighty percent success, which is a significant jump from the baseline.

Lu: The knowledge pool itself yielded gains of up to thirty-one percentage points for new targets, and the rules produced were human-interpretable.

Meng: Human-interpretable rules are good, but what about the complexity? How do we manage all that invention happening in a loop on real data when you're running this at scale?

Lalam: The authors address that by showing how they use an initial Representation Check to decide if invention is even necessary at the start of learning.

Jane: So, before they even start inventing rules, the system quickly checks if the existing background knowledge is already rich enough to solve the problem.

Tom: Right, and if that check fails—which it often does—then you enter this iterative invention loop where they're synthesizing new predicates while Prolog verifies them.

Lu: The authors also talk about a four-stage pipeline for each learning task, starting with that Representation Check, then the invention loop, a post-invention check, and finally the main ILP rule induction.

Meng: It sounds like it’s quite complex to set up all those stages just to find one new predicate. What's the actual cost in terms of computation for running this whole ADVENT mechanism?

Lalam: The paper suggests that the verification step is key here because it prevents the LLM from proposing things that are structurally invalid before you waste time running full induction.

Jane: So, if we can get a system like this to reliably generate and verify new predicates, it fundamentally changes how we approach knowledge acquisition in these logical learning systems.

Tom: It does. Because right now, predicate invention is stuck because it requires too much human domain expertise to even suggest what kind of new predicate might be useful.

Jane: And ADVENT tries to automate that suggestion process by using the LLM’s pattern recognition on structured data and verifying the results with a formal system.

Tom: The implication for me is that if we can make this loop reliable, we move away from hand-crafted background knowledge toward systems that discover their own necessary structure.

Jane: It's promising because it bridges the gap between the creative, pattern-finding power of Large Language Models and the rigorous, verifiable constraints of formal logic programming.