On the Effectiveness of Large Language Models in Writing Alloy Formulas

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "On the Effectiveness of Large Language Models in Writing Alloy Formulas".

Jane: The paper was written by Yang Hong, Shan Jiang, Yulei Fu and Sarfraz Khurshid from University of Texas at Austin.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Title: Tom: Welcome back to the show, everyone! Today we're looking at a paper that's got me genuinely fired up — it's called "On the Effectiveness of Large Language Models in Writing Alloy Formulas," from researchers at UT Austin.

Jane: And Tom, I have to say, when I first saw the title, I thought, "Alloy? Like, the metal?" But no, this is about a specification language — a way to describe exactly how a system should behave before you even write the code.

Tom: Right! And that's the part that gets me excited. We've seen LLMs write Python, Java, all the mainstream stuff. But Alloy is this niche, academic language based on logic and relations. It's not something you see in everyday software development.

Jane: Exactly. And the paper is basically asking: can these AI models handle a language that's so different from what they usually see? The authors tested two models, ChatGPT and DeepSeek, on eleven different properties — things like "this graph has no cycles" or "this relation is symmetric."

Tom: And the results? Honestly, they surprised me. The models weren't just getting it right sometimes — they were generating multiple correct answers. Like, for one property, DeepSeek came up with nineteen correct formulas out of twenty attempts.

Jane: That's wild. But here's what I love about this paper — they didn't just ask the models to write formulas from scratch. They also tested whether the models could take an existing Alloy formula and rewrite it in a different but equivalent way.

Tom: So it's like asking someone to explain the same idea in three different ways, to prove they actually understand it, not just memorize one phrasing.

Jane: Precisely. And the models did that too. They generated lots of unique, correct alternatives. Some of them were so unusual that even the researchers said they gained new insights into how to formalize these properties.

Tom: That's the part that blows my mind. The AI isn't just regurgitating training data — it's actually finding creative ways to express logical ideas. And that has huge implications for how we build reliable software.

Jane: Absolutely. Because if we can trust AI to write these specifications, we can catch bugs before they ever make it into the code. That's a game-changer.

Tom: And we're just getting started. Next up, we're going to dig into the actual methodology — how they set up these experiments and what exactly they asked the models to do.

Jane: Can't wait. This is the kind of research that makes you rethink what these models are capable of.

Summary: Tom: So Jane, we've established that this paper, "On the Effectiveness of Large Language Models in Writing Alloy Formulas," is about testing AI's ability to write formal specifications. But let's get into the nitty-gritty of how they actually did it.

Jane: Good idea. So they set up three different kinds of tasks. The first one is what they call "English to Alloy" — they give the model a property described in plain English, like "no element in S is related to itself," and ask it to write the Alloy formula.

Tom: And the second task is "Alloy to Alloy" — they give the model an existing Alloy formula and ask it to come up with different but equivalent formulas.

Jane: Right. And the third one is really clever. It's called "sketching." They give the model a partial formula with holes in it — like a fill-in-the-blank exercise — and the model has to figure out what expressions and operators go in those holes to make the formula correct.

Tom: And that's important because in traditional programming, when you have a sketch, you need test cases to guide the completion. But here, the models just used the natural language description to figure it out. No tests needed.

Jane: Exactly. And they validated everything using the Alloy analyzer, which is like a built-in proofreader. It checks whether the model's formula is actually equivalent to the correct answer.

Tom: So they weren't just eyeballing it — they had a rigorous, automated way to verify correctness.

Jane: And the results were strong. For the English to Alloy task, both models found at least one correct solution for every single property. For some properties, they found fifteen or more correct solutions out of twenty.

Tom: And for the sketching task, they succeeded on ten out of eleven properties on the first try. The two failures were just syntax errors, and when the researchers pointed that out and asked them to try again, they got it right.

Jane: So it's not like the models didn't understand the logic — they just made a small formatting mistake. Once corrected, they nailed it.

Tom: What I find fascinating is that the presence of a reference formula didn't always help. When they gave the model an existing Alloy formula and asked for equivalents, it didn't necessarily produce more correct answers than when they just gave it the English description.

Jane: That's a great point. It suggests the models are actually reasoning about the logic, not just pattern-matching on the input. Having one solution in front of you doesn't automatically make it easier to come up with many different ones.

Tom: And that's the kind of depth that makes this research so compelling. We're not just seeing if AI can copy — we're seeing if it can truly understand.

Jane: Next, we're going to talk about what this means for the future. What improvements does this paper suggest, and where could this research lead?

Improvements: Tom: So Jane, we've talked about what the paper found. But what really gets me thinking is where this could go. What improvements does this research point toward?

Jane: Well, first off, the paper itself suggests that LLMs could become an integral part of writing specifications — just like they're becoming part of writing code. And that's a huge deal because formal methods have always had a learning curve problem.

Tom: Right. Alloy is powerful, but it's hard to learn. The syntax is different from anything most developers use daily. If an AI can translate your English description into correct Alloy, suddenly that barrier disappears.

Jane: And it's not just about writing from scratch. The sketching capability is huge. Imagine you have a partially written specification and you're not sure how to complete it. You could just describe what you want in English, and the AI fills in the blanks.

Tom: That's like having a senior engineer sitting next to you who knows Alloy inside and out. But the paper also hints at something deeper — the models generated solutions that were so unusual that the researchers learned from them.

Jane: That's the part that excites me. The AI isn't just a tool — it's a collaborator. It can show you ways of expressing a property that you might never have thought of. That could lead to new insights in how we formalize requirements.

Tom: And let's not forget the practical side. The paper mentions that Alloy has been used in software design, test case generation, even security analysis. If AI makes it easier to write Alloy, all those applications become more accessible.

Jane: Exactly. And the researchers are already thinking about future work — making LLMs a standard part of the specification-writing process, just like they're becoming standard for writing implementations.

Tom: I also love that they used publicly available models with no fine-tuning. They didn't train ChatGPT or DeepSeek on Alloy specifically. They just asked them directly. And it worked.

Jane: That's remarkable. It means this capability is already available to anyone. You don't need a specialized AI — you can use the same chatbot you already have access to.

Tom: So the improvement isn't in the models themselves — it's in how we use them. We just need to ask the right questions.

Jane: And that's what makes this paper so forward-looking. It's not just reporting results — it's showing us a new way to work. Next up, we'll wrap up with our final thoughts on what this means for the world.

Conclusion: Tom: Alright, Jane, let's bring it home. We've been discussing "On the Effectiveness of Large Language Models in Writing Alloy Formulas," and I think it's fair to say this paper is a big deal.

Jane: Absolutely. The core finding is simple but powerful: LLMs can write Alloy formulas effectively, whether from English descriptions, from existing formulas, or by completing sketches. And they can do it in multiple creative ways.

Tom: And they validated everything with the Alloy analyzer, so we know these aren't just plausible-looking answers — they're actually correct.

Jane: The implications go beyond just Alloy. This suggests that AI can handle specialized, logic-based languages that are very different from mainstream programming languages. That opens the door for AI-assisted formal methods across the board.

Tom: And for the average developer, that means safer software. If we can write precise specifications more easily, we can catch bugs earlier, build more reliable systems, and maybe even make formal methods a standard part of the development process.

Jane: I also love that the paper highlights the collaborative potential. The models didn't just replicate known solutions — they came up with novel formulations that even surprised the researchers. That's the kind of partnership that could push the field forward.

Tom: So what's the takeaway for our listeners? If you're a developer, this is a tool you should explore. If you're a researcher, this is a direction worth pursuing. And if you're just curious about AI, this is another example of how far these models have come.

Jane: Well said, Tom. We're going to say goodbye to this paper and get ready to discuss the next one. But before we go, I want to thank our listeners for joining us.

Tom: And remember — the future of software might just be written in a language you've never heard of, with a little help from AI.

Jane: See you next time!