How far do on-premises open LLMs get on Text-to-SQL? A cross-family size and technique frontier on BIRD

summary

The gist

The gist: Generation matters more than raw size, self-correction generalizes across three families and is significant, schema linking does not help even with a high-recall embedding retriever, and

In short

The study benchmarked various open-weight models for Text-to-SQL on a BIRD database split, testing different model families and serving methods. It found that self-correction is the most effective technique across models, while schema linking provides no benefit. Self-consistency was deemed not worth the cost, suggesting practitioners should prioritize strong generation and add self-correction to achieve good performance.

Key concepts

Text-to-SQL: This task involves taking a natural language question and translating it into a precise SQL query that can be executed against a database. The study measures how accurately different AI models can perform this translation on specific database schemas.
Self-Correction: This technique involves having the model generate an initial answer, then critically review its own output and make corrections based on the execution results. The research found this method consistently improved accuracy across different model families, making it a robust way to boost performance.
Schema Linking: This is a process where models try to connect natural language terms in a question to specific database tables or columns. The study showed that even using advanced embedding linkers did not significantly improve accuracy on narrow BIRD schemas, meaning this technique is often unnecessary for these types of tasks.
Self-Consistency: This involves running the model multiple times with different sampling methods and aggregating the results to find a consensus. The research concluded that this method provided very little accuracy gain for a high computational cost, indicating it is not a worthwhile technique for improving Text-to-SQL performance.

Terminology used across episodes

Reinforcement Learning (RL) — explained in 25 episodes
Retrieval-Augmented Generation (RAG) — explained in 21 episodes
Sample Complexity — explained in 20 episodes
Flow Matching — explained in 19 episodes
Quantum Fisher Information (QFI) — explained in 19 episodes
Federated Learning — explained in 17 episodes

This episode discusses

The paper

How far do on-premises open LLMs get on Text-to-SQL? A cross-family size and technique frontier on BIRD · Read on arXiv

Vladimir Beskorovainyi

Moscow Institute of Physics and Technology (MIPT)

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Hi, I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "How far do on-premises open LLMs get on Text-to-SQL? A cross-family size and technique frontier on BIRD".

Jane: Here's the gist of the paper. The gist: Generation matters more than raw size, self-correction generalizes across three families and is significant, schema linking does not help even with a high-recall embedding retriever, and self-consistency is not worth its cost.

Tom: How it works The study evaluates Text–to–SQL performance on the BIRD development split using three open model families—Qwen2.5–Coder, CodeLlama, and Llama–3.x—across a ∼5× size range, serving them on-premises with vLLM

6: (Page 1).

Jane: The evaluation employs a matched protocol where model-agnostic recipes are ablated component by component: schema linking, self-correction, and self-consistency

7: (Page 2).

Tom: Execution Accuracy is measured based on whether the execution returns the gold result set

1: (Page 3).

Paper summary: Jane: So we're looking at this paper now titled "How far do on-premises open LLMs get on Text-to-SQL? A cross-family size and technique frontier on BIRD". The authors are really showing that when you’re stuck running AI locally without a cloud API, generation quality and specific recipe choices matter way more than just how big your model is or how many parameters it has.

Tom: They focused heavily on comparing different model families—Qwen2 point 5–Coder and CodeLlama against Llama-three variants—to see where the real performance gains come from when you're working on narrow, specific schemas like BIRD’s > <ref:2606.29733#pg1>

Lu: They organized the study around four research questions: RQ1 How do accuracy gains compare across model size and across model family on-prem? RQ2 Which recipe components improve accuracy, and do the gains generalize across families? RQ3 Does schema linking help if we replace a lexical linker with a stronger retrieval/embedding one? And RQ4 Where do configurations sit on the accuracy–cost frontier in real terms > <ref:2606.29733#pg2>

Meng: The main point they drive home is that self-correction is the only thing that consistently works across all those different model families they tested >

Jane: That means for a Text-to-SQL setup on private data, the best bet is to focus your compute on adding that self-correction step instead of chasing every little linking trick >

Lu: They also found that schema linking doesn't help much at all, even with a better embedding retriever with ninety-six point five percent gold-table recall, it was statistically indistinguishable from no linking > <ref:2606.29733#pg2>

Meng: That rules out the idea that you need to spend compute just to make your schema connections better >

Tom: And self-consistency only changed accuracy by zero point one three points at about five times the tokens, which is not significant >

Jane: So it means you don't need all those fancy recipe additions if your goal is just to get a usable result without spending extra compute time on things that don't actually help >

Lu: The paper is really important because it shows that when you’re dealing with specific, narrow schemas like BIRD’s, the prompt itself isn't usually the bottleneck; generation and self-correction are what actually make the difference in accuracy > <ref:2606.29733#pg5>

Meng: It also gives us a clear picture of where to focus our engineering efforts—on improving the base model family and adding that single self-correction step rather than chasing every little linking trick >

Tom: So we've covered how generation beats size, that self-correction is the key component, and that schema linking doesn't offer any real benefit >

Jane: We’re going to wrap up here with a look at what this all means for Text-to-SQL when you're running models locally. Next time we look at how these results change when we try to scale those self-correction steps up for production environments.

Conclusion: SEGMENT: Conclusion — Title and Authors**

Tom: So we've covered how generation beats size, that self-correction is the key component, and that schema linking doesn't offer any real benefit in this paper about Text-to-SQL.

Jane: Right. The authors of this study are looking at how far on-premises open LLMs can get on Text-to-SQL when you’re working with narrow schemas like BIRD’s.

Lu: They put a lot of work into comparing different model families—Qwen2 point five-Coder and CodeLlama against Llama three variants—to see where the actual performance gains happen when you're running things locally.

Meng: The main point they drive home is that self-correction is the only thing that consistently works across all those different model families they tested on.

Tom: That’s what I liked, seeing a component that works reliably across different architectures. So, if you’re deploying this, you can focus your compute there instead of chasing every little linking trick.

Jane: It shows that generation matters more than just raw size when you look at which recipe works best on those specific schemas.

Meng: That rules out the idea that you need to spend compute just to make your schema connections better. And self-consistency only changed accuracy by zero point one three points at about five times the tokens, which is not significant.

Tom: So the practical implication is that if you’re setting up a Text-to-SQL system on private data, the best bet is to pick a strong, recent family, run it at a mid-to-large size model, add that self-correction step we talked about and just skip both the schema linking and self-consistency work.

Jane: It means you don't need all those fancy recipe additions if your goal is just to get a usable result without spending extra compute time on things that don't actually help.

Tom: So we’ve covered how generation beats size, that self-correction is the key component, and that schema linking doesn't offer any real benefit.

More episodes

← Home