The Death of Schema Linking? Text-to-SQL in the Age of Well-Reasoned Language Models
cs.CL, cs.AI, cs.IR
Submitted: 2024-08-14
Updated: 2024-08-18
Code: https://github.com/gkamradt/LLMTest_NeedleInAHaystack
License: http://creativecommons.org/licenses/by-sa/4.0/
The gist: Schema linking is a crucial step in Text-to-SQL pipelines.
Terminology
Abstract
Schema linking is a crucial step in Text-to-SQL pipelines. Its goal is to retrieve the relevant tables and columns of a target database for a user's query while disregarding irrelevant ones. However, imperfect schema linking can often exclude required columns needed for accurate query generation. In this work, we revisit schema linking when using the latest generation of large language models (LLMs). We find empirically that newer models are adept at utilizing relevant schema elements during generation even in the presence of large numbers of irrelevant ones. As such, our Text-to-SQL pipeline entirely forgoes schema linking in cases where the schema fits within the model's context window in order to minimize issues due to filtering required schema elements. Furthermore, instead of filtering contextual information, we highlight techniques such as augmentation, selection, and correction, and adopt them to improve the accuracy of our Text-to-SQL pipeline. Our approach ranks first on the BIRD benchmark achieving an accuracy of 71.83%.
Sources
- Natural Language Interfaces to Databases - An Introduction
- MAGIC: Generating Self-Correction Guideline for In-Context Text-to-SQL
- C3: Zero-shot Text-to-SQL with ChatGPT
- Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation
- Towards Complex Text-to-SQL in Cross-Domain Database with Intermediate Representation
- X-SQL: reinforce schema representation with context
- Next-Generation Database Interfaces: A Survey of LLM-based Text-to-SQL
- Summary of a Haystack: A Challenge to Long-Context LLMs and RAG Systems
- MCS-SQL: Leveraging Multiple Prompts and Multiple-Choice Selection For Text-to-SQL Generation
- The Dawn of Natural Language to SQL: Are We Fully Ready?
- Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs
- NeedleBench: Evaluating LLM Retrieval and Reasoning Across Varying Information Densities
- Bridging Textual and Tabular Data for Cross-Domain Text-to-SQL Semantic Parsing
- A Survey of Text-to-SQL in the Era of LLMs: Where are we, and where are we going?
- Hybrid Ranking Network for Text-to-SQL
- End-to-end Text-to-SQL Generation within an Analytics Insight Engine
- DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction
- Before Generation, Align it! A Novel and Effective Strategy for Mitigating Hallucinations in Text-to-SQL Generation
- Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
- Reflexion: Language Agents with Verbal Reinforcement Learning
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering