SANE Schema-aware Natural-language Evaluation of Biological Data

High-throughput microscopy generates large, structured datasets capturing cellular responses to pharmacological perturbations, but accessing these datasets typically requires SQL expertise. SANE presents a novel paradigm for domain-specific text-to-SQL evaluation by generating schema-grounded, automatically generated benchmarks tied to real and specific experimental structure.

The gist: Using SANE, we evaluate a few-shot large language model and show that, under constrained schemas with structured prompting and guardrails, accurate query generation is achievable without any model training or finetuning.

System Context

In high-throughput microscopy, such as drug screenings, datasets typically follow a hierarchical experimental design where experiments include one or more cell lines tested against multiple drugs at various concentrations. These experiments generate structured datasets containing both raw measurements and derived analytical values like EC50 and drug sensitivity scores, stored in a relational database. Effective data access often requires SQL expertise due to the limited views available in platforms like Cell-Profiler. Large language models (LLMs) offer a promising solution for natural-language interaction with this structured data via text-to-SQL systems.

LLM Processing Pipeline

The LLM translates natural-language queries into SQL through three stages, as visualized in Figure 2. First, the LLM is tasked to determine whether sufficient context is provided to answer the query, outputting a binary label (missing/sufficient). If the label is missing, a clarification request is returned. Otherwise, a SQL statement is generated using schema-aware prompting. The returned database result is then interpreted by another LLM pass into a natural language response. For SQL generation, schema-aware prompting injects database structure, domain terminology, filtering rules (e.g., excluding border artifacts), and dataset-specific context and examples to guide the generation.

SANE Framework Mechanism

SANE is a framework that systematically generates natural language queries and corresponding ground truth SQL statements directly from database contents and experimental structure. The framework queries a preexisting database from real experiments to extract schema information and representative data instances. Based on this information, SANE constructs queries spanning multiple complexity levels, including simple retrieval, complex analytical queries, and multi-step interactions. It introduces controlled perturbations such as typographical errors, abbreviations, and missing context to simulate realistic user interaction scenarios, with generated queries filled with randomly sampled real data.

Evaluation Benchmarks and Results

The framework defines 69 fine-grained query categories. These include:

1. Simple queries (straightforward retrieval).

2. Complex queries (requiring joins, aggregations, rankings, or listings).

3. Errors (queries with abbreviations or typographical errors).

4. Contextless queries (underspecified and returning overly large result sets).

5. Schema queries (requesting structured information across tables).

6. Multi-step queries (conversational scenarios with incremental refinement of information).

The evaluation compares the predicted context label with ground truth; for missing queries, correctness is determined by label agreement, while for sufficient queries, it requires that the generated SQL execution results match the reference results (result set equivalence). The few-shot system achieved 97.2% overall accuracy across 572 queries. Simple, Complex and Schema related questions achieved high accuracies of 100%, 98.4% and 96.9%, respectively, demonstrating reliable performance without any training or fine-tuning.

Failure Analysis and Limitations

Among the 16 few-shot failures listed in Table 1, 10 involve incorrect missing context label prediction, which is likely driven by the model’s primary focus on accurate SQL generation. In 5 cases, the generated SQL was slightly incorrect, such as omitting drug identifiers in listings or answering overly broad questions instead of asking for clarification. The analysis suggests that "the effective error rate in practical usage for SQL generation is lower than the benchmark suggests, since most failure cases occur due to wrong context label prediction or omitting of names, not because of wrong numerical values. Future improvements should prioritize interactive query refinement and disambiguation rather than model fine-tuning as the model seems to struggle with missing context and unclear questions rather than accurate text-to-SQL generation as shown in the results."

Conclusion

The few-shot LLM achieves high reliability in querying complex biological databases when combined with schema-aware prompting and domain-specific constraints. This approach significantly lowers barriers to accessing high-throughput biological data through prompting alone, eliminating both SQL expertise requirements and the need for model training. Future work could focus on interactive disambiguation mechanisms for the LLM for ambiguous questions and extending the SANE framework to broader biomedical domains. The system demonstrates strong performance, highlighting its potential as a research assistant in constrained, domain-specific environments.

**(Self-Correction Note: The prompt requested a structure that starts with an orienting paragraph, followed by 3 to 5 sections starting with bold headers.

Here are specific improvements to AI systems derived from the SANE framework, focusing on enhancing reliability, scalability, and practical usability in domain-specific data access:

1. Dominance of Schema-Aware Prompting for High Reliability:

2. Systematic Generation of Domain-Specific Benchmarks for Rigorous Evaluation:

3. Integration of Interactive Query Refinement Mechanisms for Ambiguity Resolution:

4. Implementation of Context Classification Guardrails to Mitigate Hallucination Risks:

5. Dominance of Schema-Aware Prompting for High Reliability:

The improved system will leverage the SANE principle where LLMs are explicitly guided by the database schema, domain terminology, and specific filtering rules (e.g., excluding artifacts). By injecting this structured context directly into the prompt (as described in Section 3), the model's output is constrained to generate syntactically correct and logically sound SQL queries tailored precisely to the biological data structure.

The improved AI system can reliably access complex, hierarchical biological data (like high-throughput microscopy results) with a significantly reduced error rate compared to general text-to-SQL models, ensuring that generated queries are both executable and contextually relevant to the specific experimental design.

2. Systematic Generation of Domain-Specific Benchmarks for Rigorous Evaluation:

The system will utilize the SANE framework's ability to automatically generate 572 non-trivial, schema-grounded test cases tied directly to real experimental structures. This moves evaluation away from generic datasets (like WikiSQL) toward benchmarks that mirror the exact complexity and constraints of high-throughput biological data.

The improved AI system can undergo systematic, reproducible benchmarking tailored specifically to novel or proprietary biological databases without requiring manual query writing by human experts, allowing for objective assessment of text-to-SQL performance in specialized domains.

3. Integration of Interactive Query Refinement Mechanisms for Ambiguity Resolution:

Building on the failure analysis (Section 4), the system will implement a multi-stage processing pipeline (as outlined in Figure 2) where, upon detecting a missing context label or low confidence, the LLM is prompted to return a clarification request instead of guessing. This mechanism should be enhanced with synonym expansion and domain-specific entity recognition modules before asking for clarification.

The improved AI system can handle ambiguous or underspecified natural language queries gracefully by proactively engaging in conversational refinement rather than generating incorrect SQL or providing generic dismissive answers, drastically improving practical usability in real-world research assistant applications.

4. Implementation of Context Classification Guardrails to Mitigate Hallucination Risks:

The system will explicitly evaluate the LLM's ability to correctly classify context sufficiency (missing vs. sufficient) as a primary metric for correctness, rather than relying solely on the generated SQL result matching ground truth when context is present. This separation of intent interpretation from execution validation helps isolate where model failures occur—whether in understanding the query's scope or generating accurate syntax.

The improved AI system will exhibit higher robustness against hallucination and incorrect data retrieval by focusing its primary accuracy metric on the model's ability to correctly interpret the user's intent and determine if sufficient context exists, effectively reducing errors stemming from misinterpretation of underspecified inputs.