Functional Entropy: Predicting Functional Correctness in LLM-Generated Code with Uncertainty Quantification

arXiv:2605.28500 · cs.CL, cs.AI, cs.LG · Submitted 2026-05-27 · Read on arXiv

cs.CL, cs.AI, cs.LG

Submitted: 2026-05-27

Updated: 2026-08-26

Code: https://github.com/cvs-health/uqlm

Terminology

• Benchmark — 37× in this paper · explained in 2 episodes
• Functional Entropy — 26× in this paper
• Calibration — 24× in this paper · explained in 7 episodes
• Clustering — 23× in this paper · explained in 2 episodes
• Uncertainty Quantification — 21× in this paper · explained in 8 episodes
• Accuracy — 20× in this paper · explained in 1 episode
• Hallucination — 20× in this paper · explained in 3 episodes
• Consistency — 18× in this paper · explained in 3 episodes
• Functional Sets Confidence — 11× in this paper
• Functional Equivalence Rate — 10× in this paper
• Verbalized Confidence — 7× in this paper
• Cosine Similarity — 6× in this paper · explained in 3 episodes
• Embeddings — 6× in this paper · explained in 2 episodes
• Hallucination Detection — 6× in this paper · explained in 2 episodes
• Probability Margin — 6× in this paper
• Minimum Token Probability — 4× in this paper
• Text-to-SQL — 4× in this paper · explained in 3 episodes
• Ablation — 3× in this paper · explained in 1 episode
• Adaptation — 3× in this paper · explained in 1 episode
• Adaptive Decoding — 3× in this paper · explained in 1 episode
• Natural Language Inference (NLI) — 3× in this paper · explained in 2 episodes
• Aggregation — 2× in this paper · explained in 1 episode
• Benchmarking — 2× in this paper · explained in 3 episodes
• LLM judge — 2× in this paper · explained in 2 episodes
• Length-Normalized Sequence Probability — 2× in this paper
• Normalized Cosine Similarity — 2× in this paper
• Pairwise Comparisons — 2× in this paper · explained in 2 episodes
• Verification — 2× in this paper · explained in 2 episodes
• Adam — 1× in this paper · explained in 2 episodes
• Brier Score and Expected Calibration Error — 1× in this paper
• Domain Adaptation — 1× in this paper · explained in 5 episodes
• Failure Modes — 1× in this paper · explained in 3 episodes
• Large Language Models (LLMs) — 1× in this paper · explained in 6 episodes
• Monte Carlo Sequence Probability — 1× in this paper
• Tokenization — 1× in this paper · explained in 2 episodes

Sources

• A Systematic Literature Review of Code Hallucinations in LLMs: Characterization, Mitigation Methods, Challenges, and Future Directions for Reliable AI
• AdaDec: A Uncertainty-Guided Lookahead Decoding Framework for LLM-Based Code Generation
• MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generation
• Quantifying Uncertainty in Answers from any Language Model and Enhancing their Trustworthiness
• Selectively Answering Ambiguous Questions
• RED-CT: A Systems Design Methodology for Using LLM-labeled Data to Train and Deploy Edge Classifiers for Computational Social Science
• DeBERTa: Decoding-enhanced BERT with Disentangled Attention
• Internal Representations as Indicators of Hallucinations in Agent Tool Selection
• A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions
• LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
• Collu-Bench: A Benchmark for Predicting Language Model Hallucinations in Code
• Language Models (Mostly) Know What They Know
• Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation
• Generating with Confidence: Uncertainty Quantification for Black-box Large Language Models
• Beyond Functional Correctness: Exploring Hallucinations in LLM-Generated Code
• Uncertainty Estimation in Autoregressive Structured Prediction
• SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models
• Semantic Density: Uncertainty Quantification for Large Language Models through Confidence Measurement in Semantic Space
• CodeBLEU: a Method for Automatic Evaluation of Code Synthesis
• EAGer: Entropy-Aware GEneRation for Adaptive Inference-Time Scaling

Related papers

• Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
• Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
• Subliminal Steering: Stronger Encoding of Hidden Signals
• MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• Untangling the Mechanisms of Misleading Context in Medical Question Answering