A rubric landscape for evaluating clinical reasoning in large language models: what exists, what is missing, and what needs to be combined
cs.CL, cs.AI
Submitted: 2026-10-01
Updated: 2026-10-01
Terminology
Sources
- Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting
- ER-Reason: A Benchmark Dataset for LLM Clinical Reasoning in the Emergency Room
- OLAPH: Improving Factuality in Biomedical Long-form Question Answering
- Possible or Definite? A Benchmark for Evaluating Diagnostic Uncertainty Preservation in Clinical Text
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
- ExpertLongBench: Benchmarking Language Models on Expert-Level Long-Form Generation Tasks with Structured Checklists
- Beyond Precision: Importance-Aware Recall for Factuality Evaluation in Long-Form LLM Generation
- Two-Level Meta-Rubrics for Evaluating Open-Ended Generation: GAMUT, a Benchmark for Factual Completeness
- All Claims Are Equal, but Some Claims Are More Equal Than Others: Importance-Sensitive Factuality Evaluation of LLM Generations
- Autorubric: A Unifying Framework for Rubric-Based LLM Evaluation on Non-Verifiable Tasks
- A rubric-based controlled comparison of frontier language models on expert-authored clinical reasoning tasks
- Chain-of-Thought Reasoning In The Wild Is Not Always Faithful
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering