Experts Rise Where LLMs Disagree: Using Cross-Model Disagreement to Target Expert Effort in LLM Codebook Revision for Large-Scale Annotation

arXiv:2609.26926 · cs.CL, cs.AI, cs.HC, cs.LG · Submitted 2026-09-22 · Read on arXiv

cs.CL, cs.AI, cs.HC, cs.LG

Submitted: 2026-09-22

Updated: 2026-09-22

Terminology

• Accuracy — 44× in this paper · explained in 1 episode
• Rationale Labeling — 34× in this paper
• Question Answering — 28× in this paper
• Verification — 19× in this paper · explained in 2 episodes
• Aggregation — 14× in this paper · explained in 1 episode
• Codebook Verifying — 10× in this paper
• Core — 5× in this paper · explained in 2 episodes
• Persona Prompting — 3× in this paper · explained in 2 episodes
• Commentary — 2× in this paper · explained in 2 episodes
• Large Language Models (LLMs) — 2× in this paper · explained in 6 episodes
• Twelfth International Conference on Learning Representations — 2× in this paper
• Abstraction — 1× in this paper · explained in 1 episode
• Adaptation — 1× in this paper · explained in 1 episode
• Alignment — 1× in this paper · explained in 1 episode
• Benchmark — 1× in this paper · explained in 2 episodes
• F1 Score — 1× in this paper · explained in 3 episodes
• Fidelity — 1× in this paper · explained in 2 episodes
• Generative AI — 1× in this paper · explained in 2 episodes
• Prompt Engineering — 1× in this paper · explained in 4 episodes

Sources

• DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines
• ChatGPT-4 Outperforms Experts and Crowd Workers in Annotating Political Twitter Messages with Zero-Shot Learning
• Learning from Crowds with Crowd-Kit
• Large Language Models as Optimizers
• Large Language Models Are Human-Level Prompt Engineers
• Tutor Move Taxonomy: A Theory-Aligned Framework for Analyzing Instructional Moves in Tutoring

Related papers

• Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
• Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
• Subliminal Steering: Stronger Encoding of Hidden Signals
• MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• Untangling the Mechanisms of Misleading Context in Medical Question Answering