What Makes Something Hard(er)? Explaining Question Difficulty in Natural Language

arXiv:2610.01627 · cs.CL, cs.AI · Submitted 2026-10-01 · Read on arXiv

cs.CL, cs.AI

Submitted: 2026-10-01

Updated: 2026-10-01

Code: https://github.com/pengcuix/explain-difficulty

Terminology

• Benchmark — 25× in this paper · explained in 2 episodes
• Accuracy — 12× in this paper · explained in 1 episode
• Embeddings — 9× in this paper · explained in 2 episodes
• Causal Reasoning — 7× in this paper · explained in 2 episodes
• Item Response Theory — 5× in this paper
• Adam — 4× in this paper · explained in 2 episodes
• Physical Constraints — 4× in this paper · explained in 7 episodes
• Verification — 3× in this paper · explained in 2 episodes
• Actor — 2× in this paper · explained in 3 episodes
• Foundation Model — 2× in this paper · explained in 3 episodes
• Foundation Models — 2× in this paper · explained in 7 episodes
• Verifier — 2× in this paper · explained in 2 episodes
• Ablation — 1× in this paper · explained in 1 episode
• Accountability — 1× in this paper · explained in 2 episodes
• Adaptation — 1× in this paper · explained in 1 episode
• Alignment — 1× in this paper · explained in 1 episode
• Alternation — 1× in this paper · explained in 1 episode
• Benchmarking — 1× in this paper · explained in 3 episodes
• Clustering — 1× in this paper · explained in 2 episodes
• Consistency — 1× in this paper · explained in 3 episodes
• Copula — 1× in this paper · explained in 2 episodes
• Core — 1× in this paper · explained in 2 episodes
• Cosine Similarity — 1× in this paper · explained in 3 episodes
• Decode — 1× in this paper · explained in 2 episodes
• Feature Engineering — 1× in this paper · explained in 3 episodes
• Feature Selection — 1× in this paper · explained in 4 episodes
• Fine-Tuning — 1× in this paper · explained in 5 episodes
• Generalization — 1× in this paper · explained in 13 episodes
• In-Context Learning — 1× in this paper · explained in 4 episodes
• LLM judge — 1× in this paper · explained in 2 episodes
• Regret — 1× in this paper · explained in 3 episodes
• Regularization — 1× in this paper · explained in 3 episodes
• Transformer — 1× in this paper · explained in 2 episodes

Sources

• Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
• Qwen Technical Report
• Training Verifiers to Solve Math Word Problems
• Domino: Discovering Systematic Errors with Cross-Modal Embeddings
• The Llama 3 Herd of Models
• Mistral 7B
• Gemma 3 Technical Report
• Textbooks Are All You Need II: phi-1.5 technical report
• RoBERTa: A Robustly Optimized BERT Pretraining Approach
• LLMs Encode How Difficult Problems Are
• Gemma: Open Models Based on Gemini Research and Technology
• Qwen2.5 Technical Report
• Qwen2 Technical Report
• Yi: Open Foundation Models by 01.AI
• TinyLlama: An Open-Source Small Language Model
• Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models

Related papers

• Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
• Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
• Subliminal Steering: Stronger Encoding of Hidden Signals
• MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• Untangling the Mechanisms of Misleading Context in Medical Question Answering