Does CoT-Pass@k Really Check the CoT? A Multilingual Mathematical Audit

arXiv:2609.32622 · cs.CL · Submitted 2026-09-26 · Read on arXiv

cs.CL

Submitted: 2026-09-26

Updated: 2026-09-26

Code: https://github.com/ttasalti/evalhub

Terminology

• Benchmark — 114× in this paper · explained in 2 episodes
• Thinking mode — 15× in this paper · explained in 2 episodes
• Verification — 15× in this paper · explained in 2 episodes
• Accuracy — 6× in this paper · explained in 1 episode
• Verifier — 6× in this paper · explained in 2 episodes
• Reinforcement Learning — 5× in this paper · explained in 4 episodes
• Consistency — 4× in this paper · explained in 3 episodes
• Verifiable Rewards — 4× in this paper · explained in 2 episodes
• Benchmarking — 2× in this paper · explained in 3 episodes
• LLM judge — 2× in this paper · explained in 2 episodes
• Bayesian Framework — 1× in this paper · explained in 3 episodes
• Chain-of-Thought Prompting — 1× in this paper · explained in 2 episodes
• Core — 1× in this paper · explained in 2 episodes
• Error Taxonomy — 1× in this paper · explained in 2 episodes
• Failure Modes — 1× in this paper · explained in 3 episodes
• Generalizability — 1× in this paper · explained in 6 episodes
• LLM-as-a-Judge — 1× in this paper · explained in 3 episodes
• Low-Resource Languages — 1× in this paper · explained in 4 episodes
• Test-time Scaling — 1× in this paper · explained in 2 episodes
• Trace — 1× in this paper · explained in 2 episodes
• Twelfth International Conference on Learning Representations — 1× in this paper

Sources

• Measuring what Matters: Construct Validity in Large Language Model Benchmarks
• Large Language Monkeys: Scaling Inference Compute with Repeated Sampling
• Evaluating Large Language Models Trained on Code
• DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
• DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
• Beyond Pass@k: Breadth-Depth Metrics for Reasoning Boundaries
• Cetvel: A Unified Benchmark for Evaluating Language Understanding, Generation and Cultural Capacity of LLMs for Turkish
• The Last Word Often Wins: A Format Confound in Chain-of-Thought Corruption Studies
• Gemma 4 Technical Report
• Don't Pass@k: A Bayesian Framework for Large Language Model Evaluation
• Can Large Language Models Detect Errors in Long Chain-of-Thought Reasoning?
• A Chain-of-Thought Is as Strong as Its Weakest Link: A Benchmark for Verifiers of Reasoning Chains
• Measuring Faithfulness in Chain-of-Thought Reasoning
• Let's Verify Step by Step
• Are Your LLMs Capable of Stable Reasoning?
• Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations
• C2-Faith: Benchmarking LLM Judges for Causal and Coverage Faithfulness in Chain-of-Thought Reasoning
• Qwen2.5 Technical Report
• Qwen3 Technical Report
• Linguistic Generalizability of Test-Time Scaling in Mathematical Reasoning

Related papers

• Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
• Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
• Subliminal Steering: Stronger Encoding of Hidden Signals
• MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• Untangling the Mechanisms of Misleading Context in Medical Question Answering