Speculative Evaluation of Stochastic LLMs
stat.ML, cs.AI, cs.LG
Submitted: 2026-09-23
Updated: 2026-09-23
Code: https://github.com/ShenQianli/SpecEval
Terminology
Sources
- Towards Reproducible LLM Evaluation: Quantifying Uncertainty in LLM Benchmark Scores
- Multiple-Prediction-Powered Inference
- Beyond Benchmarks: MathArena as an Evaluation Platform for Mathematics with LLMs
- Active Measurement: Efficient Estimation at Scale
- Measuring Massive Multitask Language Understanding
- Measuring Mathematical Problem Solving With the MATH Dataset
- Learning as Reasoning Unfolds: Progressive Rollout Allocation for Efficient Reinforcement Learning
- Instance-Optimal Estimation with Multiple LLM Judges on a Budget
- Active Testing of Large Language Models via Approximate Neyman Allocation
- TS-Neyman: Posterior Sampling for Adaptive Stratified Estimation
- Humanity's Last Exam
- tinyBenchmarks: evaluating LLMs with fewer examples
- Qwen2.5 Technical Report
- GPQA: A Graduate-Level Google-Proof Q&A Benchmark
- A Self-Improving Coding Agent
- LLM-as-Judge on a Budget
- SuperGPQA: Scaling LLM Evaluation across 285 Graduate Disciplines
- Efficient Evaluation of LLM Performance with Statistical Guarantees
Related papers
- Behavior of prediction performance metrics with rare events
- Optimal Estimation of Generic Dynamics by Path-Dependent Neural Jump ODEs
- A Posterior-Dynamics Framework for Imaging Inverse Problems with Pretrained Diffusion Priors
- One Permutation Is All You Need: Fast, Deterministic Feature Importance and Model Stress-Testing
- Online Conformal Prediction for Non-Exchangeable Panel Data
- Deep Time-Series Forecasting in 10 Years: A Survey