GenRubric: Self-Evolving Rubric Generation for Scalable LLM Evaluation

arXiv:2608.29856 · cs.CL · Submitted 2026-08-30 · Read on arXiv

cs.CL

Submitted: 2026-08-30

Updated: 2026-08-30

Code: https://github.com/foggpoy/GenRubric

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Consistency — 15× in this paper · explained in 3 episodes
• Reinforcement Learning — 12× in this paper · explained in 4 episodes
• Accuracy — 9× in this paper · explained in 1 episode
• Benchmark — 7× in this paper · explained in 2 episodes
• Redundancy — 7× in this paper · explained in 2 episodes
• Generalization — 5× in this paper · explained in 13 episodes
• Self-consistency — 5× in this paper · explained in 2 episodes
• Aggregation — 4× in this paper · explained in 1 episode
• Ablation — 3× in this paper · explained in 1 episode
• LLM judge — 3× in this paper · explained in 2 episodes
• Alignment — 2× in this paper · explained in 1 episode
• Benchmarking — 2× in this paper · explained in 3 episodes
• Core — 2× in this paper · explained in 2 episodes
• Credit assignment — 2× in this paper · explained in 3 episodes
• LLM-as-a-Judge — 2× in this paper · explained in 3 episodes
• Ablation Studies — 1× in this paper · explained in 1 episode
• Adversarial Co-evolution — 1× in this paper · explained in 1 episode
• Distillation — 1× in this paper · explained in 2 episodes
• Fine-Tuning — 1× in this paper · explained in 5 episodes
• GRPO — 1× in this paper · explained in 6 episodes
• Instruction Tuning — 1× in this paper · explained in 2 episodes
• Large Language Models (LLMs) — 1× in this paper · explained in 6 episodes
• Robustness — 1× in this paper · explained in 5 episodes
• Transferability — 1× in this paper · explained in 4 episodes
• Verifier — 1× in this paper · explained in 2 episodes

Sources

• OpenCodeInstruct: A Large-scale Instruction Tuning Dataset for Code LLMs
• HealthBench: Evaluating Large Language Models Towards Improved Human Health
• Evaluating Large Language Models Trained on Code
• LexRubric: A Rubric-Guided Diagnostic Benchmark for Open-Ended Legal Tasks
• DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models
• EvoRubrics: Dynamic Rubrics as Rewards via Adversarial Co-Evolution for LLM Reinforcement Learning
• DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents
• LEXam: Benchmarking Legal Reasoning on 340 Law Exams
• Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains
• Measuring Mathematical Problem Solving With the MATH Dataset
• FinanceBench: A New Benchmark for Financial Question Answering
• LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
• RUBRIC-ARROW: Alternating Pointwise Rubric Reward Modeling for LLM Post-training in Non-verifiable Domains
• Prometheus: Inducing Fine-grained Evaluation Capability in Language Models
• Kimi K2.5: Visual Agentic Intelligence
• ReportBench: Evaluating Deep Research Agents via Academic Survey Tasks
• Learning Query-Specific Rubrics from Human Preferences for DeepResearch Report Generation
• Fino1: On the Transferability of Reasoning-Enhanced LLMs and Reinforcement Learning to Finance
• Proximal Policy Optimization Algorithms
• DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

Related papers

• Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
• Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
• Subliminal Steering: Stronger Encoding of Hidden Signals
• MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• Untangling the Mechanisms of Misleading Context in Medical Question Answering