Codifying the Judge: Scalable Evaluation via Program Distillation
cs.AI, cs.LG
Submitted: 2026-05-29
Updated: 2026-09-26
Project page: https://sprocketlab.github.io/PAJAMA
Terminology
Sources
- LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods
- Self-Taught Evaluators
- RubiCap: Rubric-Guided Reinforcement Learning for Dense Image Captioning
- DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research
- OpenAI GPT-5 System Card
- One Token to Fool LLM-as-a-Judge
- Can You Trust LLM Judgments? Reliability of LLM-as-a-Judge
- CARE: Confounder-Aware Aggregation for Reliable LLM Evaluation
- ScriptoriumWS: A Code Generation Assistant for Weak Supervision
- Multimodal Data Curation via Object Detection and Filter Ensembles
- Shrinking the Generation-Verification Gap with Weak Verifiers
- Evaluating Sample Utility for Efficient Data Selection by Mimicking Model Weights
- Large Language Model Routing with Benchmark Datasets
- Tryage: Real-time, intelligent Routing of User Prompts to Large Language Models
- RLHF Workflow: From Reward Modeling to Online RLHF
- GPT-4 Technical Report
- Gemma 3 Technical Report
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems
- VSAL: A Vision Solver with Adaptive Layouts for Graph Property Detection