Benchmarking Code LLM-as-a-Judge via Constrained Perturbation and Calibration

summary

The gist

The gist The proposed AXIOM framework reframes code evaluation as measuring refinement effort, enabling the creation of large-scale, balanced benchmarks to rigorously test existing LLM-as-a-Judge

In short

The AXIOM framework reframes code evaluation as measuring refinement effort to create balanced benchmarks for LLM-as-a-Judge metrics. It uses rule-guided perturbation to synthesize programs with controlled quality and a multisource calibration stage to streamline human annotation. This allows researchers to rigorously test existing metrics against a diverse, high-quality dataset.

Key concepts

Rule-Guided Perturbation: This stage uses LLMs and predefined rules to systematically modify existing code. It creates programs with specific target quality scores (e.g., 1 to 5 points) by applying sequences of controlled changes, ranging from minor flaws to complete context disruption.
Multisource Quality Calibration: Instead of complex logic, human annotators review the differences between a program and its original high-quality version. They use multiple data sources—like static analysis reports and LLM quality assessments—to calibrate the final score, simplifying subjective judgment.
Refinement Effort as Score: AXIOM changes how scores are viewed: instead of judging the final code directly, the score represents how much effort (refinement) is needed to get there. This metric is more objective and helps control program quality distribution across a benchmark.

Terminology used across episodes

Reinforcement Learning (RL) — explained in 25 episodes
Sample Complexity — explained in 19 episodes
Federated Learning — explained in 17 episodes
Retrieval-Augmented Generation (RAG) — explained in 17 episodes
Quantum Fisher Information (QFI) — explained in 16 episodes
Interpretability — explained in 15 episodes

This episode discusses

The paper

Benchmarking Code LLM-as-a-Judge via Constrained Perturbation and Calibration · Read on arXiv

Ruiqi Wang, Xinchen Wang, Cuiyun Gao, Chun Yong Chong, Xin Xia, Qing Liao

Harbin Institute of Technology Institute of Technology Shenzhen University Monash University Malaysia Zhejiang University

Code from LLM-assisted development may contain defects like API hallucinations, motivating the development of code evaluation metrics to assess such code before deployment. As conventional rule-based metrics merely score code based on similarity to reference programs, and execution-based metrics require expensive manual preparation for tests, researchers have developed LLM-as-a-judge metrics for both behavioral and nonbehavioral code evaluation, and curated multiple benchmarks to validate their effectiveness. However, such benchmarks suffer from critical limitations: Some feature coarse-grained binary labels of passing/failing unit tests, unable to distinguish between fundamental and minor flaws; others utilize fine-grained criteria with manually annotated scores as ground-truth, but such criteria are often vaguely defined, introducing subjectivity and unreliability in human scores. Furthermore, they often directly prompt LLMs to generate code for evaluation, which may lead to unbalanced code quality distributions, failing to test the metrics' discriminative capability across varying quality levels. To curate a diverse benchmark with a balanced program quality distribution and streamline manual annotation, we propose AXIOM, a novel perturbation-based framework. It reframes program scores as the refinement effort needed for deployment, with two stages: (1) Quality-constrained semantic perturbation, prompting LLMs to apply sequences of predefined perturbation rules to existing programs, precisely controlling each program's target quality to achieve a balanced distribution. (2) Hybrid quality calibration, selecting semantically distinct programs for benchmark diversity, and performing streamlined manual annotation with multisource context. With AXIOM, we curate AXIOMBENCH, a large-scale multilingual benchmark with 1,962 code evaluation problems in a more balanced score distribution...

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Benchmarking Code LLM-as-a-Judge via Constrained Perturbation and Calibration".

Jane: The gist The proposed AXIOM framework reframes code evaluation as measuring refinement effort, enabling the creation of large-scale, balanced benchmarks to rigorously test existing LLM-as-a-Judge metrics.

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So, we're looking at this paper today, "Benchmarking Code LLM-as-a-Judge via Constrained Perturbation and Calibration." It sounds like they're trying to fix some big problems with how we evaluate code written by large language models.

Jane: Yeah, the title itself tells you a lot about what they’re doing here. They're using some kind of constrained perturbation and calibration to test those LLM-as-a-Judge metrics, which have been around for a while.

Lu: What this paper is setting up is a way to get past the limitations of existing code evaluation benchmarks that are often small and unbalanced, or have unreliable scores. They're aiming for something much bigger and more reliable for meta-evaluating those LLM-as-a-judge metrics themselves, which is a pretty ambitious goal.

Meng: I wonder if this means they're building a benchmark that actually reflects what real software engineering looks like, instead of just surface-level similarity.

Lalam: It seems like the core idea is to move away from simple scoring and toward measuring the actual refinement effort needed to get code ready for deployment, which is a much more concrete way to look at code quality.

The paper's summary: Tom: So, AXIOM, that's what they call their framework. The main idea is reframing code evaluation not as judging the final output directly, but as measuring the hypothetical effort needed to repair programs for deployment.

Jane: That’s a big shift. Instead of just saying "this code is good," they want to assign scores based on how much work it would take to polish it up, which gives them a really fine-grained, interpretable score.

Lu: They achieve this through a two-stage process. First is rule-guided perturbation where they prompt LLMs to modify existing high-quality programs using sequences of predefined rules to control the target score precisely.

Meng: So they're not just asking the LLM to generate new code, but making it change something that already exists, and controlling the outcome with these rules. That sounds like it would be much more controllable than letting an LLM go wild on its own.

Lalam: And then there’s the second stage, multisource quality calibration where they use a mix of LLMs, static analysis reports, and execution reports to determine the final score for each perturbed program.

The paper's improvements: Tom: The paper outlines several improvements over what we have now. They propose using rule-guided perturbation to synthesize programs with predetermined target scores, which helps them create a large-scale multilingual benchmark with one thousand nine hundred sixty-two problems in a balanced score distribution <ref:2512.20159#pg1>.

Jane: That sounds like they are solving the problem of having small, unbalanced benchmarks by systematically generating programs across different quality levels. They use exemplar augmentation to get more production-ready programs and deteriorative perturbation to make realistic flaws.

Lu: They also have this multisource quality calibration step where human annotators don't do complex logic reasoning anymore. They just review the difference between the program and its parent, using all that context to calibrate the final score into a ground truth score.

Meng: That’s smart because it frees up the human annotators from having to act like deep logic experts, which is a major hurdle in these kinds of studies. They are simplifying their task to focusing on the diff and using reports from multiple sources.

Lalam: They also mention representative candidate selection, where they use metrics like normalized token edit distance and a greedy approach to pick the most diverse set of programs for final calibration, making sure the benchmark isn't just focused on one type of code.

Conclusion: Tom: So, to wrap up, AXIOM is this two-stage framework that uses perturbation to create a large-scale benchmark and then uses multisource calibration to get reliable scores for LLM-as-a-Judge metrics.

Jane: The main implication is that it shows we can synthesize code evaluation data in a way that's grounded in measurable effort, rather than just subjective judgment. It helps us meta-evaluate those judges themselves.

Lu: It’s about creating a foundation for testing those LLM-as-a-judge metrics rigorously across a much wider range of scenarios and score distributions.

Meng: From an engineering standpoint, the focus on controlling the scores via perturbation sounds like it addresses the inconsistency issues we see when we try to run these metrics at scale on real traffic.

Lalam: I think this work provides a way to systematically generate data that’s balanced and reliable enough for serious research into how good those LLM-as-a-judge metrics really are.

More episodes

← Home