2608.07038-Beyond Text Matching: Towards Reference-Free Evaluation for Human-Oriented Binary Reverse Engineering

Introduction to the show: ident: Paper Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "Beyond Text Matching: Towards Reference-Free Evaluation for Human-Oriented Binary Reverse Engineering".

Jane: The paper was written by Xiuwei Shang, Li Hu, Xiao Jiang, Jieke Shi, Junda He et al. from University of Science and Technology of China and Singapore Management University and University of Alberta and Alberta Machine Intelligence Institute and Anhui Province Key Laboratory of Digital Security.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Paper summary: Tom: A new paper landed from the binary reverse engineering world, and we're genuinely buzzing about it.

Jane: It tackles a nasty problem nobody has properly cracked yet — how do you score tools that make decompiled binary code readable?

Lu: Exactly. Function name recovery, code summarization, decompilation optimization. All these tasks produce outputs that are semantically right but worded totally differently.

Meng: So the old-school metrics like BLEU or CodeBLEU just fail. They compare text shape, not meaning. And human experts are too slow and too expensive.

Tom: So the authors went in a different direction. They built BinJudgeBench — an expert-annotated benchmark with 1,233 samples across those three tasks.

Jane: Then they ran 9 different LLMs as judges, testing how well their scores matched human experts. Average correlation, 63.20 percent. Traditional metrics only hit 35.04 percent.

Lalam: That gap is huge. It says LLMs genuinely get the semantics of binary analysis in a way that text matching can't.

Lu: But here's the twist. They found no single judge configuration works best for every task and every sample.

Meng: Right, the optimal setup keeps changing. Small models need chain-of-thought, big models do better with few-shot examples. Temperature matters too.

Jane: So they built BinJudge, a lightweight router that picks the best configuration per sample.

Tom: That boosts human correlation by 4.5 to 24.7 percent. And it slashes API costs to as little as six percent of what static setups pay.

Jane: Six percent! When's the last time you heard a cost reduction like that?

Tom: I think we need to unpack how they built the benchmark before we celebrate the router.

Jane: Agreed. Let's start from the top.

Page 1 of the paper: Tom: So we mentioned the problem, but page one really lays out why reference-based evaluation is so fragile.

Jane: The core issue is that these HOBRE tasks — human-oriented binary reverse engineering — happen precisely when you have no source code.

Lu: In malware analysis or firmware forensics, the source is gone. It's just stripped binaries and decompiled pseudocode.

Meng: So if you want to score a generated function name or summary, you can't compare against ground truth because there is no ground truth.

Tom: The paper makes a sharp observation about this. Even in open-source projects, only a small fraction of functions have usable developer comments, and their quality varies wildly.

Jane: And they make an even deeper point. After compilation and stripping, the binary and the source are semantically mismatched. Forcing source code to be the reference is questionable.

Lu: That destroys the foundation of traditional methods. BLEU, BERTScore, all of them depend on high-quality reference text that just isn't available in the real world.

Meng: And what about the test-based approach, the re-executability rate used in some decompilation work? You need unit tests and a runtime environment.

Tom: The paper calls out how that limits you to isolated toy functions. Real-world binaries have dependencies that are nearly impossible to simulate.

Jane: So every existing evaluator has a fundamental problem: they can't assess readability, and they can't handle "in-the-wild" binaries.

Lalam: That's where LLM-as-a-Judge enters. LLMs have binary comprehension skills, they're trained to align with human preferences, and they don't need references.

Tom: The paper claims LLMs can verify generated artifacts against the code's internal logic directly.

Jane: But even with that potential, nobody had systematically tested whether these judges actually agree with human experts.

Meng: The authors stepped in to fill that gap for the first time. But to test judges, you first need a reliable "gold standard" of human judgment.

Tom: Which brings us to the benchmark construction.

Jane: I want to hear about the scale of what they created.

Page 3 of the paper: Tom: Building that benchmark took serious engineering effort.

Jane: They went to the GNU repository and selected 51 real-world projects — actual code people rely on.

Meng: Then they compiled those projects into 24 distinct binary variants using GCC 8.2.0.

Lu: T h a t's a spread of six target architectures — ARM_32, ARM_64, X86, X64, MIPS_32, MIPS_64 — and four optimization levels from -O0 to -O3.

Tom: Each and every binary exists in two versions: one stripped of symbols and one with debugging information intact.

Jane: Then IDA Pro decompiles both versions. And here's the clever part: function boundaries stay consistent even after stripping, so they can align the original symbols to the stripped pseudocode.

Meng: That gives you a clean pairing of stripped code and its true meaning. Oh, and they also parse the source with srcML to pull developer comments.

Tom: In total, the pipeline produced 346,596 function-level samples.

Jane: Three hundred forty-six thousand. And each sample carries the stripped pseudocode, the original pseudocode, the source function, and the source name.

Lu: Then they needed responses to evaluate. They picked 8 models per task — binary-specific models like SymLM and HexT5, general-purpose LLMs like GPT-4o, and even binary-domain LLMs like ReCopilot.

Meng: All these models generated candidate outputs for every sample. That multiplies out to over 2.7 million candidate responses per task.

Tom: You can't have humans annotate 2.7 million anything. That's where the sampling strategy comes in.

Jane: They computed the sample size for a 95 percent confidence level with a 5 percent confidence interval. That's 385 samples.

Meng: But they also wanted coverage across all 192 possible combinations of model, architecture, and optimization level.

Lu: The initial random sample covered 166 of those combinations. A targeted supplement of 26 more samples rounded it out.

Tom: So 411 samples per task. Times three tasks. That becomes the benchmark.

Jane: Let's talk about those human annotators then. They had to make judgment calls on ambiguous, ugly, stripped code.

Tom: And judging that much of it took two weeks per expert.

Meng: That's why the scoring protocol had to be solid before they began.

Page 5 of the paper: Jane: The human scoring protocol reads like a carefully designed rubric, and I love that it has both shared and task-specific dimensions.

Tom: Every annotator sees the stripped pseudocode, the candidate output, and the source code for reference. But the model's identity stays anonymous to prevent bias.

Lu: The two shared dimensions are semantic correctness and analyst utility. Does the output reflect what the code does, and does it reduce cognitive load?

Meng: Then each task adds its own twist. Function names get judged on distinctiveness and naturalness. Summaries on information coverage and density. Optimized pseudocode on idiomization and faithfulness.

Tom: Scoring runs from 1 to 5. A 5 requires outputs that are accurate and "significantly reduce cognitive burden."

Jane: The three annotators are authors themselves, each with over three years of binary reverse engineering experience.

Lu: They calibrated on 10 samples per task before going independent. Then measured agreement with ordinal Krippendorff's Alpha — got 0.7996 for FNR, 0.7077 for BCS, 0.6619 for DO.

Meng: Those are within the "satisfactory agreement" range, though not flawless. Which is exactly why they added a resolution step.

Tom: For samples where scores spanned too wide — range two or more — the experts held review meetings. 73 for FNR, 105 for BCS, 76 for DO.

Jane: And for minor disagreements, the mode becomes the final score. That's a sound protocol.

Lu: What really jumps out is the score distribution. Function name recovery averaged just 2.42, the lowest of the three. Lots of bad names out there.

Meng: Binary code summarization was the most balanced at 2.81 on average. Decompilation optimization sat at 2.66, with a notable scarcity of high scores.

Tom: So generated function names are generally pretty weak, while summaries are closer to human-quality descriptions.

Jane: That aligns with intuition. Picking a concise, distinctive name is genuinely hard for models.

Tom: But now we have a stable ground-truth benchmark. The next question is whether LLM judges can match those human scores.

Meng: And honestly, the empirical results are the part I've been waiting for.

Page 7 of the paper: Tom: The big comparison table is something else when you look at the raw numbers.

Jane: LLM judges average 63.20 percent correlation across all three tasks. Traditional metrics scrape by at 35.04 percent.

Meng: The best traditional metric, METEOR, only hits 49.97 percent on function name recovery. The worst LLM, Phi-4, still manages 48.64.

Lu: So even the weakest LLM judge basically ties the strongest old-school metric.

Tom: Actually there's a fun nuance. ChrF++, a character-level metric, hits 60.37 percent on FNR, which actually beats Phi-4's 56.50. But that's the exception, not the rule.

Jane: The gap is biggest on binary code summarization. LLMs are 35 percent better on Kendall's tau there.

Meng: That's because summarization has infinitely many valid wordings. An n-gram overlap metric just can't see that two totally different sentences mean the same thing.

Tom: And on decompilation optimization, LLMs can spot what they call "superficially plausible but logically flawed" refactorings. Traditional metrics completely miss that.

Jane: I also want to talk about the self-evaluation bias test. They checked whether GPT-4o, DeepSeek-V3.2, or Qwen3-Coder favored their own outputs when judging.

Lu: And the result? No statistically significant bias. These models stayed objective even when evaluating their own generation style.

Meng: But they did find what they call the "fluency trap." 12.08 percent of samples — 149 out of 1,233 — had substantial disagreement between humans and LLMs.

Tom: Manual analysis broke those hard samples into three patterns. 34 percent lack semantic anchors, meaning no strings or API calls to latch onto. 14 percent lack context from callers and callees. 12 percent reward fluent but unsupported output.

Jane: In that fluent-looking but semantically unsupported pool, LLM judges gave scores above 3 in 59 percent of cases, versus 46.7 percent for unsupported samples broadly.

Lu: So there's a real but bounded bias toward fluent-sounding answers. The paper says it's within a "controllable and acceptable margin."

Tom: Which is reassuring. But this is average behavior across all LLMs — what about the specific configuration of the judge?

Jane: That's what RQ2 digs into. Backbone LLM, prompting strategy, temperature, and the cost tradeoffs in between.

Page 9 of the paper: Tom: The configuration study is where things get really interesting.

Jane: They tested nine LLMs, three prompting strategies, and three temperatures. That's 81 possible judge configurations in total.

Meng: The first big insight: prompting strategy effects flip with model scale.

Lu: For ultra-large models like GPT-4o or Gemini-2.5-Flash, few-shot learning works best. The examples anchor them on the right scoring patterns.

Tom: Gemini-2.5-Flash hits 69.02 percent on function name recovery with few-shot at temperature 0.1. That beats its zero-shot score of 66.56 and its chain-of-thought score of 57.23.

Jane: But for smaller models like Phi-4 and Codestral? Few-shot actually hurts them. They get anchored to specific patterns and lose generalization.

Meng: Chain-of-thought rescues those small models. Phi-4 peaks at 43.66 percent average correlation with CoT, and Codestral at 46.77.

Lu: So explicit reasoning compensates for weaker native reasoning. But for the big models, too much reasoning can trigger hallucinations and mess up the judgment.

Tom: Temperature mostly favors the low end. 0.1 or 0.5 keeps the evaluation scale consistent for large models.

Jane: And there's a peculiar quirk — for the small models, high temperature sometimes helps. The paper suggests randomness lets weak models escape local scoring traps.

Meng: The cost data is wild though. Claude-3.5-Sonnet gets 61.65 percent average correlation at a cost of .774. Gemini-2.5-Flash gets 61.36 percent at .131.

Tom: That's twenty-one times the cost for essentially the same performance. Cost and performance are definitely not linearly linked.

Lu: Which brings us to RQ3. If the best configuration changes across tasks, can we at least pick one static winner per task?

Tom: And the answer there is a firm no. Task-level rank correlations between configuration rankings are only moderate. BCS and DO configurations correlate at just 52.56 percent on Kendall's tau.

Jane: So a configuration that's great for summarization may flop on decompilation optimization. And cost rankings stay stable across tasks — but performance rankings flip.

Meng: The oracle gap seals it. Static best configuration maxes out at 70.31 percent correlation, while a sample-aware oracle hits 92.95 percent.

Lu: That gap is the motivation for BinJudge. You need per-sample adaptation.

Tom: Now let's talk about how they actually built that router.

Page 11 of the paper: Jane: BinJudge is built on a neat idea — instead of hoping one judge works for everything, train a tiny model to pick the right judge per sample.

Tom: It takes the stripped pseudocode and the task type as inputs. A frozen UniXcoder encoder extracts the code features.

Lu: Then a learned task embedding gets concatenated. That gives it cross-task awareness, so it knows function name recovery is different from summarization.

Meng: The fused features go through a stacked MLP, which outputs a preference distribution over all 81 judge configurations.

Jane: And here's the clever training trick. They don't just teach it which configuration wins. They align the full probability distribution using KL divergence.

Tom: The utility function combines two things: the squared error between the judge's score and human score, plus a penalty proportional to API cost. Log-scaled to normalize.

Lu: With lambda set to 0.1, the router explicitly trades off between accuracy and cost.

Meng: Then they tested BinJudge with five-fold cross-validation on BinJudgeBench. The results are genuinely impressive.

Tom: BinJudge beats every static configuration on all three tasks. Kendall's tau jumps by 4.5 to 12.0 percent on function name recovery, 6.4 to 20.4 percent on summarization, and 5.8 to 24.7 percent on decompilation optimization.

Jane: Cost-wise it's stunning. Compared to random configuration selection, BinJudge reduces API expenditure to 14 percent, 15 percent, and 25 percent per task.

Lu: Against static best configurations, it's down to 0.06 times to 0.84 times the cost.

Meng: Even the oracle configuration costs two to three times more than BinJudge. So the router is closing the gap while staying cheap.

Tom: They also tried fine-tuning judge models directly — UniXcoder and Qwen2.5-Coder-7B. Both significantly underperformed BinJudge.

Jane: That's a profound finding. Under the same annotation budget, a specialized evaluator that learns to score generalizes poorly. But a router that leverages diverse LLM strengths wins.

Lalam: And those results hold up against the discussion section's caveats too. The disagreements between human and machine are well characterized.

Meng: The authors acknowledge LLM judges aren't full replacements for humans. But they're a clearly superior option to traditional metrics.

Tom: So where does that leave the field?

Jane: Let's sum it all up.

Conclusion: Tom: This paper essentially redraws the map for evaluating binary reverse engineering tools.

Jane: It gives us a benchmark, a systematic empirical study, and a router — three contributions in one package.

Lu: The benchmark, BinJudgeBench, gives future researchers a solid ground truth with 1,233 samples spanning three tasks.

Meng: The empirical study proves LLM-as-a-Judge is the way forward, hitting 63.20 percent correlation against human experts versus 35.04 percent for traditional metrics.

Tom: And BinJudge itself shows that you don't need one almighty judge. You need the right judge for each sample.

Jane: The cost reduction is the part I keep returning to. A router that costs 0.06 times to 0.84 times static best configurations while improving correlation.

Lu: That combination of accuracy and economy is what makes scalable evaluation realistic for real security work.

Meng: Think about what this unlocks. Malware analysts, firmware researchers, vulnerability hunters — they all can trust automated scoring of decompiler outputs.

Tom: And the "fluency trap" finding is a healthy reminder that LLM judges still have blind spots.

Jane: But the paper doesn't hand-wave that away. They characterize the failure modes, quantify them, and build the router to navigate around them.

Meng: The future work implied here is rich. More tasks, more languages, more architectures. Maybe specialized routers for specific malware families.

Tom: And this is exactly the kind of incremental but meaningful progress that moves the field forward.

Jane: A cleaner path from stripped binary to human understanding, with a reliable way to measure success along the way.

Tom: Until next time.

Jane: Thanks for listening.

Lu: And keep decompiling.