From Detection to Decision: Calibration and Deferral for Prompt-Injection Detection

Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.

Nadia: Today's paper: "From Detection to Decision".

Elias: The gist The results show that training for calibrated outputs alone does not ensure reliable security decisions under distribution shift.

Nadia: First, who's behind it and why it matters.

Title and authors: Nadia: Now we're looking at the specific title and who wrote this paper: "From Detection to Decision: Calibration and Deferral for Prompt-Injection Detection." It really zeroes in on moving beyond just having a detector and focusing on making actual security decisions like allow, block, or defer.

Elias: The authors are Qianlong Lan, Anuj Kaul, Vinothini Pandurangan, and Indranil Sanyal from eBay Inc. They set up this comparison between Jev, the Prompt Guard two classifiers at twenty-two million and eighty-six million parameters, and an LLM judge across forty-seven thousand six hundred eleven held-out examples covering various attack types.

The paper's summary: Nadia: The paper summarizes their findings by showing that while some models have great discrimination, like Jev with an AUROC of zero point eight seven one, their native probabilities aren't always better calibrated than the temperature-scaled classifier scores <ref:2610.20907#pg1>.

Elias: They give us specific numbers for this calibration check: Jev gets ECE zero point five four one, Brier zero point four four zero, and NLL one point four three two when compared to calibrated Prompt Guard two at eighty-six million parameters which got ECE zero point five seven one, Brier zero point four three nine, and NLL one point one zero four for instance <ref:2610.20907#pg2>.

Priya: That difference in calibration metrics is significant because it suggests that a simple training goal isn't enough to ensure the model's output is trustworthy when we deploy it in a changing environment.

Nadia: Right, Priya. And they also looked at the decision policy behavior, seeing how uncertainty at the gate translates into those allow block defer decisions when you have that frozen imperfect reviewer.

Elias: They found that to hit a very strict false negative rate of less than one percent, you needed over ninety-one point eight percent of inputs to be escalated to that reviewer for Jev native, and similar high rates for the Prompt Guard variants <ref:2610.20907#pg2>.

Priya: So even with those high escalation rates, the false positive rate at these crossing points was still quite high, hovering around forty to forty-five percent. That shows that just hitting a low false negative rate doesn't automatically mean you have a good overall system policy.

The paper's improvements: Nadia: One of the main improvements suggested by the authors is moving toward a risk-aware decision pipeline where we realize that a probability of zero point nine only matters if high scores reliably indicate attacks and stay meaningful even when attacks or benign prompts change.

Elias: They also argue that for routing decisions, you can use native models like Jev because their native probabilities aren't consistently better calibrated than temperature-scaled classifier scores, as we saw with the numbers comparing Jev to Prompt Guard two at eighty-six million parameters <ref:2610.20907#pg1>.

Priya: And they suggest that if we want to improve the reliability of those LLM-based security decisions, we should use temperature scaling on the classifier scores because it substantially improves metrics like ECE and NLL without changing how good the model is at discrimination.

Nadia: So, one way to fix things is to apply that specific calibration adjustment—temperature scaling—to enhance the utility of those classifier scores for deployment.

Elias: And they also propose establishing operational policies based on empirical escalation rates, which means finding the smallest escalation rate on the deferral curve where you hit that target false negative rate of less than one percent <ref:2610.20907#pg2>.

Priya: But they also stress a limitation here: calibration alone doesn't determine a useful decision policy, so we still need to evaluate reviewer behavior and the security and utility objectives of the deployed system together.

Conclusion: Nadia: To wrap up, the authors conclude that discrimination, calibration, reviewer behavior, and policy utility all need to be evaluated as parts of one single decision pipeline when dealing with prompt injection detection under distribution shift.

Elias: They reiterate that pooled metrics can obscure composition effects because the confirmatory cohort is heavily weighted by attacks from BIPIA <ref:2610.20907#pg3>, and the quality of the reviewer itself limits how much benefit deferral can actually achieve.

Priya: So for us, it means we need to be careful not to just look at one metric in isolation; we have to look at how those metrics work together under real operational constraints.

Nadia: They also point out that the comparison itself differs because Prompt Guard two is temperature-scaled on target calibration data while Jev uses its native probability, which is another factor we have to consider when comparing these systems <ref:2610.20907#pg1>.

Elias: And they flag a practical limitation: the cost model used in their evaluation is just illustrative, so we should use application-specific costs and review budgets instead of that general model.

Priya: Ultimately, this work on "From Detection to Decision: Calibration and Deferral for Prompt-Injection Detection" shows us that while detection models are getting better at spotting injections, the real challenge remains making a dependable decision when those attacks evolve in ways we haven't seen before.