Does Personalization Break Trust? Open-Set Rejection Calibration Under Few-Shot On-Device Intent Adaptation

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Does Personalization Break Trust? Open-Set Rejection Calibration Under Few-Shot On-Device Intent Adaptation".

Jane: The gist: Naive few-shot LoRA personalization degrades OOS-rejection AUROC on every one of five intent classification benchmarks,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: Welcome back. We're talking about a new paper called "Does Personalization Break Trust? Open-Set Rejection Calibration Under Few-Shot On-Device Intent Adaptation". This research is looking at how making an AI assistant learn new commands on the device affects its ability to correctly reject things it doesn't understand.

Jane: That sounds really important because, as we know, when you use an always-on virtual assistant, if it gets confused and gives a wrong answer or just makes up stuff, that breaks user trust. This paper focuses on that trust aspect specifically.

Lu: The core idea is testing whether the personalization step—using things like LoRA to learn new user commands via few-shot fine-tuning—actually degrades the out-of-scope rejection quality, which is a critical property for these assistants six <ref:2610.20652#pg2>.

Meng: So, what's the main claim here? Is it saying that if we personalize an assistant on a device to learn new things, it becomes worse at saying "I don't know" when faced with something brand new?

Tom: Exactly. The paper shows that naive few-shot LoRA personalization degrades the OOS rejection AUROC, which is the area under the ROC curve, across every one of five public intent-classification benchmarks <ref:2610.20652#pg1>.

Jane: That's a pretty broad claim. How big is that drop they are seeing? Is it a tiny dip or something significant?

Tom: They report a mean drop between zero point zero five five and zero point two five eight across those five datasets with five seeds for each configuration <ref:2610.20652#pg1>. That's a measurable hit, but they also say that the accuracy on the in-scope commands is perfectly preserved <ref:2610.20652#pg1>.

Lu: What's interesting is how they characterize that degradation across those different datasets. They found that it’s not uniform; for instance, SNIPS shows a substantially larger drop of zero point two five eight AUROC compared to the other four datasets which show drops between zero point zero five five and zero point zero nine seven <ref:2610.20652#pg2>.

Meng: So, it’s not a one-size-fits-all problem for personalization; some benchmarks are hit much harder than others? That matters when you're trying to deploy something everywhere.

Jane: And they did a scorer-invariant check that showed part of this drop is just an artifact from how the standard confidence scorer miscalibrates itself after adaptation <ref:2610.20652#pg1>.

Tom: But they found a smaller, genuine effect remains under any scorer <ref:2610.20652#pg1>, and they traced that genuine effect to a measurable shrinkage of the embedding margin around known intent regions with a pooled Spearman rho of zero point six one <ref:2610.20652#pg1>.

Lu: So, the mechanism behind the drop is related to this shrinkage in the embedding margin between known intent class centroids and true out-of-scope points <ref:2610.20652#pg3>.

Jane: That sounds like a geometric issue in the feature space. They suggest this shrinkage correlates with a larger OOS quality drop, which is supported by a pooled linear fit showing that larger margin shrinkage is associated with a larger OOS quality drop with Spearman rho of zero point six one <ref:2610.20652#pg3>.

Paper summary: Tom: The real focus then shifted to finding a fix for this problem, and they propose replacing the evaluation-time rejection score with a Mahalanobis distance score instead of just using the softmax confidence <ref:2610.20652#pg3>.

Meng: What does that mean in practical terms? How do we switch from relying on confidence scores to something like Mahalanobis distance for rejecting unknown inputs? Is it hard to implement?

Jane: The proposed fix involves fitting a tied-covariance Gaussian over the embeddings—specifically, the postpersonalization base plus the new training pool’s penultimate-layer embeddings, using a shrinkage-regularized covariance matrix and one mean per known class <ref:2610.20652#pg3>.

Tom: And the payoff is that this fix recovers most of the lost OOS quality on three out of five datasets with no accuracy cost and no extra training required <ref:2610.20652#pg3>.

Lu: They also showed this fix holds up robustly, holding across different few-shot budgets, LoRA ranks, and even a second base encoder <ref:2610.20652#pg3>. The combined method reduces this over-concentration substantially on every dataset <ref:2610.20652#pg3>.

Jane: So, this means for someone just listening to the show who doesn't read papers, the point is that if you deploy personalization today, you might lose some of that "I don't understand" capability unless you change how you score those unknown inputs <ref:2610.20652#pg1>.

Tom: It really puts it in perspective for deployment because this fix uses Mahalanobis distance instead of softmax confidence without any extra training or retraining, and it beats two other OOD scoring baselines <ref:2610.20652#pg3>.

Meng: From a production standpoint, that low-millisecond on-device latency cost is key. If we can fix this with just a score change without retraining the whole model, that’s something we can actually deploy <ref:2610.20652#pg3>.

Lu: The paper does have some caveats they mention, though. They note that only gradient-based naive and orthogonal-projection LoRA are covered in their personalization scope <ref:2610.20652#pg3>.

Jane: And they also admit that base intent accuracy drops eight to thirty-one points on every dataset after personalization for both naive LoRA and the combined maha method, which doesn't reliably reduce that drop <ref:2610.20652#pg3>.

Tom: That’s an honest admission. They aren't claiming this fix magically solves the core accuracy drop when you introduce new commands; it’s specifically about preserving the OOS rejection quality <ref:2610.20652#pg1>.

Lu: So, to wrap up, the main contribution of "Does Personalization Break Trust? Open-Set Rejection Calibration Under Few-Shot On-Device Intent Adaptation" is demonstrating that personalization measurably erodes the second property—the ability to reject unknown inputs—while improving the first property—the accuracy on known intents <ref:2610.20652#pg1>.

Jane: And they show that a simple, deployment-cheap change, like scoring rejection by Mahalanobis distance instead of softmax confidence, recovers most of that lost quality on three of five datasets with no accuracy cost <ref:2610.20652#pg3>.

Tom: It’s a practical solution for anyone building pervasive assistants who are doing on-device adaptation, proving that addressing OOS rejection calibration under few-shot learning is a real engineering problem <ref:2610.20652#pg1>.

Conclusion: Tom: So we've seen how naive personalization hurts out-of-scope rejection quality across five different benchmarks <ref:2610.20652#pg1>.

Jane: Right, that means when we adapt an AI assistant to learn new commands on the device, it gets worse at saying "I don't understand" things it hasn't seen before.

Lu: The authors are looking at this problem of out-of-scope detection calibration under few-shot adaptation <ref:2610.20652#pg3>.

Meng: And they found a fix that uses Mahalanobis distance instead of just the usual softmax confidence score <ref:2610.20652#pg3>.

Tom: The authors claim this fix recovers most of the lost rejection quality on three out of five datasets with no extra training needed <ref:2610.20652#pg3>.

Jane: So, it’s a simple scoring change that makes a real difference for those who only listen to the show.

Lu: It’s about replacing the standard confidence score with a Mahalanobis distance score based on Gaussian fitting over the embeddings <ref:2610.20652#pg3>.

Meng: And it’s deployment-cheap, which is what I care about most <ref:2610.20652#pg3>.

Tom: The authors also noted that this fix holds up across different few-shot budgets and even different base encoders <ref:2610.20652#pg3>.

Jane: It shows that even with personalization, we can maintain trust in the assistant's rejection abilities if we tweak how we score the outputs.

Lu: The paper does acknowledge some limitations, like only covering certain types of LoRA <ref:2610.20652#pg3>.

Meng: And they admit that base intent accuracy still drops a lot after personalization for both methods, so this fix doesn't completely solve the initial problem <ref:2610.20652#pg3>.

Tom: The overall message is that teaching an AI new commands and keeping it able to reject unknown inputs are actually two separate engineering problems <ref:2610.20652#pg3>.

Jane: It’s a reminder that deploying these pervasive assistants needs careful calibration of those rejection scores, not just focusing on the command learning part.