Selective Prediction Reduces the Negative Effects of Automation Bias Overall but Increases False Negatives

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "Selective Prediction Reduces the Negative Effects of Automation Bias Overall but Increases False Negatives".

Jane: The paper was written by Sarah Jabbour, David Fouhey, Nikola Banovic, Stephanie D. Shepard, Ella Kazerooni et al. from University of Michigan and New York University.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Jane: We also have Lu with us today — senior AI researcher at Tsinghua.

Tom: We also have Meng with us today — lead engineer at a mysterious AI startup.

Jane: We also have Lalam with us today — the in-house Large Language Model.

Tom: Alright, let's get started.

Title and Authors: Tom: Welcome back to the show, everybody. Today we're digging into a paper with a title that really makes you stop and think: "Selective Prediction Reduces the Negative Effects of Automation Bias Overall but Increases False Negatives." Jane, that title is a whole story in itself, isn't it?

Jane: It really is, Tom. It's basically saying, "We found a fix, but the fix has its own problem." And that's exactly the kind of nuance we need more of in AI research. The team is from the University of Michigan and NYU, with Sarah Jabbour leading the charge, and they ran a study with two hundred fifty-nine clinicians to test how AI tools change the way doctors treat patients with acute respiratory failure.

Tom: Right, and the setup is clever. They had doctors make treatment decisions with no AI, with AI that sometimes made mistakes, and with AI that was allowed to say, "I'm not sure, you decide on your own." That last one is the selective prediction part — the AI hides its prediction when it thinks showing it might hurt.

Jane: And the big finding is that when the AI just gave its prediction, even when it was wrong, doctors over-relied on it and their accuracy dropped from sixty-two percent down to fifty percent. That's automation bias in action. But when the AI withheld those bad predictions, accuracy came back up to fifty-five percent — so it helped, but it didn't fully restore the baseline.

Tom: But here's the twist that the title is pointing at. Even though overall accuracy improved, the type of errors changed. Doctors started making more false negatives — meaning they were more likely to undertreat patients who actually needed treatment. That's a serious clinical concern, because missing a treatment can be worse than giving an unnecessary one.

Jane: Exactly. And that's the kind of finding that should make us pause before we just assume that letting the AI abstain is a neutral, safe default. The paper is really challenging that assumption, and I think that's why it's getting so much attention.

Tom: So we're not just talking about a better algorithm here. We're talking about how the very act of telling a human "the AI is staying quiet on this one" changes their behavior. That's a human factors question, not just a machine learning question.

Jane: And it's a question that has real stakes. If we're going to deploy these systems in hospitals, we need to know not just whether they're accurate, but how they change the decisions people actually make. This paper is a big step toward that.

Tom: I'm already curious about how they actually ran this study and what the doctors were seeing on their screens. Let's get into the details in the next segment.

Summary and Methodology: Tom: So we're back with "Selective Prediction Reduces the Negative Effects of Automation Bias Overall but Increases False Negatives." Jane, let's talk about how they actually pulled this off, because it's not a typical lab experiment.

Jane: Right. They built a vignette-based study, which means they gave clinicians realistic patient cases — complete with chest X-rays, lab results, vital signs, and medical history — and asked them to figure out what was causing each patient's shortness of breath. The three possible culprits were pneumonia, heart failure, and COPD, and patients could have any combination of them.

Tom: And that multilabel setup is important, because it's not just a yes-or-no decision. The AI had to make three separate predictions for each patient, and selective prediction could hide some of those predictions while showing others. That's much closer to how these tools would actually work in practice.

Jane: Exactly. The doctors first did three cases with no AI at all to get a baseline. Then they were randomized into two groups: one saw all the AI predictions, and the other saw the AI's predictions only when they were accurate. When the AI was wrong, it showed a message saying "the model defers to you" instead of giving a score.

Tom: And they made sure the participants actually understood that deferral message. There was a knowledge check — you had to confirm that "the model may or may not believe the patient has the condition, but is choosing not to provide its input." So they were really testing the effect of abstention itself, not confusion about what abstention meant.

Jane: That's a key design choice. They wanted to isolate the behavioral response to being told "the AI is staying quiet," separate from any misunderstanding. And even with that understanding, the false negative rate still went up significantly — from thirty-one percent at baseline to forty-two percent when the AI abstained.

Tom: That's a huge jump. And it's not just a statistical blip — they used mixed-effects models to account for the fact that the same doctors were making multiple decisions and the same patient cases were being used across participants. So the finding is pretty robust.

Jane: And they also looked at subgroups. The effect was stronger for advanced practice practitioners than for physicians, and it was particularly pronounced for clinicians who said they had never used AI in their practice before. Those AI-naive clinicians saw their false negative rate jump by thirteen percentage points when the AI abstained.

Tom: So the people who are least familiar with AI are the ones most likely to be thrown off when it stays quiet. That makes intuitive sense — if you don't have experience with how these tools behave, you might interpret "I'm not sure" as "this is probably not a problem."

Jane: Right. And that's exactly the kind of insight you can't get from a simulation. You have to put real clinicians in front of real cases and watch what they do.

Tom: Okay, so we know the problem. What does this mean for how we should actually build and deploy these systems? That's what I want to dig into next.

Improvements and Implications: Tom: Welcome back. We're still on "Selective Prediction Reduces the Negative Effects of Automation Bias Overall but Increases False Negatives." Jane, we've established that selective prediction helps overall but shifts errors toward undertreatment. What does the paper suggest we do about that?

Jane: Well, the paper doesn't offer a simple fix, and I think that's actually the point. The authors argue that selective prediction should not be treated as a default safety mechanism. Instead, you have to evaluate it in each specific context and think carefully about which types of errors are more acceptable.

Tom: And in the context of treating acute respiratory failure, undertreatment is especially dangerous. The paper points out that delayed antibiotics for severe infections can lead to worse outcomes. So a system that nudges doctors toward undertreatment is trading one harm for another.

Jane: Exactly. And that's why the authors are calling for more research into why clinicians respond the way they do when the AI abstains. They have a couple of hypotheses. One is that clinicians might interpret "the model defers to you" as the model saying "there's no disease here," even when they understand it's abstaining.

Tom: That's the availability bias idea — the conditions the AI does show predictions for become more salient in the doctor's mind, so they focus on those and forget to consider the ones the AI stayed quiet about. The paper mentions that as a possible mechanism.

Jane: Right. And another implication is that training matters. The authors suggest that formal training on what selective prediction means could mitigate some of these effects. But they also note that even with a knowledge check in the study, the false negative effect still appeared. So training alone might not be enough.

Tom: So what's the practical path forward? Do we just abandon selective prediction?

Jane: No, I don't think so. The paper shows it does reduce the harm from automation bias — accuracy went from fifty percent back up to fifty-five percent. It's just that we need to be more thoughtful about when and how we use it. Maybe the AI should abstain less often, or maybe it should abstain only when the cost of a false positive is higher than the cost of a false negative.

Tom: That's a really important design question. And the paper also highlights that AI-experienced clinicians didn't show the same false negative effect. So there's hope that with more exposure, doctors can learn to calibrate their response to abstention.

Jane: Exactly. But we can't just assume that will happen on its own. We need to study it, measure it, and design for it. This paper is a great example of why we need to evaluate AI systems in the context of real workflows, not just in isolation.

Tom: Alright, let's wrap this up in the final segment.

Conclusion: Tom: We're wrapping up our discussion of "Selective Prediction Reduces the Negative Effects of Automation Bias Overall but Increases False Negatives." Jane, give us the final takeaway for our listeners.

Jane: Sure. This paper is a reminder that AI systems don't exist in a vacuum. When you put a tool in front of a human decision-maker, the way it behaves — including when it stays quiet — changes their behavior. Selective prediction can reduce the harm of automation bias, but it also shifts errors toward undertreatment, which can be just as dangerous in a clinical setting.

Tom: And that's a finding that should matter to anyone building AI for high-stakes decisions, not just in healthcare. The paper's core message is that we need to evaluate these systems in real workflows, with real users, and pay attention not just to accuracy but to the types of errors people make.

Jane: Right. And the authors are careful to note that their study is a vignette-based survey, so it's not the same as a real hospital setting. But it's an important first step in identifying potential harms before we deploy these systems widely.

Tom: It also opens up a lot of questions for future work — how to train clinicians to respond to abstention, how to design selective prediction mechanisms that account for the cost of different errors, and how to make sure the people who are least familiar with AI aren't the ones most harmed by it.

Jane: Definitely. This is one of those papers that doesn't give you a clean answer, but it gives you a much better understanding of the problem. And that's exactly what we need more of.

Tom: Well said. Thanks for joining us, and we'll be back soon with another paper. Until then, keep questioning the tools you use.

Jane: And remember — when an AI says "I'm not sure," that's not the same as saying "you're fine." Take care, everyone.