Assessing AI vs Human-Authored Spear Phishing SMS Attacks: An Empirical Study

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "Assessing AI vs Human-Authored Spear Phishing SMS Attacks: An Empirical Study".

Jane: The paper was written by Jerson Francia, Derek Hansen, Benjamin Schooley, Matthew Taylor, Shydra Valynn Murray et al. from Brigham Young University.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Title: Tom: Welcome back to the channel, everyone! Today we're digging into a paper that's going to make you think twice before you click that text message link. It's called "Assessing AI vs Human-Authored Spear Phishing SMS Attacks: An Empirical Study."

Jane: And Tom, I have to say, just the title alone gives me chills. We're talking about the difference between messages crafted by a person and messages crafted by a machine, and which one is better at tricking you.

Tom: Exactly! And the authors are from Brigham Young University — Jerson Francia, Derek Hansen, Ben Schooley, Matthew Taylor, Shydra Murray, and Greg Snow. They put together this really clever experiment to see how GPT-four stacks up against actual human authors.

Jane: So for our listeners who might not be deep in the cybersecurity weeds, let's break this down. Spear phishing is when someone sends you a message that's personalized — they know your name, your job, maybe what you posted on social media. It's not the generic "you've won a prize" spam. It's targeted.

Tom: Right, and the scary part is that according to the paper, spear phishing made up seventy-four percent of phishing attacks in two thousand twenty-two. Bulk phishing was only eight percent. So the bad guys have already figured out that personalization works.

Jane: And now we have AI that can generate these personalized messages at scale. The researchers wanted to know if GPT-four could write a convincing text message that would make someone click a malicious link, compared to messages written by actual people.

Tom: They recruited twenty-five targets — real people with real jobs, real hobbies — and had them share personal details. Then they had both human authors and GPT-four craft spear phishing SMS messages tailored to each person.

Jane: And here's the kicker, Tom. When they asked the targets to rank which messages were most convincing, the AI-generated ones ranked slightly higher on average. Not by a huge margin, but higher.

Tom: But here's what really got me — the targets could only guess which messages were AI-generated fifty-two percent of the time. That's basically a coin flip.

Jane: So the title of this paper is really asking the question we all should be asking: can we even tell the difference anymore? And the answer seems to be no.

Tom: And that's just the beginning. We need to talk about what made some messages more convincing than others, and why job-related messages were twice as likely to get a click.

Jane: That's coming up next. Stick around.

Summary: Jane: So Tom, we've established that the paper "Assessing AI vs Human-Authored Spear Phishing SMS Attacks: An Empirical Study" found AI messages are basically indistinguishable from human ones. But what actually made those messages work?

Tom: Great question, Jane. The researchers broke down the content characteristics that made people say "yeah, I'd click that." And the biggest factor, mentioned by seventy-six percent of participants, was personal relevance.

Jane: So if the message talked about their actual job, their actual hobby, something they actually posted online — that made it way more convincing.

Tom: Exactly. And this is where it gets interesting. Job-related messages had a thirty-eight percent click rate. Hobby messages? nineteen percent. Social media posts? seventeen percent. So if you want to trick someone, talk about their work.

Jane: That makes sense. If I get a text that says "your paycheck was delayed, click here to fix it," I'm going to panic and click. But if it says "we saw you like gardening, here's a coupon," I might be more skeptical.

Tom: Right. And the other big factors were the URL itself — sixty-four percent of people mentioned that. If the link looked legitimate, like it had HTTPS or a trusted domain, they were more likely to click. But if it was a shortened link like bit.ly, that raised red flags.

Jane: And what about the style of the message? I feel like that would matter.

Tom: It did. forty percent of participants mentioned style. But here's the twist — some people found casual, informal messages more convincing because they felt more authentic. Others found them less convincing because they seemed unprofessional.

Jane: So it's not one-size-fits-all. What works on one person might not work on another.

Tom: That's exactly what the researchers found. They even mentioned that some features that dissuade one person might persuade another. That's a scary thought for defenders, because it means attackers could potentially tailor messages to individual preferences.

Jane: And that's where AI becomes really dangerous. Because a human attacker has to manually craft each message. But GPT-four can generate hundreds of personalized messages in minutes.

Tom: The paper also mentioned the scarcity principle — urgency and fear. thirty-two percent of participants talked about that. But here's the weird part — some people were more likely to click when a message was urgent, because they wanted to fix the problem. Others were less likely because urgency felt like a warning sign.

Jane: So urgency works on some people and backfires on others. That's a really important nuance.

Tom: And it ties into what the researchers call the TRAPD methodology — their new way of testing these messages ethically. We should talk about that, because it's a big contribution.

Jane: Definitely. But first, let's talk about what happens when people try to figure out if a message was written by a machine.

Improvements: Jane: So we're back with the paper "Assessing AI vs Human-Authored Spear Phishing SMS Attacks: An Empirical Study." And Tom, I want to dig into the part about how people tried to identify AI-generated messages.

Tom: Yeah, this is where it gets really fascinating. The researchers asked participants to guess which messages were AI-generated. And like we said, they got it right only fifty-two percent of the time. But the reasons they gave for their guesses were all over the place.

Jane: Give me an example.

Tom: So forty percent of people mentioned style. Some thought AI messages were too formal. Others thought they were too informal. Some thought the messages were too perfect — like the grammar was flawless, so it had to be a machine.

Jane: And that's the trap, right? Because humans make mistakes, so people assume perfect grammar means AI. But that's not a reliable indicator.

Tom: Exactly. And here's a really interesting one — emojis. twenty percent of participants mentioned emojis. But half of them thought emojis meant AI, and the other half thought emojis meant human.

Jane: Wait, really? So people were split on whether a robot would use emojis?

Tom: Yes! Some people thought "AI can't use emojis, so this must be human." Others thought "AI always uses emojis, so this must be AI." And the reality? AI used emojis in sixty-six percent of its messages, while humans only used them in two percent.

Jane: So the people who thought emojis meant AI were actually right, but the ones who thought the opposite were completely wrong.

Tom: And that's the problem — people don't have reliable mental models for what AI can and can't do. The paper even mentioned that forty-eight percent of participants said they had no idea how to tell the difference.

Jane: That's a huge number. Nearly half of the people in the study just gave up on trying to figure it out.

Tom: And the researchers also found that people thought AI messages would be longer. They were right about that — AI messages averaged three hundred thirty-seven characters, while human messages averaged two hundred thirty-seven. But that's not a reliable indicator either, because attackers can just prompt the AI to write shorter messages.

Jane: So what does this mean for improving our defenses? The paper suggests we need better training, but also better detection tools.

Tom: Right. And one of the interesting suggestions is that the TRAPD methodology itself could be used for training. Instead of just telling people "be careful," you have them actually rank personalized messages and see what tricks them.

Jane: That's a really practical improvement. It's like fire drills instead of just reading a fire safety pamphlet.

Tom: Exactly. And the researchers also noted that AI is only going to get better. They used GPT-four but newer models are already out there. So the window for humans to catch up is closing fast.

Jane: I want to bring in Lu and Meng to get their take on this. Lu, you work with AI every day — does this match what you see?

Lu: It absolutely does, Jane. What strikes me is how the study shows AI doesn't need to be perfect to be dangerous. It just needs to be good enough, and it already is. The fact that people can't reliably tell the difference means we're past the point where human intuition is a viable defense.

Meng: And from an engineering standpoint, the scalability is the real threat. A human attacker might craft ten messages a day. An AI can craft ten thousand. Even if the AI's messages are only as good as a human's, the volume changes the game completely.

Tom: That's a great point, Meng. So the improvements the paper suggests aren't just about better training — they're about building automated defenses that can detect AI-generated content.

Jane: And that's where we're heading next. Let's wrap this up.

Conclusion: Tom: Alright, we've spent some time with "Assessing AI vs Human-Authored Spear Phishing SMS Attacks: An Empirical Study," and it's time to wrap up our thoughts.

Jane: So let's recap the big takeaways. First, AI-generated spear phishing messages are just as convincing as human-written ones, if not more so. The study found an eighty percent probability that GPT-four is at least as good as humans at this task.

Tom: Second, job-related messages are the most dangerous — thirty-eight percent click rate compared to under twenty percent for hobbies and social media.

Jane: And third, people cannot reliably tell the difference between AI and human messages. fifty-two percent accuracy is essentially guessing.

Tom: The researchers introduced the TRAPD methodology, which is an ethical way to test personalized deceptive messages. That's a real contribution to the field, because it lets researchers study this without actually tricking people.

Jane: And the implications are pretty serious. As Lu pointed out, AI doesn't need to be perfect to be dangerous. And as Meng said, the scalability is the real game-changer.

Lu: I think the most important takeaway is that we need to stop relying on people to spot these attacks. The human firewall is no longer sufficient. We need technical solutions that can detect AI-generated content automatically.

Meng: And we need to think about this from a systems perspective. If AI can generate convincing phishing messages at scale, then our defenses also need to be automated and scalable. It's an arms race.

Lalam: If I may add, the cultural implication here is significant. As AI becomes better at mimicking human communication, trust in digital messages will erode. We'll need new social norms and verification mechanisms — not just for security, but for everyday communication.

Tom: That's a deep point, Lalam. We're not just talking about cybersecurity anymore. We're talking about how we trust information in general.

Jane: And that's why this paper matters. It's not just a technical study — it's a warning about the future of communication.

Tom: Well said, Jane. We're going to say goodbye to this paper and get ready for the next one. Thanks for listening, everyone. Stay safe out there.

Jane: And maybe think twice before clicking that link in a text message. Even if it looks like it's from your boss.

Tom: See you next time!