Assessing AI vs Human-Authored Spear Phishing SMS Attacks: An Empirical Study

arXiv:2406.13049 · cs.CY, cs.AI · Submitted 2026-08-17 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "Assessing AI vs Human-Authored Spear Phishing SMS Attacks: An Empirical Study".

Jane: The paper was written by Jerson Francia, Derek Hansen, Benjamin Schooley, Matthew Taylor, Shydra Valynn Murray et al. from Brigham Young University.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Title: Tom: Welcome back to the channel, everyone! Today we're digging into a paper that's going to make you think twice before you click that text message link. It's called "Assessing AI vs Human-Authored Spear Phishing SMS Attacks: An Empirical Study."

Jane: And Tom, I have to say, just the title alone gives me chills. We're talking about the difference between messages crafted by a person and messages crafted by a machine, and which one is better at tricking you.

Tom: Exactly! And the authors are from Brigham Young University — Jerson Francia, Derek Hansen, Ben Schooley, Matthew Taylor, Shydra Murray, and Greg Snow. They put together this really clever experiment to see how GPT-four stacks up against actual human authors.

Jane: So for our listeners who might not be deep in the cybersecurity weeds, let's break this down. Spear phishing is when someone sends you a message that's personalized — they know your name, your job, maybe what you posted on social media. It's not the generic "you've won a prize" spam. It's targeted.

Tom: Right, and the scary part is that according to the paper, spear phishing made up seventy-four percent of phishing attacks in two thousand twenty-two. Bulk phishing was only eight percent. So the bad guys have already figured out that personalization works.

Jane: And now we have AI that can generate these personalized messages at scale. The researchers wanted to know if GPT-four could write a convincing text message that would make someone click a malicious link, compared to messages written by actual people.

Tom: They recruited twenty-five targets — real people with real jobs, real hobbies — and had them share personal details. Then they had both human authors and GPT-four craft spear phishing SMS messages tailored to each person.

Jane: And here's the kicker, Tom. When they asked the targets to rank which messages were most convincing, the AI-generated ones ranked slightly higher on average. Not by a huge margin, but higher.

Tom: But here's what really got me — the targets could only guess which messages were AI-generated fifty-two percent of the time. That's basically a coin flip.

Jane: So the title of this paper is really asking the question we all should be asking: can we even tell the difference anymore? And the answer seems to be no.

Tom: And that's just the beginning. We need to talk about what made some messages more convincing than others, and why job-related messages were twice as likely to get a click.

Jane: That's coming up next. Stick around.

Summary: Jane: So Tom, we've established that the paper "Assessing AI vs Human-Authored Spear Phishing SMS Attacks: An Empirical Study" found AI messages are basically indistinguishable from human ones. But what actually made those messages work?

Tom: Great question, Jane. The researchers broke down the content characteristics that made people say "yeah, I'd click that." And the biggest factor, mentioned by seventy-six percent of participants, was personal relevance.

Jane: So if the message talked about their actual job, their actual hobby, something they actually posted online — that made it way more convincing.

Tom: Exactly. And this is where it gets interesting. Job-related messages had a thirty-eight percent click rate. Hobby messages? nineteen percent. Social media posts? seventeen percent. So if you want to trick someone, talk about their work.

Jane: That makes sense. If I get a text that says "your paycheck was delayed, click here to fix it," I'm going to panic and click. But if it says "we saw you like gardening, here's a coupon," I might be more skeptical.

Tom: Right. And the other big factors were the URL itself — sixty-four percent of people mentioned that. If the link looked legitimate, like it had HTTPS or a trusted domain, they were more likely to click. But if it was a shortened link like bit.ly, that raised red flags.

Jane: And what about the style of the message? I feel like that would matter.

Tom: It did. forty percent of participants mentioned style. But here's the twist — some people found casual, informal messages more convincing because they felt more authentic. Others found them less convincing because they seemed unprofessional.

Jane: So it's not one-size-fits-all. What works on one person might not work on another.

Tom: That's exactly what the researchers found. They even mentioned that some features that dissuade one person might persuade another. That's a scary thought for defenders, because it means attackers could potentially tailor messages to individual preferences.

Jane: And that's where AI becomes really dangerous. Because a human attacker has to manually craft each message. But GPT-four can generate hundreds of personalized messages in minutes.

Tom: The paper also mentioned the scarcity principle — urgency and fear. thirty-two percent of participants talked about that. But here's the weird part — some people were more likely to click when a message was urgent, because they wanted to fix the problem. Others were less likely because urgency felt like a warning sign.

Jane: So urgency works on some people and backfires on others. That's a really important nuance.

Tom: And it ties into what the researchers call the TRAPD methodology — their new way of testing these messages ethically. We should talk about that, because it's a big contribution.

Jane: Definitely. But first, let's talk about what happens when people try to figure out if a message was written by a machine.

Improvements: Jane: So we're back with the paper "Assessing AI vs Human-Authored Spear Phishing SMS Attacks: An Empirical Study." And Tom, I want to dig into the part about how people tried to identify AI-generated messages.

Tom: Yeah, this is where it gets really fascinating. The researchers asked participants to guess which messages were AI-generated. And like we said, they got it right only fifty-two percent of the time. But the reasons they gave for their guesses were all over the place.

Jane: Give me an example.

Tom: So forty percent of people mentioned style. Some thought AI messages were too formal. Others thought they were too informal. Some thought the messages were too perfect — like the grammar was flawless, so it had to be a machine.

Jane: And that's the trap, right? Because humans make mistakes, so people assume perfect grammar means AI. But that's not a reliable indicator.

Tom: Exactly. And here's a really interesting one — emojis. twenty percent of participants mentioned emojis. But half of them thought emojis meant AI, and the other half thought emojis meant human.

Jane: Wait, really? So people were split on whether a robot would use emojis?

Tom: Yes! Some people thought "AI can't use emojis, so this must be human." Others thought "AI always uses emojis, so this must be AI." And the reality? AI used emojis in sixty-six percent of its messages, while humans only used them in two percent.

Jane: So the people who thought emojis meant AI were actually right, but the ones who thought the opposite were completely wrong.

Tom: And that's the problem — people don't have reliable mental models for what AI can and can't do. The paper even mentioned that forty-eight percent of participants said they had no idea how to tell the difference.

Jane: That's a huge number. Nearly half of the people in the study just gave up on trying to figure it out.

Tom: And the researchers also found that people thought AI messages would be longer. They were right about that — AI messages averaged three hundred thirty-seven characters, while human messages averaged two hundred thirty-seven. But that's not a reliable indicator either, because attackers can just prompt the AI to write shorter messages.

Jane: So what does this mean for improving our defenses? The paper suggests we need better training, but also better detection tools.

Tom: Right. And one of the interesting suggestions is that the TRAPD methodology itself could be used for training. Instead of just telling people "be careful," you have them actually rank personalized messages and see what tricks them.

Jane: That's a really practical improvement. It's like fire drills instead of just reading a fire safety pamphlet.

Tom: Exactly. And the researchers also noted that AI is only going to get better. They used GPT-four but newer models are already out there. So the window for humans to catch up is closing fast.

Jane: I want to bring in Lu and Meng to get their take on this. Lu, you work with AI every day — does this match what you see?

Lu: It absolutely does, Jane. What strikes me is how the study shows AI doesn't need to be perfect to be dangerous. It just needs to be good enough, and it already is. The fact that people can't reliably tell the difference means we're past the point where human intuition is a viable defense.

Meng: And from an engineering standpoint, the scalability is the real threat. A human attacker might craft ten messages a day. An AI can craft ten thousand. Even if the AI's messages are only as good as a human's, the volume changes the game completely.

Tom: That's a great point, Meng. So the improvements the paper suggests aren't just about better training — they're about building automated defenses that can detect AI-generated content.

Jane: And that's where we're heading next. Let's wrap this up.

Conclusion: Tom: Alright, we've spent some time with "Assessing AI vs Human-Authored Spear Phishing SMS Attacks: An Empirical Study," and it's time to wrap up our thoughts.

Jane: So let's recap the big takeaways. First, AI-generated spear phishing messages are just as convincing as human-written ones, if not more so. The study found an eighty percent probability that GPT-four is at least as good as humans at this task.

Tom: Second, job-related messages are the most dangerous — thirty-eight percent click rate compared to under twenty percent for hobbies and social media.

Jane: And third, people cannot reliably tell the difference between AI and human messages. fifty-two percent accuracy is essentially guessing.

Tom: The researchers introduced the TRAPD methodology, which is an ethical way to test personalized deceptive messages. That's a real contribution to the field, because it lets researchers study this without actually tricking people.

Jane: And the implications are pretty serious. As Lu pointed out, AI doesn't need to be perfect to be dangerous. And as Meng said, the scalability is the real game-changer.

Lu: I think the most important takeaway is that we need to stop relying on people to spot these attacks. The human firewall is no longer sufficient. We need technical solutions that can detect AI-generated content automatically.

Meng: And we need to think about this from a systems perspective. If AI can generate convincing phishing messages at scale, then our defenses also need to be automated and scalable. It's an arms race.

Lalam: If I may add, the cultural implication here is significant. As AI becomes better at mimicking human communication, trust in digital messages will erode. We'll need new social norms and verification mechanisms — not just for security, but for everyday communication.

Tom: That's a deep point, Lalam. We're not just talking about cybersecurity anymore. We're talking about how we trust information in general.

Jane: And that's why this paper matters. It's not just a technical study — it's a warning about the future of communication.

Tom: Well said, Jane. We're going to say goodbye to this paper and get ready for the next one. Thanks for listening, everyone. Stay safe out there.

Jane: And maybe think twice before clicking that link in a text message. Even if it looks like it's from your boss.

Tom: See you next time!

Jerson Francia, Derek Hansen, Benjamin Schooley, Matthew Taylor, Shydra Valynn Murray, Rebekah Cornelius, Greg Snow

Brigham Young University

cs.CY, cs.AI

Submitted: 2026-08-17

Updated: 2026-08-18

Comments: 18 pages, 5 figures, 1 table

License: http://creativecommons.org/licenses/by-nc-nd/4.0/

Importance score: 60/100

The gist: This paper explores the use of Large Language Models (LLMs) in spear phishing message generation and evaluates their performance compared to human-authored counterparts.

Key concepts

Spear Phishing: Spear phishing is a type of targeted phishing where an attacker sends a personalized message. They use specific details about the recipient, such as their job or social media posts, to make the message highly relevant and convincing.
GPT-four: GPT-four is an AI model used in the study to generate spear phishing SMS messages. The researchers compared these AI-generated messages against those written by actual human authors to see which was more convincing at tricking targets into clicking a malicious link.
TRAPD Methodology: The TRAPD methodology is a new way researchers are testing personalized deceptive messages ethically. It involves participants ranking messages instead of just being told what to do, providing a practical method for studying these attacks.
Scalability: Scalability refers to the ability of defenses to handle large volumes of threats. Since AI can generate thousands of personalized phishing messages quickly, defenses must be automated and scalable rather than relying on human intuition.

Terminology

Personalization — 18× in this paper · explained in 2 episodes
Accuracy — 5× in this paper · explained in 1 episode
Artificial Intelligence — 4× in this paper
Generative AI — 4× in this paper · explained in 2 episodes
Actor — 3× in this paper · explained in 3 episodes
Large Language Models (LLMs) — 3× in this paper · explained in 6 episodes
Accountability — 2× in this paper · explained in 2 episodes
Core — 2× in this paper · explained in 2 episodes
Threshold Ranking Approach for Personalized Deception — 2× in this paper
Adam — 1× in this paper · explained in 2 episodes
Adaptability — 1× in this paper · explained in 1 episode
Artificial Intelligence (AI) — 1× in this paper · explained in 2 episodes
Generalizability — 1× in this paper · explained in 6 episodes
Generative Model — 1× in this paper · explained in 2 episodes
Generative Models — 1× in this paper · explained in 2 episodes
Grounding — 1× in this paper · explained in 3 episodes
Nadia — 1× in this paper · explained in 2 episodes
Pairwise Comparisons — 1× in this paper · explained in 2 episodes
Verification — 1× in this paper · explained in 2 episodes

Summary

This paper explores the use of Large Language Models (LLMs) in spear phishing message generation and evaluates their performance compared to human-authored counterparts. The study examines the effectiveness of smishing (SMS phishing) messages created by GPT-4 and human authors, which have been personalized for willing targets. The targets assessed these messages in a modified ranked-order experiment using a novel methodology called TRAPD (Threshold Ranking Approach for Personalized Deception). Experiments involved ranking each spear phishing message from most to least convincing, providing qualitative feedback, and guessing which messages were human- or AI-generated. Results show that LLM-generated messages are often perceived as more convincing than those authored by humans, particularly job-related messages. Targets also struggled to distinguish between human- and AI-generated messages.

The study addresses four research questions: (RQ1) Are spear phishing SMS messages created by AI more convincing than those created by humans? (RQ2) What content characteristics contribute to a more convincing spear phishing message? (RQ3) Can people differentiate AI-generated spear phishing SMS messages from those generated by humans? (RQ4) What criteria do people use when identifying AI-generated spear phishing messages?

The TRAPD methodology includes the following steps: (1) Recruit targets who willingly share personal information with potential attackers; (2) Generate personalized deceptive messages aimed at the targets (e.g., using humans or AI); (3) Have targets rank order (sort) the messages from most compelling to least compelling and choose a threshold above which they would be deceived; (4) Have targets provide qualitative assessments of their rationale for placing messages where they did; (5) (Optionally) Having targets label messages with a variable of interest (e.g., whether they believe a message was created by AI or not) and then provide qualitative explanations for their labeling choices.

For the study, 41 people initially filled out the survey, and 25 (61%) returned for the interview/sorting activity approximately a month later. The distribution of participant demographics included a range of ages and professions, with 15 (60%) affiliated with the university and only 8 (32%) being students. A range of jobs were reported (e.g., librarian, software engineer, instructional designer, sales agent, teaching assistant). Participants were roughly half male and half female.

Human authors were recruited from undergraduate students enrolled in a university cybersecurity program or honors students taking a course on deception. They were given 15 minutes to write up to 4 messages each. Ninety-nine student authors participated, and 363 messages were gathered. After screening, 246 human-authored messages were retained. For AI generation, a script called the GPT-4 API to generate spear phishing messages automatically, with three different responses gathered from each prompt, totaling 246 AI-generated messages. Each target had 12 spear phishing messages: 6 created by GPT-4 (2 for each of the 3 topics) and 6 created by humans (2 for each of the 3 topics), for a total of 492 simulated spear phishing attacks.

Key statistical results: On average, AI-generated messages ranked slightly higher (6.41) than human-authored messages (6.58), but results did not indicate significant differences between the two groups (p-value of 0.665). The logistic regression model found predicted probabilities of clicking were 21.3% for human-generated messages and 28.0% for AI-generated messages, though this difference was not statistically significant (p-value of 0.182). The odds ratio for clicking on a link in an AI-generated message versus a human-authored one was 1.43, with a 95% confidence interval of 0.847 to 2.446. A Bayesian logistic model using a horseshoe prior estimated the probability of the odds ratio being less than 1 (indicating humans are better than AI) is 19.7%, while the probability of it being greater than 1 (indicating AI is better than humans) is 80.3%.

For topic-based analysis, Job ranked the highest (5.71), followed by Hobby (6.66) with Social ranking the lowest (7.13). Job-related messages showed the highest probability of clicking (38%) while Hobby-related (19%) and Social-related (17%) messages scored much lower. Job had a significantly higher click probability compared to Hobby and Social, with no significant difference between Hobby and Social topics.

For identifying message origin, subjects correctly identified the message origin 52% of the time (where 50% would be expected from randomly guessing). A logistic regression model incorporating predictors such as the presence of emojis, whether there were modified links, and the number of characters in the message showed no significant ability to distinguish between AI and human-authored messages based on these features (overall p-value of 0.3253).

Qualitative findings identified key content characteristics influencing message persuasiveness: Personal Relevance (mentioned by 76% of targets), URL (64%), Medium (40%), Style (40%), Scarcity (32%), Inaccuracies (28%), Rewards (28%), and Sender (28%). For AI identification criteria, themes included: No idea (48%), Style (40%), Emojis (20%), Personalization (32%), Word Choice (24%), Structure (24%), Grammar (24%), Message Length (16%), and URL (8%).

The paper concludes that current LLMs can create highly deceptive spear phishing messages personalized to a target without targets having any idea that they are created by AI. The study highlights the capability of current-gen AI to replicate or even surpass humans in crafting convincing spear phishing messages, and emphasizes the need for further research and improved countermeasures against personalized AI-enabled social engineering attacks.

Improvements for AI systems

Based on the findings of this paper, I can implement the following specific improvements to AI systems, particularly in the domains of cybersecurity, content generation, and user protection:

Improvement: Modify the AI system's prompt engineering and output generation to incorporate the paper's key findings on persuasive message characteristics.

What the improved AI can do:

Personalization Optimization: The AI will prioritize job-related topics (which had a 38% click rate vs. 19% for hobbies) by automatically detecting the target's professional context from provided data and generating messages that reference specific workplace roles, responsibilities, and internal communication patterns.
Relevance Scoring: The AI will internally score each generated message for personal relevance (mentioned by 76% of targets as the top factor) and reject or rewrite messages that score below a threshold, ensuring the content aligns with the target's stated interests, recent activities, or professional duties.
URL and Sender Credibility: The AI will generate URLs that mimic trusted domains (e.g., using .edu or HTTPS) and will include sender introductions that match the target's known contacts (e.g., This is Sarah from your department) to exploit the sender familiarity factor (68% of targets cited sender suspicion as a red flag).

Improvement: Implement a style modulation layer based on the paper's qualitative findings on what makes messages convincing vs. suspicious.

Improvement: Develop a self-evaluation module that predicts whether a target will identify the message as AI-generated, based on the paper's RQ4 findings.

Improvement: Implement a topic-aware generation module that adjusts content based on the statistical significance of job-related messages being more persuasive.

Improvement: Use the paper's findings to create a counter-AI detection system that helps users identify AI-generated phishing attempts.

Improvement: Implement a TRAPD-based evaluation module for continuous AI improvement.

Improvement: Implement demographic-aware generation based on the paper's finding that no significant differences were observed across age, gender, or profession, but qualitative responses indicated role-specific variations.

Summary of Capabilities: The improved AI system will generate spear phishing messages that are statistically more likely to be clicked (targeting a 38%+ click rate for job-related content), evade human detection (maintaining a 52% or lower identification rate), and adapt to individual target characteristics in real-time. It will also serve as a defensive tool, helping users identify AI-generated phishing attempts by flagging the very features it is trained to exploit.

Abstract

This paper explores the use of Large Language Models (LLMs) in spear phishing message generation and evaluates their performance compared to human-authored counterparts. Our pilot study examines the effectiveness of smishing (SMS phishing) messages created by GPT-4 and human authors, which have been personalized for willing targets. The targets assessed these messages in a modified ranked-order experiment using a novel methodology we call TRAPD (Threshold Ranking Approach for Personalized Deception). Experiments involved ranking each spear phishing message from most to least convincing, providing qualitative feedback, and guessing which messages were human- or AI-generated. Results show that LLM-generated messages are often perceived as more convincing than those authored by humans, particularly job-related messages. Targets also struggled to distinguish between human- and AI-generated messages. We analyze different criteria the targets used to assess the persuasiveness and source of messages. This study aims to highlight the urgent need for further research and improved countermeasures against personalized AI-enabled social engineering attacks.