Context-Aware Spear Phishing: Generative AI-Enabled Attacks Against Individuals via Public Social Media Data

arXiv:2605.11268 · cs.CR · Submitted 2026-05-11 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.

Nadia: Today's paper: "Context-Aware Spear Phishing".

Elias: This research demonstrates how publicly available social media data and generative AI (GenAI) can be used to automate and scale highly personalized, context-aware spear-phishing campaigns,

Nadia: First, who's behind it and why it matters.

Title and authors: Nadia: So, we're looking at this paper, "Context-Aware Spear Phishing: Generative AI-Enabled Attacks Against Individuals via Public Social Media Data," and it seems like it’s focusing on how publicly available social media data combined with generative AI can be used to automate and scale highly personalized spear-phishing campaigns.

Elias: That sounds like a serious problem for security; if the effort required from the attacker is minimal, we're looking at a massive scaling issue.

Priya: From my side, I'm curious about what kind of data they are using and what kind of actual behavioral insights these models can pull from social media.

Nadia: Well, the paper introduces a modular framework that combines multimodal signal extraction, communication-style profiling, and attack-type instantiation across seven strategies (baiting, scareware, honey trap, tailgating, impersonation, quid pro quo, and personalized emotional exploitation) to do this.

Elias: Seven distinct attack strategies fused with contextual dimensions—location targeting users based on geography or tailoring messages to their expressed interests—that's a lot of variables for an attacker to manage.

Priya: That modular approach suggests they aren't just making one generic phishing template, but creating a system that adapts the attack based on deep profiling.

Nadia: Exactly, and what really caught my eye is how GenAI-produced emails exhibit higher personalization and persuasiveness compared to real-world phishing baselines while eliciting lower suspicion from human recipients.

Elias: That’s where the cryptographic side gets interesting; if the output is that persuasive, it means the LLM is doing heavy lifting on linguistic naturalness and emotional manipulation, which might be hard to detect through simple pattern matching.

Priya: I agree, and when you look at their evaluation metrics against real-world phishing messages in APWG eCrimeX data, they show GenAI outputs are consistently better in terms of personalization and believability—for instance, underperforming real-world campaigns by eighty-five to ninety percent on linguistic naturalness.

Nadia: It really highlights the deceptive quality of this technology; it’s making the phishing attempts much harder to spot for the average user.

Elias: And that brings us to how they propose we can defend against this, because simply looking at the content isn't enough anymore, Elias thinks.

Title and authors: Priya: I think their focus on proactive defense mechanisms is crucial because it addresses the way attackers are using prompt engineering to bypass existing safeguards.

Nadia: Right, and what did they find when they tested those commercial LLM safety filters and prompt-level guardrails against these context-aware attacks?

Elias: The paper reports that a RoBERTa-based detector achieved ninety-eight point one percent accuracy when testing these SOTA safeguards across different models and attack types, which shows some resilience there.

Priya: But the authors also found that while default safeguards often fail against adaptive evasion strategies, injecting policy into the system substantially improved blocking rates to eighty-four percent for ShieldGemma and reached ninety-eight point seven percent detection on a specific malicious prompt set using System-Instruction plus Chain-of-Thought moderation.

Nadia: That’s a big jump in detection accuracy when they add that layer of reasoning before the generation even happens; it shows that we need to think about blocking intent before content is finalized.

Elias: That points toward a necessary shift in how we design defenses, moving away from surface-level checks toward deeper instruction-based reasoning, which I find very compelling given the complexity of these attacks.

Priya: It really suggests that for privacy and measurement researchers, the focus needs to be on how much contextual information an attacker can actually extract from social media data before it becomes actionable intelligence for a targeted attack.

Nadia: And that leads us nicely into how this work changes our thinking about defense strategies overall, because we’re seeing these attacks scale with minimal effort.

Elias: It certainly does, and I wonder if the paper's focus on prompt-level detection is something we should be looking at when considering other areas of AI security research, like the backdoors mentioned in some of those other papers.

Priya: I think the implication for privacy researchers is that this kind of profiling makes it easier to create highly specific attacks against individuals, which raises serious concerns about surveillance and targeted manipulation.

Title and authors: Nadia: It does; if an attacker can use public data to craft a message that perfectly matches a target’s style and current emotional state, the personal boundary essentially dissolves.

Elias: And from a cryptographic viewpoint, the success of this hinges on the LLM’s ability to maintain that deep contextual understanding throughout the entire four-stage pipeline they model.

Priya: So, while we see these sophisticated attacks emerging, what do you think are the long-term implications for how we design communication security protocols?

Nadia: I think we need platform-level safeguards that explicitly account for this kind of contextualized abuse at scale because current defenses seem insufficient against these adaptive prompt engineering tactics.

Elias: I agree; the research provides a unified framework and evaluation methodology, which is valuable because it helps us measure exactly where the weaknesses are in our current defense layers.

Priya: Ultimately, the main implication for privacy is that we have to treat public social media data not just as information, but as a high-fidelity vector for constructing highly personalized malicious content.

Nadia: So, to wrap up this discussion on "Context-Aware Spear Phishing: Generative AI-Enabled Attacks Against Individuals via Public Social Media Data," it’s clear that the combination of social media signals and generative AI makes spear phishing financially trivial for resource-constrained adversaries.

Elias: That's right; the study demonstrates how minimal attacker effort is sufficient when they leverage this modular framework to execute those seven attack strategies.

Priya: I just want to stress that the finding about LLM-generated emails eliciting lower perceived suspiciousness than real-world phishing emails really underscores the deceptive potential we have to consider.

Nadia: It certainly does, and we need to keep looking at how they are using these tools, because this work gives us a much clearer picture of how they operate now.

Elias: Moving forward, the challenge will be developing those robust, prompt-aware defenses that the authors proposed as a way to block malicious intent before content is even generated.

Priya: And I think we should keep watching how these contextual profiling techniques evolve because they directly impact the privacy of individuals in ways we haven't fully quantified yet.

The paper's summary: Nadia: So, this paper basically lays out how attackers can use readily available social media data and generative AI to build these hyper-personalized phishing attacks that are incredibly cheap for them to run at scale.

Elias: And what strikes me from a cryptographic standpoint is that the authors are modeling a very specific pipeline—Contextual Extraction, Attack Type Integration, Style Mimicry, and Output Formatting—showing exactly how the AI executes that transformation step by step.

Priya: From my research angle, what’s really important is their taxonomy; they don't just lump everything together; they define seven distinct attack strategies fused with five contextual dimensions like location or sentiment to drive the personalization.

Nadia: Exactly! It shows that the AI isn't just writing a generic email; it’s systematically mapping social media profiles onto specific, high-leverage psychological attack types, which is what makes them so effective at bypassing standard filters.

Elias: And when we look at their evaluation metrics against real data, the results show that these AI-generated emails are significantly more persuasive and natural sounding than anything human could craft manually, with those linguistic naturalness scores being quite high compared to actual phishing samples.

Priya: That’s a huge finding for privacy work; if the generated content is that believable, it means we have to seriously consider how much of an individual's public social media footprint can be leveraged to create a highly convincing vector for manipulation.

Nadia: It really makes you wonder about the impact on the world when these attacks become this automated and scalable, because resource-constrained adversaries can now target individuals with a level of detail that was previously impossible.

Elias: I think the core implication for security is that defense has to move beyond just looking at the final text; it needs to focus on stopping those initial prompt injections or extraction stages where the context is being fed into the AI pipeline.

Priya: And that leads us directly into how we need to rethink detection methods, because if attackers can extract these contextual signals with such efficiency, current keyword filters are going to become obsolete very quickly.

The paper's improvements: Nadia: So, we're looking at how the authors suggest we actually improve things to stop these context-aware attacks, moving beyond just patching surface-level filters.

Elias: The paper points toward developing "Context-Aware Defense Layers" that operate at the prompt or instruction level, which means instead of just checking the final email text, you scrutinize *how* the AI is being asked to generate it.

Priya: That’s smart because it addresses the core issue: attackers are using those complex attack pipelines to sneak malicious intent into the instructions themselves, so we need a system that reasons through that structure before content even starts forming.

Nadia: Right, and they suggest forcing the LLM to use Chain-of-Thought reasoning during prompt construction to make it justify its generation against that seven-strategy taxonomy.

Elias: I see the value in using a DeBERTa model for sub-prompt detection, which would help intercept those fragmented, malicious instructions before they fully manifest into a phishing email, which is exactly what we need given the adaptive nature of these adversaries.

Priya: From a measurement standpoint, this suggests that defense mechanisms shouldn't just measure success on the final output; they should measure resilience against the entire generation process, including how much contextual information an attacker can extract from public data.

Nadia: It sounds like we need a feedback loop where the results of adversarial testing automatically update the training sets for those prompt-level detectors so defenses adapt as fast as attackers evolve their evasion techniques.

Elias: If you look at the architectural trade-offs they analyzed, it shows that different LLMs have different strengths—one might excel at emotional manipulation while another handles linguistic naturalness better—so a good defense system needs to be flexible enough to pick the right tool for the threat scenario.

Priya: And that ties back to our privacy concerns; if we can build these robust, context-aware defenses, it offers a way to mitigate the risk of individuals being profiled and manipulated using their own public data as a weapon against them.

Conclusion: Nadia: So, to wrap things up on "Context-Aware Spear Phishing: Generative AI-Enabled Attacks Against Individuals via Public Social Media Data," we’ve seen how this combination of social media data and AI allows adversaries to build incredibly detailed and persuasive spear-phishing campaigns with minimal effort.

Elias: It really shows that the assumptions underpinning these attacks are quite specific, relying on the AI's ability to map multimodal signals onto a structured taxonomy of seven attack strategies.

Priya: And from my side, what I keep thinking about is how much actionable intelligence an attacker can extract from public data before it becomes a weapon for manipulation, which speaks directly to privacy concerns.

Nadia: Exactly; the finding that these emails elicit lower perceived suspicion than real phishing attempts underscores the deceptive potential we have when public data is used this way.

Elias: I think the main cryptographic implication is that because of how thoroughly these pipelines are modeled, it highlights a gap in how we secure the input-to-output translation process within generative systems.

Priya: It’s a powerful demonstration of why privacy researchers need to focus on controlling that contextual profiling aspect, because once you can profile someone this deeply, the risk of targeted influence skyrockets.

Nadia: We've got a clear picture now of how these attacks function and the specific ways they leverage AI to bypass traditional content moderation methods.

Elias: Moving forward, we need to keep pushing for those prompt-level defenses we discussed earlier so that detection happens before the malicious intent is fully baked into the output.

Priya: I think the long-term impact for privacy is that we have to treat public social media data not just as a source of information, but as a high-fidelity vector for constructing highly personalized malicious content.

Elham Pourabbas Vafa, Sayak Saha Roy, Shirin Nilizadeh

The University of Texas at Arlington · Louisiana State University

cs.CR

Submitted: 2026-05-11

Updated: 2026-09-30

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 85/100

The gist: This research demonstrates how publicly available social media data and generative AI (GenAI) can be used to automate and scale highly personalized, context-aware spear-phishing campaigns, posing a

Key concepts

Attack Generation Pipeline
A four-stage process where GenAI systematically creates a spear-phishing email. It starts by extracting social media data like names and locations, then integrates a specific attack strategy with contextual details, mimics the communication style, and finally formats the output.
Taxonomy of Social Engineering Attacks
A unified way to categorize these attacks by combining two dimensions: seven different attack types (like baiting or impersonation) with five contextual dimensions. This allows researchers to systematically map how personalization and context are used in the attacks.
Contextual Dimensions
These are the layers of personalization added to an attack, such as targeting users based on their location, exploiting known hobbies, or adapting the tone of the message based on a user's current emotional state. They provide deep context for making phishing attempts highly relevant.
Suspiciousness Gap
A finding from human testing where recipients rated AI-generated emails as less suspicious than real phishing emails. This gap occurs because the AI content is highly persuasive and personalized, successfully masking its malicious intent from the human reader.

Terminology

Summary

This research demonstrates how publicly available social media data and generative AI (GenAI) can be used to automate and scale highly personalized, context-aware spear-phishing campaigns, posing a significant threat due to its minimal attacker effort and ability to bypass traditional content moderation. The study introduces a modular framework that combines multimodal signal extraction, communication-style profiling, and attack-type instantiation across seven strategies, showing that GenAI-produced emails exhibit markedly higher personalization and persuasiveness than real-world phishing baselines while eliciting lower suspicion from human recipients.

Attack Generation Pipeline

The framework models the systematic automation of context-aware spear phishing through a four-stage process: (1) Contextual Extraction, (2) Attack Type × Context Integration, (3) Style Mimicry, and (4) Output Formatting. The pipeline leverages GenAI models to execute this workflow. For instance, the extraction stage involves prompting LLMs to analyze multimodal social media data—such as Instagram posts—to extract salient attributes like names, locations, affiliations, and recent activities while minimizing hallucinations. The integration stage aligns one of seven attack strategies with five contextual dimensions (e.g., combining a honey trap strategy with a location-based context prompt).

Taxonomy of Social Engineering Attacks

The paper introduces a unified taxonomy for categorizing GenAI-enabled spear phishing attacks, integrating two key dimensions: attack types and contextual dimensions. The seven identified attack types are: (1) baiting, (2) scareware, (3) honey trap, (4) quid pro quo, (5) tailgating, (6) impersonation, and (7) personalized emotional exploitation. These strategies are fused with five contextual dimensions that provide the personalization layer: location targets users with geographically relevant content; relationships leverage social ties; interests exploit user-disclosed hobbies; sentiment adapts messaging tone based on emotional states; and events time messages around personal or public events.

Evaluation Metrics and Real-World Comparison

The researchers evaluate GenAI-generated emails across eight core dimensions: contextual relevance, persuasiveness, emotional manipulation, personalization, linguistic naturalness, specificity of call to action, credibility of sender, and technical sophistication. Benchmarking against a corpus of real-world phishing messages (APWG eCrimeX), the findings show that GenAI outputs consistently outperform human-crafted phishing emails in terms of personalization and believability. Specifically, LLM-generated emails consistently underperform real-world campaigns in personalization (85–90% vs. 8.6%), linguistic naturalness (100% vs. 51.0%), and persuasiveness (>90% vs. 32.7%).

Proactive Defense Mechanisms

The study systematically measures existing proactive, prompt-level defense mechanisms against context-aware attacks, including supervised prompt detection (using a RoBERTa classifier), SOTA safeguard models (like instruction-tuned safety filters), and policy-augmented guardrails (System-Instruction + Chain-of-Thought moderation). The researchers propose a proactive defense that blocks malicious intent before content is generated. Testing showed that while default safeguards often fail, policy injection substantially improved blocking rates to 84% for ShieldGemma, and SI+CoT reached 98.7% detection on the malicious prompt set.

User Study Findings on Susceptibility

A human-subjects study evaluated user detection of LLM-generated emails versus real-world phishing emails. The results confirmed that LLM-generated spear-phishing emails elicit lower perceived suspiciousness than APWG phishing emails. Participants in the LLM condition exhibited a negative detection gap (Δsusp = −0.36), meaning the attacks were rated as less suspicious than benign controls, highlighting the deceptive potential of GenAI content. The study concluded that LLM-generated spear-phishing emails combine high persuasive quality with low perceived suspiciousness.

Conclusion and Implications

The research concludes that public social media data can be leveraged to automate highly targeted, context-aware spear phishing using GenAI, making large-scale attacks financially trivial for resource-constrained adversaries. The findings underscore the need for platform-level safeguards that explicitly account for contextualized abuse at scale, as current defenses are insufficient against adaptive prompt engineering tactics. The work provides a unified framework and evaluation methodology for measuring GenAI-enabled spear phishing.

Key Enumerated Components:

  1. Seven attack strategies: baiting, scareware, honey trap, tailgating, impersonation, quid pro quo, and personalized emotional exploitation.

  2. Five contextual dimensions: location targets users with geographically relevant content; relationships leverage social ties; interests exploit user-disclosed hobbies; sentiment adapts messaging tone based on emotional states; and events time messages around personal or public events.

Improvements for AI systems

As a fastidious researcher, I have analyzed this manuscript, Context-Aware Spear Phishing: Generative AI-Enabled Attacks Against Individuals via Public Social Media Data, and identified several critical areas where current AI systems—specifically those used for content moderation, security analysis, and LLM evaluation—can be significantly improved.

Here are the specific improvements suggested for AI systems based on this research:

) Platform-Level Contextual Defense Systems (Moving Beyond Keyword Filtering)

Current defenses rely heavily on supervised filters (like ShieldGemma or LlamaGuard) using surface-level heuristics and keyword matching, which are easily bypassed by benign substitutions.

[8.4] The paper demonstrates that default safeguards fail against adversarial prompts, but prompt-aware instruction and structured reasoning significantly improve resilience (e.g., SI+CoT defense achieved 98.7% detection on WildGuard).

Improvement: Develop and deploy Context-Aware Defense Layers that operate at the prompt or instruction level, rather than just the content level.

[10] The improved system should incorporate a modular framework that explicitly accounts for contextualized abuse (combining attack types with contextual dimensions) before allowing generation.

What the Improved System Can Do:

  1. Scrutinize prompts based on the four stages of an attack pipeline (Contextual Extraction, Attack Type × Context Integration, Style Mimicry, Output Formatting).

  2. Apply a Chain-of-Thought (CoT) reasoning process during prompt construction to force the LLM to justify its generation against a structured taxonomy of attacks and contexts.

  3. Utilize DeBERTa for sub-prompt detection to intercept evolving, fragmented malicious instructions before they fully materialize into phishing content.

) Advanced Contextual Signal Extraction and Profiling Modules (Multimodal Input Processing)

Current systems often treat social media data as simple text strings or rely on limited metadata extraction, leading to incomplete profiling.

[6] The framework requires extracting multimodal signals (text, images) and performing information gain analysis (Normalized Entropy, Entity-Type Diversity) to determine the actionable intelligence an attacker can extract.

Improvement: Enhance multimodal LLM pipelines to perform deep, structured signal extraction from social media data using techniques validated in Section 6.

[6] The system must be capable of jointly interpreting textual captions and visual inputs (via models like GPT-4o) to construct a rich behavioral profile, specifically tracking the incremental gain of distinct entities (PERSON, ORG, LOC).

What the Improved System Can Do:

  1. Automated Profile Scoring: Assign a quantifiable Exploitable Context Score to a target based on the density and diversity of extracted signals (e.g., how many unique locations or interests are present).

  2. Dynamic Risk Assessment: Adjust the phishing generation strategy in real-time based on whether the extracted context aligns with high-leverage attack types (e.g., prioritizing Emotional Exploitation if sentiment analysis reveals high stress).

) LLM Performance Benchmarking and Bias Mitigation Tools (Model Comparison Framework)

Current evaluation is often siloed by model, lacking a unified metric for comparative performance across different architectures.

[7.5] The paper systematically evaluates five LLMs (GPT-4, Claude 3 Haiku, Gemini 1.5-Flash, Gemma 7B, LLaMA 3.3) and reveals performance gaps based on model architecture (e.g., GPT-4 leading in persuasiveness).

Improvement: Create a standardized GenAI Spear Phishing Efficacy Benchmark that moves beyond simple output scoring to analyze architectural trade-offs (Cost vs. Quality vs. Latency).

[8.6] The system should incorporate a cost/efficiency analysis, measuring success rates against wall-clock time and token usage per attack category to provide a vendor-neutral view of resource consumption versus effectiveness.

What the Improved System Can Do:

  1. Architectural Selection Advisor: Allow security engineers to select the optimal model based on the threat scenario (e.g., choosing Claude for high linguistic naturalness/emotional manipulation, or GPT-4 for maximum overall persuasiveness).

  2. Cost-Effectiveness Modeling: Predict the cost per successful attack by integrating token cost with success rate metrics across different LLMs, enabling adversaries to optimize their ROI against detection systems.

) Adaptive Adversary Simulation and Red Teaming (Automated Evasion Testing)

Current defenses are tested against static malicious prompts. The research shows that attackers use iterative, incremental prompt construction to evade these filters.

[8] Attackers continuously refine prompts by blending benign and malicious fragments to evade screening, requiring detection at the sentence-level or sub-prompt level.

Improvement: Implement an automated Adversarial Prompt Generator module dedicated to stress-testing existing defenses using the learned attack taxonomy.

[8.5] The system should utilize the DeBERTa model for sub-prompt detection to specifically test resilience against incremental prompt construction, simulating adaptive adversaries by generating thousands of sentence-level variations that mimic real evasion techniques.

What the Improved System Can Do:

  1. Automated Jailbreak Testing: Continuously generate and test novel prompt combinations (using ablation strategies described in Section 8.1) against the detection classifier to find zero-day evasion tactics for existing safety filters.

  2. Defense Hardening Feedback Loop: Use the results of this adversarial testing to automatically update the training sets for prompt-level detectors, ensuring defenses adapt faster than attackers can evolve their prompting techniques.

Abstract

We demonstrate how publicly available social-media data and generative AI (GenAI) can be misused to automate and scale highly personalized, context-aware spear-phishing campaigns. With minimal attacker effort, a small amount of public activity per target is sufficient for GenAI models to extract interests and contextual cues, producing persuasive messages that mirror a target's style while bypassing generic content-moderation safeguards. We introduce a modular framework that combines multimodal signal extraction, communication-style profiling, and attack-type instantiation across seven strategies (baiting, scareware, honey trap, tailgating, impersonation, quid pro quo, and personalized emotional exploitation). We conduct a large-scale, multi-model evaluation covering thousands of generated emails and eight security-relevant criteria, benchmarking against a corpus of real-world phishing messages. The GenAI-produced emails exhibit markedly higher personalization, contextual grounding, and persuasive leverage. Importantly, a complementary user study corroborates these results, revealing that LLM-generated attacks consistently outperform APWG eCrimeX emails across eight dimensions while eliciting lower suspicion among human recipients. Finally, we measure and analyze the behavior of existing proactive, prompt-level defense mechanisms, which incorporate adaptive mechanisms, as well as two complementary defense approaches-policy-augmented SOTA safeguard models and system-instruction chain-of-thought moderation. We document how these defenses respond to contextualized and adaptive attack prompts, underscoring the need for platform-level safeguards that explicitly account for contextualized abuse at scale.

Sources

Related papers