ORCAGen: Orchestrating Context-Aware Malware Deception with RAG-Guided Generative AI

arXiv:2610.12415 · cs.CR · Submitted 2026-10-08 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.

Nadia: Today's paper: "ORCAGen: Orchestrating Context-Aware Malware Deception with RAG-Guided Generative AI".

Elias: The gist: ORCAGen takes a different approach to malware defense by using GenAI to build malware-specific deception playbooks offline, validate them before deployment, and enforce only verified logic at runtime.

Nadia: First, who's behind it and why it matters.

Title and authors: Nadia: Moving on, what the paper actually summarizes is this whole approach of ORCAGen which combines Retrieval Augmented Generation with structured prompt engineering to generate both proof-of-concept malware and corresponding deception orchestration code offline >

Elias: It really boils down to using a curated knowledge base that maps malware procedures directly to active defense strategies, which serves as the foundation for grounding the entire generation process >

Priya: So, if I’m hearing this right, they aren't just letting the AI guess defenses; they are forcing it to retrieve specific rules about what a particular malware family does and then match those rules against known countermeasures >

Nadia: That’s right. They construct this structured knowledge base with Malware Procedures and Active Defenses, which is designed to provide the specific grounding needed for threat-specific deception generation >

Elias: And they use this knowledge base during the generation phase by retrieving relevant malware procedures and defense strategies and injecting them into structured prompt templates >

Priya: So, what does that mean for the practical application? It means instead of just asking an AI to write a generic defensive script, you feed it specific behavioral details from their database >

Nadia: Right. The paper highlights that this grounding helps the model generate code that is specific to the target malware behavior rather than something generic or hallucinated >

Elias: It moves the output away from being just text and towards being executable deception logic, which is a big step for making this practical >

The paper's summary: Nadia: Now let’s look at what the authors actually claim as improvements over previous methods. They focus on their new architecture that separates the playbook construction from runtime enforcement >

Elias: That separation is key because it means the LLM generates all the code and playbooks offline, while runtime deployment uses a pre-tested Super DLL that only enforces logic that has already passed rigorous validation >

Priya: So, to make sure I understand this improvement, if the system builds it offline, how do they ensure that when it runs live in a process, it’s actually running the exact same logic they tested before >

Nadia: They validate by first executing the PoC malware in a controlled environment to verify its behavior and then testing the orchestration code against that specific malware to see if it can redirect or suppress the activity >

Elias: The improvement here is that only deception strategies that pass this validation are compiled into a reusable playbook, which they call a Super DLL >

Priya: That means the system isn't just deploying whatever code the AI spits out; it’s enforcing only pre-verified, deterministic logic during runtime enforcement >

Nadia: It’s about moving away from live LLM inference during malware execution, which would introduce a lot of latency and safety concerns in a real-time scenario >

Elias: And they also point out that by combining RAG and structured prompt engineering this way, ORCAGen is the first framework to combine GenAI and RAG for offline generation, validation, and compilation of malware-specific deception playbooks >

The paper's improvements: Nadia: So wrapping up on ORCAGen: they’ve shown a system that uses AI to build these malware-specific deception playbooks offline, validates them thoroughly before deployment, and only enforces the verified logic at runtime through a Super DLL >

Elias: The paper emphasizes that this separation of generation and enforcement is what makes it more deployable compared to systems where you might be relying on just direct prompting or RAG alone >

Priya: From a measurement standpoint, what this means for us is that the effectiveness across different real-world malware families showed strong results, neutralizing ninety-two percent of keyloggers and ninety-six percent of ransomware samples based on their testing >

Nadia: It suggests that these playbooks can transfer from synthesized PoC validation to real-world malware when those malicious behaviors interact with monitored input or API targets >

Elias: The performance measurements showed that GPT-five point five was strongest for rapid playbook construction, while Gemini three point five Flash was the most efficient in terms of response time and overhead >

Priya: But they also noted a limitation, which is that the method doesn't fully cover every possible edge case, and it relies heavily on the quality and completeness of that initial curated knowledge base >

Nadia: That’s a fair point. So to recap, ORCAGen uses RAG-grounded structured prompting for behaviorally consistent PoC generation, an offline playbook validation workflow where they test the deception in a controlled environment, and separates generation from enforcement via a validated Super DLL >

Elias: It’s an interesting framework because it addresses the challenge of needing threat-specific countermeasures without requiring you to manually write every single line of interception code >

Priya: It shows that combining generative capabilities with structured knowledge retrieval can yield very effective, targeted defenses when you have the right data structure to ground the AI >

Conclusion: Nadia: So we’re done with ORCAGen: Orchestrating Context-Aware Malware Deception with RAG-Guided Generative AI. Essentially, they built a system that uses Retrieval Augmented Generation to create malware deception logic offline and then rigorously tests it before letting it run live >

Elias: Exactly. The core idea is grounding the AI in specific knowledge about malware procedures and defense strategies so it doesn't just spit out generic garbage >

Priya: From what I’m seeing with the data, these playbooks showed strong effectiveness against three different real-world malware families, neutralizing keyloggers and ransomware pretty well >

Nadia: And the numbers are interesting—they found that GPT-five point five was best for quickly building those playbooks, while Gemini three point five Flash was fastest in terms of processing speed and low overhead >

Elias: I’m looking at the mechanics here, and it seems like they really nailed the separation between building the code offline and actually enforcing it during runtime with that Super DLL >

Priya: The real value for someone listening is seeing how this moves from a theoretical idea to something you can actually test against actual malicious behavior before you deploy it in production >

Nadia: It means we’re not just guessing defenses anymore; we’re using AI to generate and then validate the specific logic needed to disrupt malware in a controlled way >

Elias: It shows that this structured approach, using those two different knowledge bases for procedures and active defenses, is what lets the model produce code that actually makes sense >

Priya: I just wonder how robust it is when you move beyond just those three families they tested; does it generalize well to totally new attack techniques >

Nadia: That’s a fair question. The authors did flag that the system relies heavily on the quality of their initial knowledge base, so expanding that data will definitely be key for future work >

Elias: It makes sense. They also mentioned using iterative refinement prompts to clean up any syntax errors in the generated code without needing a human to manually edit everything >

Priya: So the takeaway is that this approach gives us a much more reliable way to test and deploy active deception logic for things like file system hooking or API-level targets >

Nadia: It definitely shifts how we think about defense, moving towards an AI that can build and test specific countermeasures without needing constant manual coding >

Elias: Yeah, ORCAGen shows how you can use generative AI not just to write text, but to construct executable orchestration code that actually gets tested against threats >

Priya: Anyway, we’ll take a quick break and then we’ll look at some of those papers on black-box forensics for conversational LLM agents next.

Shihab Ahmed, Md Sajidul Islam Sajid, Teryl Taylor, Frederico Araujo, Tariqul Islam

Towson University · IBM Research

cs.CR

Submitted: 2026-10-08

Updated: 2026-10-08

Comments: Accepted at 2026 IEEE 8th International Conference on Trust, Privacy and Security in Intelligent Systems, and Applications (TPS-ISA)

Code: https://github.com/sahmed09/ORCAGen

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

The gist: The gist: ORCAGen takes a different approach to malware defense by using GenAI to build malware-specific deception playbooks offline, validate them before deployment, and enforce only verified logic

Key concepts

Retrieval-Augmented Generation (RAG)
RAG combines searching a curated knowledge base with generative AI. ORCAGen uses this to ground the LLM in concrete malware behaviors and corresponding defense strategies. This prevents generic or made-up outputs by ensuring the generated code is based on specific, factual information retrieved from structured data.
Knowledge Base (KB) Structure
ORCAGen uses two structured knowledge bases: one for Malware Procedures (how malware acts) and another for Active Defenses (how to counter it). Each entry includes details like the malware family, attack technique, behavior description, and a specific deception strategy. This layered structure ensures that both the generated malware sample and the orchestration code are highly relevant to each other.
Offline Validation Loop
Before any deception logic is deployed, ORCAGen tests it offline. It first executes a proof-of-concept (PoC) malware sample to confirm its intended behavior. Then, it runs the generated orchestration code against this PoC to verify if the deception strategy successfully disrupts or redirects the malware's actions. Only validated strategies proceed.
Runtime Enforcement via Super DLL
The validated deception logic is compiled into a reusable 'Super DLL'. This module is then injected into target processes at runtime. It enforces pre-tested responses, actively monitoring behavior and applying the specific redirection, suppression, or misleading actions determined during the offline validation phase.

Terminology

Summary

The gist: ORCAGen takes a different approach to malware defense by using GenAI to build malware-specific deception playbooks offline, validate them before deployment, and enforce only verified logic at runtime.

How it works

ORCAGen combines Retrieval-Augmented Generation (RAG) with structured prompt engineering to generate both proof-of-concept (PoC) malware and corresponding deception orchestration code during the offline phase. A curated knowledge base (KB) of malware procedures and active defense strategies grounds the generation process, helping the LLM produce threat-specific and executable deception logic rather than generic or hallucinated outputs.

The framework separates playbook construction from runtime enforcement. During the offline phase, it uses RAG and structured prompt engineering to generate PoC malware and orchestration code. The generated PoC malware provides a safe and reproducible way to test whether a deception strategy can disrupt, redirect, or suppress targeted malware behavior before the strategy is added to the runtime playbook.

Knowledge Base Creation for RAG

ORCAGen uses two structured knowledge bases (KBs) to ground generation in concrete malware behavior and corresponding deception actions. The first KB, Malware Procedures, captures how a malware family performs a target behavior. The second KB, Active Defenses, maps those behaviors to deception strategies that can redirect, suppress, or mislead the malware

To populate the KBs for this work, a collection workflow was used that combined lexical search and semantic search. The final KBs are stored in structured JSON format and indexed for retrieval during RAG Each entry contains four core fields: i) malware family, ii) attack technique, iii) behavioral description, and iv) deception strategy This layered structure is important because PoC generation requires low-level behavioral details while orchestration generation requires defense strategies that can generalize across related malware behaviors

Prompt Engineering and Code Generation

ORCAGen uses structured prompt templates that specify the task, output constraints, retrieved context, and examples needed for malware-specific generation. The initial prompt defines a project level scaffold describing three components: a controlled PoC malware sample, a DLL-based API hooking module, and an injector for deploying the DLL into a target process.

ORCAGen applies the template in two generation stages. In the first stage, it is grounded with malware procedure entries to generate a controlled PoC that reproduces a specific behavior. In the second stage, it combines the generated PoC with active defense entries to generate deception orchestration code.

Offline Validation and Runtime Enforcement

ORCAGen validates generated deception logic before adding it to the runtime playbook. After PoC and orchestration code are generated, the PoC is first executed in a controlled environment to verify that it reproduces the intended behavior. The orchestration code is then compiled and executed against the PoC to determine whether it can redirect, delay, suppress, or mislead the malware behavior

Only deception strategies that pass validation are compiled into a reusable playbook implemented as a Super DLL. During runtime enforcement, the Super DLL is injected into suspicious or policy-selected processes and enforces pre-tested deception responses.

Evaluation Results

The evaluation across GPT-5.5 showed that it required the fewest refinements (0.07/task), produced no observed hallucinations, and maintained low overhead (27.69ms) with a 54.21s response time. Gemini 3.5 Flash achieved the fastest response time (34.92s) and lowest overhead (18.32ms) while requiring only 0.27 refinements and producing no hallucinations.

Against real-world malware samples, ORCAGen achieved strong effectiveness across all three real-world malware families. It completely neutralized 92% of keyloggers, 100% of information stealers, and 96% of ransomware samples. This suggests that ORCAGen’s playbooks can transfer from synthesized PoC validation to real-world malware when malicious behavior interacts with monitored input, credential, file-system, or API-level targets.

Comparison with Other Systems

ORCAGen outperformed SPADE across all evaluated models and refinement thresholds. Compared with direct prompting and RAG-only prompting, ORCAGen reduces refinement effort. Compared with SPADE, ORCAGen improves deployability by testing generated orchestration code against active malware behavior before adding it to the playbook

The performance measurement shows that GPT-5.5 was strongest for rapid playbook construction, Gemini 3.5 Flash was most efficient in response time and runtime overhead, and Qwen3-Coder required the most post-processing <ref:2610.124199,GPT-5.5 was strongest for rapid playbook construction, Gemini 3.

Improvements for AI systems

  1. Bold header: RAG-grounded structured prompting for behaviorally consistent PoC generation. This improves AI systems by allowing LLMs to produce behaviorally consistent PoC malware and deception orchestration code without additional model training by grounding generation in a curated knowledge base of malware procedures and active defense strategies.

  2. Bold header: Offline playbook validation workflow. The system improves by implementing a process that adds only verified logic to the runtime playbook by (i) creating PoC malware, (ii) generating corresponding deception logic, and (iii) testing the deception in a controlled environment, ensuring only verified logic is added to the deployable playbook.

  3. Bold header: Separation of generation and enforcement phases. This system can now separate playbook construction from runtime enforcement, meaning the LLM generates code offline, while runtime deployment uses a validated Super DLL that enforces only pre-validated responses during malware execution, avoiding latency and safety concerns associated with live LLM inference during malware execution.

  4. Bold header: Structured knowledge base mapping for precise behavioral modeling. The KB structure, which maps malware procedures to active defense strategies, allows the system to model specific behaviors like Filesystem API Hooking or File I/O Hooking, enabling the generation of logic that is specific to the target malware behavior rather than generic or hallucinated.

  5. Bold header: Iterative refinement based on execution feedback. The system can be improved by using iterative refinement prompts, with no human edits to source code are allowed, allowing ORCAGen to correct specific errors like missing headers, ANSI/Unicode mismatches, C++ compatibility adjustments until runnable code is produced.

Sources

Related papers