CTIFoundry: An Agent-Native Corpus Scaffold for Cyber Threat Intelligence

Cyber threat intelligence (CTI) investigation is increasingly consumed by LLM agents, but current Retrieval-Augmented Generation (RAG) methods fail because they package threat reports as opaque chunks behind a similarity search interface. This paper introduces CTIFoundry, an agent-native corpus scaffold designed to solve this substrate bottleneck by materializing the latent structure of a CTI corpus at build time, thereby enabling agents to perform multi-step investigations with high accuracy.

How it works

CTIFoundry functions by creating a derived representation of the CTI corpus through a build-time pipeline that materializes three core artifacts: an ontology graph, a report layer, and hybrid retrieval surfaces. The ontology graph is a "deterministic ontology graph over four authoritative knowledge bases (CVE, CWE, CAPEC, ATT&CK) whose official cross-references become typed, traversable edges. This structure addresses the challenge of resolving aliases and sibling confusion by making official cross-references explicit as typed edges."

The report layer handles the narrative layer of CTI. It is a span-grounded report layer whose canonical, alias-resolved cross-vendor entities index provenance-carrying chunks. This ensures that every chunk has exact character-offset provenance, allowing the agent to verify claims by grounding them in specific text spans.

The hybrid retrieval surfaces provide flexible access. These surfaces fuse dense and lexical search under a term filter, allowing the agent to steer its search based on feedback, which addresses the failure mode where a gold entry sharing rare discriminative tokens with the query yet sitting far from it in embedding space.

The Scaffold Components

The scaffold is defined as a derived representation through which an agent accesses the corpus. It consists of:

1. An ontology graph (G) over knowledge base records and their typed official cross-reference edges (C1).

2. A span-grounded report layer (X), where chunks are indexed by canonical entities with vendor-attributed alias sets and grounding in the graph (C2).

3. Hybrid retrieval surfaces (I), fusing dense and lexical search under a steerable term filter (C3).

The Query-Time Interface

At query time, this structure is exposed through seven typed tools and three procedural skills mounted on a stock, widely-used open-source agent harness. The seven tools cover non-overlapping capabilities such as resolve entity name/alias/id, ontology neighbors traverse official cross-reference edges, and search kb hybrid dense+BM25 search over one KB. These are governed by rules like Structure before similarity: descriptions encode the substrate’s priority order (resolve names before querying, prefer an authoritative edge over text search whenever an identifier is known).

Complementing the tools are three procedural skills that encode investigation discipline. These skills are advice, not workflow engines, and they bind to structure that exists. For example, a skill for entity linking dictates: do NOT search the TARGET taxonomy first, because the description paraphrases one specific SOURCE entry; the authoritative cross-reference edge from that source entry gives the answer.

Empirical Validation

Evaluation on CTIConnect showed significant gains by swapping only the action surface. Swapping only the action surface lifts overall F1 from 0.610 to 0.829 for gpt-5.4 and from 0.470 to 0.745 for claude-haiku-4-5, demonstrating that structure makes the right investigation possible, procedure makes it reliable, and neither substitutes for the other. The ablation study shows that procedural skills recover +0.062 and structure alone recovers +0.136, resulting in a super-additive gain of +0.219 when both are used together.

Conclusion

The paper concludes that the binding constraint is the substrate, not the agent. CTIFoundry demonstrates that for any domain whose corpora carry authoritative reference structure, the highest-leverage investment in agent quality may not be a better agent at all, but a corpus deliberately built to be investigated. This approach ensures that an autonomous investigator can traverse and verify information anchored to authoritative taxonomies.

The gist

CTIFoundry materializes the latent structure of a CTI corpus—a deterministic ontology graph, span-grounded report layer, and hybrid retrieval surfaces—at build time, exposing it at query time through seven typed tools and three procedural skills to solve the substrate bottleneck in agentic CTI investigation.

Key Procedural Skills

The three procedural skills are task-specific playbooks injected into the user turn:

1. Entity Linking (RCM, ATD, ESD): Focuses on mapping behavioral descriptions to taxonomy entries by prioritizing authoritative cross-reference edges over direct target taxonomy searches.

Based on the provided research paper, CTIFOUNDRY: AN AGENT-NATIVE CORPUS SCAFFOLD FOR CYBER THREAT INTELLIGENCE, here are specific improvements for AI systems and what those improved systems can achieve:

The core improvement is shifting the bottleneck from model capability to corpus structure. Instead of relying on general retrieval-augmented generation (RAG) over flat, opaque chunks, the system should use CTIFoundry as an agent-native corpus scaffold.

Here are the specific improvements and capabilities:

2. Use a build-time pipeline that materializes a deterministic ontology graph over four authoritative knowledge bases (CVE, CWE, CAPEC, ATT&CK) into typed, traversable edges (C1).

3. Enforce a zero-fabrication invariant during the build process by checking all identifiers against pinned snapshots at write time. This guarantees that every identifier generated is verifiable and prevents fabricated entities from entering the system (RQ1).

4. Implement a span-grounded report layer (C2) that indexes vendor reports with exact character-offset provenance, resolving canonical cross-vendor entities based on deterministic signals before indexing chunks. This ensures derived claims are auditable back to the specific text and source span (C2).

5. Utilize hybrid dense+lexical retrieval surfaces (C3) that fuse embedding similarity search with term filtering, allowing the agent to steer the search pool size based on query feedback. This closes the failure mode where gold entries share rare tokens but are far from them in embedding space (C3).

6. Integrate a query-time interface consisting of seven typed tools (e.g., resolve entity name/alias/id, ontology neighbors traverse official cross-reference edges) and three procedural skills (e.g., resolve before searching, consult an alias set). This structure is exposed through a stock agent harness (C4).

7. Ensure the agent's action surface prioritizes authoritative structure over textual similarity when an identifier is known, explicitly teaching it to prefer recorded edges over semantic retrieval (RQ4/E.5).

The improved AI system, powered by CTIFoundry, can perform the following specific tasks:

2. Perform multi-step CTI investigations that require resolving aliases across different vendor reports (e.g., Identify the malware family of 'Hidden Cobra'). The system will use its built-in ontology graph to follow official cross-references, moving beyond simple textual similarity to find the authoritative answer.

3. Ground derived assertions in precise evidence by citing specific text spans from original reports (e.g., Which CTI report states that the malware uses CVE-2024-21412?). This allows for verifiable attribution of claims, which is critical for defensive triage and attribution (C2).

4. Accurately map behavioral descriptions in narrative prose to official attack patterns (e.g., "Map this description to an ATT&CK technique"). The system will use procedural skills that enforce a strict decomposition of the passage into atomic behaviors before searching the taxonomy, ensuring it selects the most specific, authoritative technique rather than a plausible but wrong one (H.3).

5. Precisely resolve and distinguish between near-miss entities with similar names across different vendors (e.g., distinguishing between two different ransomware families that share a similar name). The system uses deterministic logic (union-find over exact/span-verified evidence) to prevent state actors from collapsing into incorrect mega-clusters during entity linking (G.4).

6. Synthesize multi-document campaign profiles by systematically reading all listed reports, resolving every alias across vendors, and ordering events strictly by the dates stated in the text, attributing claims precisely to their source vendor (H.5). This allows for a complete, auditable timeline of a threat actor's activity.

7. Execute complex research workflows that scale efficiently: at 1.7× the question volume, accuracy is maintained because the structure provides reliable anchors for retrieval recall, and cost remains linear (E.3).

In summary, the improved AI system moves from being a sophisticated search engine to an agent that performs high-stakes CTI investigation by leveraging a pre-built, deterministic knowledge scaffold that enforces structural validity and procedural discipline.

Cyber threat intelligence (CTI) is increasingly consumed not by human analysts but by LLM agents that compose multi-step investigations at query time. The harness side of this shift has matured rapidly, but the corpus side has not: threat reports and vulnerability databases are still packaged for retrieval-augmented generation, as opaque chunks behind an embedding index. We argue that this substrate, not model capability, is the bottleneck on agentic CTI investigation, and present CTIFoundry, an agent-native corpus scaffold. At build time, CTIFoundry materializes the latent structure of a CTI corpus: a deterministic ontology graph over four authoritative knowledge bases (CVE, CWE, CAPEC, ATT&CK) whose official cross-references become typed, traversable edges; a span-grounded report layer whose canonical, alias-resolved cross-vendor entities index provenance-carrying chunks; and hybrid dense+lexical retrieval surfaces. At query time this structure is exposed through seven typed tools and three procedural skills mounted on a stock, widely-used open-source agent harness. On the public CTIConnect benchmark, swapping only the action surface lifts the identically-harnessed agent from 0.610 to 0.829 overall F1 with gpt-5.4 and from 0.470 to 0.745 with claude-haiku-4-5: a small model on CTIFoundry surpasses a flagship model on the flat substrate. The scaffolded agent is simultaneously more accurate and more efficient: on both Claude models it answers with roughly half the tool calls per question. The ablation distills design principles for matching corpus scaffolding to data modality, in CTI and beyond. Build-time validation guarantees zero fabricated identifiers by construction, and the scaffold sustains 1,168 investigations end-to-end at about 2.6 cents each.