AI Security Research Should Better Incentivize Defense Research

This work examines an imbalance in artificial intelligence (AI) security research, finding that the field tends to produce more work on attacking AI systems than on defending them. This disparity suggests a structural lag in defensive progress, as attack papers are often evaluated under favorable conditions while defenses are held to a stricter standard. The paper argues that AI security research should better incentivize defense research to ensure systems are not only good at showing how they fail but also good at building and validating ways to make them secure.

The Core Finding on Imbalance

The analysis is based on a quantitative assessment of cited studies from 16 Systematization-of-Knowledge (SoK) papers and 21 survey papers, covering over 1,100 attack/defense-classified citations. The primary finding is that attack papers outnumber defense papers by 1.24:1 across the SoK corpus and by an average of 2.15 in the survey papers. The authors contend this imbalance is not merely a count but reflects a structural issue where attack research is more readily published and rewarded than remediation efforts.

Patterns of Asymmetry Across Subfields

The imbalance is not uniform across all topics; it varies substantially depending on the specific area of AI security being studied. The paper identifies two distinct patterns:

1. The most attack-heavy SoKs are all organized around a specific attack class. For instance, the five most skewed cases focus on black-box adversarial attacks, data inference privacy attacks, membership inference against LLMs, neural network extraction via physical side channels, and anti-facial-recognition techniques. In these areas, literature is structured primarily around discovering, characterizing, and refining offensive capabilities.

2. The defense-heavy SoKs tend to emerge because they focus on a specific defense method, rather than being prominent in broad surveys of the threat landscape.

Structural Dilemmas for Defense Research

The paper outlines several dilemmas that contribute to the underincentivization of defensive work:

Novelty Asymmetry:

Attack papers can establish novelty by identifying a new attack vector, applying a known technique to a new target, or showing that an existing system fails under a previously untested threat model. This creates a structurally lower threshold for novelty on the attack side.

Evaluation Asymmetry:

Attacks and defenses are judged by different standards. An attack can succeed by breaking one version of one model under favorable conditions, whereas a defense is expected to remain effective across many models and datasets. Defenses are penalized for incompleteness in a way attacks are not, because defensive claims are implicitly universal.

Publication Asymmetry:

Attack papers often follow a structure that is easier to present and evaluate as a research contribution than incremental hardening (in defense research). Defense claims rely on the more fragile assertion that a method withstands all evaluated attacks or mitigates a particular class of failures under specified assumptions.

The Industry–Academia Gap

A further factor contributing to the imbalance is the gap between academic findings and industrial practice. The authors note that researchers are often not aware of industry defense solutions for certain model-extraction threat categories, suggesting that much of the most important work happens outside the kinds of outputs that paper counts can easily capture. This results in a situation where attacks are often more visible because they reveal external vulnerabilities, while companies have limited incentive to publicize their mitigations in comparable detail.

Conclusion and Proposed Incentives

The authors conclude that AI security research should better incentivize defensive capacity in a broader sense. They argue that the bottleneck is not solely a lack of defense papers but also issues with better translation of defensive ideas into systems that organizations can actually use. Therefore, the necessary investment includes stronger public defense research, more realistic defense evaluation, clearer pathways from defensive ideas to deployment, and more institutional support for mitigation-oriented work. The ultimate goal is for the field to be good at building, validating, and sharing ways to make them more secure and much safer.

Alternative Views Considered

The paper acknowledges alternative perspectives. One counterposition suggests that the imbalance might not be a problem if attack research is necessary to fully understand the main threats before defenses can be reliably developed. Another view posits that the issue is not a lack of defense papers, but rather better translation, meaning work focused on deployment and adoption is needed more than just publication counts. However, the authors maintain that without improving incentives around public defense work, this translational bottleneck alone is insufficient to fully resolve the imbalance.

Limitations

The analysis relies on cited SoK papers and surveys, which may not capture broader topics like safety or governance. Future work should aim for a more comprehensive analysis by incorporating larger independent corpora and triangulating findings with a wider literature base. Furthermore, the paper identifies that specifying which interventions would mitigate this imbalance remains an open question for future research.

Based on the provided paper, here are specific improvements that could be made to current AI security research and how these improvements would enhance AI systems:

1. Improving Defense Research Incentives:

We must transition from merely observing an imbalance to actively correcting the incentive structure. This involves:

2.1. Establishing First-Class Defense Contributions: Researchers should be explicitly rewarded for developing robust, deployable defenses (prevention, mitigation, certification) as primary scientific contributions, rather than solely for demonstrating novel attacks or vulnerabilities.

2.2. Standardizing Evaluation Criteria for Defenses: Develop rigorous, universal benchmarks that test defenses not just against isolated attacks but against adaptive adversaries across diverse models and real-world deployment conditions (e.g., testing robustness to batch size variations in FL or varying physical side-channel noise).

2.3. Bridging the Industry–Academia Gap: Create structured pathways for translating academic defensive findings into production-ready systems, including standardized engineering practices for hardening against identified vulnerabilities and sharing proprietary defense knowledge publicly where appropriate.

4. Enhancing AI System Capabilities (The Result of Improved Incentives):

The improved research focus will lead to AI systems that are fundamentally more resilient and trustworthy:

1. Robustness Against Adversarial Inputs: Systems will be significantly harder to fool by malicious, imperceptible perturbations (e.g., in vision or language inputs) because defenses will be developed that are not brittle and can withstand cherry-picked threat models, leading to safer deployment in safety-critical applications like autonomous driving.

2. Data Privacy and Model Integrity: AI models trained on sensitive data (like medical records or proprietary enterprise data) will have stronger inherent protections against inference attacks (Membership Inference Attacks) and model extraction, ensuring that sensitive training data remains private even when the model is deployed.

3. Trustworthy Deployment: Systems will move beyond mere accuracy metrics to include verifiable guarantees of performance under stress. This means AI systems can provide formal certification that they will operate reliably even when subjected to known classes of attacks or system manipulations, increasing their utility in high-stakes environments.

This work examines an imbalance in artificial intelligence (AI) security research: the field tends to produce more work on attacking AI systems than on defending them. Drawing on related academic papers, we find biased attack-to-defense ratios across subfields, including federated learning, speech recognition, membership inference, large language models, etc. The imbalance possibly means far beyond a simple count: attack papers are routinely evaluated under favorable conditions that make threats look more severe than they are in practice, while defenses are held to a stricter standard that few can meet. The result is a literature rich in demonstrated vulnerabilities and thin on usable and deployed protections. We thus argue that AI security research should better incentivize defense research.