How Do You Choose Your AI Component? An Interview Study of Secure AI Integration in Practice
Mahzabin Tamanna, Elizabeth Lin, Sparsha Gowda, Laurie Williams, Dominik Wermke
cs.SE, cs.AI, cs.CR, cs.IR, cs.LG
Submitted: 2026-07-18
Comments: 18 pages, 3 figures/tables
License: http://creativecommons.org/licenses/by-nc-sa/4.0/
The gist: The increasing adoption of Large Language Models (LLMs) as AI components in modern software systems introduces distinct security risks to the software supply chain.
Terminology
Abstract
The increasing adoption of Large Language Models (LLMs) as AI components in modern software systems introduces distinct security risks to the software supply chain. While many considerations and safety mechanisms are in place for components of the traditional software supply chain, the recent rapid adoption of AI components and platforms has overlooked these hard learned lessons. Selecting and integrating AI models without clear guidance on how these choices affect system security may leave applications vulnerable to threats, such as malicious components, data leakage, and unintended behavior. The goal of this study is to understand practitioners' decision making process and security considerations in selecting and integrating AI components through an exploratory semi-structured interview study. Toward this goal, we conducted semistructured interviews with 22 software developers, architects, and AI practitioners across diverse organizations about how they integrate AI components into their software. Our analysis finds that practitioners' model selection is predominantly driven by functional criteria, including performance, accuracy, cost, and specific features, e.g., tool calling or multimodal support, while security is rarely considered as an evaluation criterion. We observe a consistent lack of security concern throughout the AI component integration process, with established software supply chain lessons overlooked or ignored. The industry is repeating the historically costly mistakes of early software dependency management, prioritizing rapid reuse and availability over security and provenance. We distill our findings into actionable recommendations for AI adopters, model providers, and researchers, advocating for a proactive, security-by-design approach that integrates security evaluation into component selection and sustains it throughout the software development lifecycle.
Sources
- AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents
- An Empirical Analysis of the Python Package Index (PyPI)
- "I Always Felt that SomethingWasWrong.": Understanding Compliance Risks and Mitigation Strategies when Highly-Skilled Compliance Knowledge Workers Use Large Language Models
- SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Tasks
- Prompt Injection attack against LLM-integrated Applications
- Ignore Previous Prompt: Attack Techniques For Language Models
- LLMs' Reshaping of People, Processes, Products, and Society in Software Development: A Comprehensive Exploration with Early Adopters
- Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)
- Jailbreak Attacks and Defenses Against Large Language Models: A Survey
- Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents
Related papers
- Falsification-Based Verification of LLM-Generated Optimization Models: Sound Test Batteries and Their Detection Limits
- GitSkills: A Dataset of Agent Skills on GitHub
- SABER: Benchmarking Operational Safety of LLM Coding Agents in Stateful Project Workspaces
- PackMonitor: Enabling Zero Package Hallucinations Through Decoding-Time Monitoring
- IntentCoding: Amplifying User Intent in Code Generation
- Incentives and Outcomes in Bug Bounties