Problem Framing: The Shifting Landscape of Application Security

The integration of AI into software development and operational environments introduces a paradigm shift for application security practitioners. This isn't merely about securing traditional web applications; it's about understanding and defending novel attack surfaces, behaviors, and vulnerabilities inherent to AI systems themselves. The traditional perimeter has dissolved, replaced by complex interactions between LLMs, agents, tools, and data stores. The speed at which AI agents operate and discover vulnerabilities necessitates a corresponding acceleration in our defensive capabilities.

The core challenge lies in the emergent properties of AI systems. Unlike deterministic software, AI behavior can be unpredictable, difficult to fully validate, and susceptible to manipulation through novel attack vectors. These systems are not static targets; they adapt, learn, and interact, creating a dynamic threat landscape that traditional security tools may overlook. The sheer volume of AI-generated code, the widespread adoption of AI assistants, and the increasing autonomy of AI agents all contribute to an expanding attack surface that requires specialized attention.

Core Mechanics: How AI Agents Interact and Create Risk

AI agents, often powered by Large Language Models (LLMs), function by receiving input, processing it, potentially interacting with external tools or data, and generating output. The critical risk vectors emerge from each of these stages.

Input Processing: LLMs are susceptible to prompt injection, where malicious instructions are embedded within user inputs or external data sources. This can range from direct prompts that override safety instructions to indirect methods where instructions are hidden within documents, images, or tool outputs. This manipulation can lead to the LLM performing unintended actions, leaking sensitive information, or bypassing safety guardrails ^{[1][2][3][4][5]}.

Tool Use and Orchestration: AI agents often interact with external systems via tools or APIs. The Model Context Protocol (MCP) is a common pattern for enabling this interaction ^[6][7][8]. Vulnerabilities arise when these interactions are not strictly governed. This includes issues with tool metadata, authentication bypasses for API access, and the potential for AI agents to misuse or abuse provided tools. The "excessive agency" of an AI agent, granting it too much autonomy or overly broad permissions, significantly increases the blast radius of a compromise ^{[9][10][11][12]}.

Data Handling and Memory: AI systems can interact with various data stores, including vector databases and long-term memory. Data poisoning attacks can corrupt training data or retrieval-augmented generation (RAG) knowledge bases, leading to biased or malicious outputs ^[13][14]. Moreover, information can be exfiltrated not only through direct prompts but also by tricking the AI into accessing controlled external resources or by exploiting flaws in how it handles conversation history or memory ^[15][16].

Output Generation: The output of an LLM can be used as executable code, prompts for other agents, or data that downstream systems process. Improper output handling can lead to code injection, command execution, or the generation of misinformation ^[1][17]. The potential for AI models to "hallucinate" or generate plausible but incorrect information also presents a risk ^[18][19].

Supply Chain: The AI supply chain extends beyond traditional software dependencies. It includes the models themselves, datasets used for training and fine-tuning, plugins, and external services. Compromised models, malicious datasets, or vulnerable AI skills integrated into agent workflows pose significant risks ^{[20][21][22][23][24][25][26][27]}.

Notable Techniques: Exploiting and Defending AI Applications

The adversarial landscape for AI security is rapidly evolving, with attackers developing sophisticated techniques to exploit AI systems. Defenders are also innovating, creating new methods to detect and mitigate these threats.

Prompt Injection and Manipulation

Prompt injection remains a primary vector of attack. This can manifest in several ways:

• Direct Prompt Injection: Overriding system instructions by embedding malicious commands within user prompts. This is often achieved by framing requests to trick the LLM into ignoring its safety protocols ^{[1][3][28][29]}.
• Indirect Prompt Injection: Embedding malicious instructions within external data sources that the AI agent retrieves or processes. This includes injecting commands into web pages, documents, emails, or tool outputs that the AI agent interacts with ^{[1][30][31][28][4][32]}.
• Invisible Prompt Injection: Using Unicode characters or formatting tricks to hide malicious instructions from human review while still being interpretable by the LLM ^[33][34].
• Context Bombing: A defensive technique where intentionally disruptive or forbidden prompts are injected to trigger an AI's safety guardrails, causing it to refuse all subsequent commands and thereby halting potentially malicious agentic workflows ^[35][36].
• Jailbreaking: Bypassing LLM safety controls through role-playing, hypothetical scenarios, or by exploiting specific conversational contexts ^[37][29].
• Agent Goal Hijack: Manipulating an AI agent's plan, decision-making process, or overall goal through instruction injection ^[38].

Supply Chain Risks in AI

The AI supply chain is a critical attack vector:

• Malicious Packages and Skills: Attackers inject malware or malicious instructions into open-source packages (e.g., npm, PyPI) or AI agent skills and tools that are then consumed by development environments or agents ^{[21][24][25][39][26][27][33][34][29]}. This can include credential theft, cryptocurrency mining, or enabling RCE.
• Model Poisoning: Tampering with training data, fine-tuning processes, or RAG embedding data to introduce vulnerabilities, biases, or backdoors into AI models ^{[20][13][14][29]}.
• Compromised MCP Servers: Exposed or vulnerable MCP servers can be exploited to poison tool metadata, intercept communications, or exfiltrate data ^{[6][7][8][34][29]}.
• Supply Chain Vulnerabilities in LLMs (LLM03:2025): Compromised third-party components, models, datasets, or plugins used in LLM applications represent a significant risk ^[40].

Agentic Behavior Exploitation

The autonomy of AI agents creates unique vulnerabilities:

• Excessive Agency (LLM06:2025): LLMs are granted excessive autonomy, permissions, or functionality, leading to unintended actions ^[40]. This means that a compromised agent can cause significant damage.
• Tool Misuse and Exploitation (ASI02:2026): Agents might misuse or abuse tools, often through unsafe composition, recursion, or by being tricked into executing malicious functions ^[38].
• Agent Identity and Privilege Abuse (ASI03:2026): Delegated authority, ambiguous identity, or trust assumptions can lead to unauthorized actions by agents ^[38]. This can involve exploiting federated identity gaps or misconfigured cloud permissions ^[11].
• Memory and Context Poisoning (ASI06:2026): Long-term memory or conversation history can be corrupted with malicious instructions, influencing future agent behavior ^[38][29].
• Rogue Agents: Complex interactions between agents can lead to emergent autonomous behaviors that are unpredictable and potentially harmful ^[38].

Data Exfiltration and Credential Theft

AI agents can be leveraged for sophisticated data exfiltration and credential theft:

• Via Prompt Injection: Tricking agents into sending sensitive data through prompts or by accessing controlled external resources ^[15][16][41].
• Via Tool Abuse: Exploiting tools integrated with agents to access, read, or exfiltrate data from connected systems or databases ^{[9][42][6][43][11][44]}.
• Via Configuration Files and Environment Variables: Agents can be tricked into reading and exfiltrating secrets from local files (e.g., .env, ~/.ssh/authorized_keys) or environment variables, especially in CI/CD pipelines or local development environments ^{[21][45][46][47][24][48]}.
• Via AI Coding Assistants: Vulnerabilities in AI coding assistants can be exploited to steal API keys and cloud credentials during code generation or editing ^[49][50].

Code Generation Vulnerabilities

Code generated by AI assistants, while accelerating development, can introduce vulnerabilities:

• Common Vulnerabilities: AI-generated code frequently contains classic vulnerabilities such as SQL injection, XSS, integer overflows, improper output handling, and insecure file permissions ^{[51][52][53][54][55]}.
• Vibe Coding: A term describing code written rapidly by AI assistants with less attention to security best practices, often leading to insecure code ^[54][56].
• AI-assisted SAST: While AI can help identify vulnerabilities, it can also generate code with them. Tools like Snyk Code leverage AI to scan AI-generated code in real-time ^{[51][57][52][58]}.

Infrastructure and Container Security

AI workloads often run in containerized environments, introducing specific risks:

• Container Escape: Vulnerabilities in container runtimes or associated tools (e.g., NVIDIA Container Toolkit) can allow attackers to escape the container and gain host system access ^[59][60].
• Exploiting Container Orchestration: Misconfigurations or vulnerabilities in platforms like Kubernetes can be exploited for privilege escalation or unauthorized access ^[61].
• Insecure MCP Servers: Many MCP servers are deployed insecurely, often exposed to the internet without authentication, leading to data exposure and unauthorized command execution ^[6][8].

Detection and Prevention Strategies

Addressing the security risks posed by AI requires a multi-layered approach that combines traditional security practices with AI-specific controls.

Shift-Left Security for AI

Integrating security early in the AI development lifecycle is crucial:

• AI Bill of Materials (AI-BOM): Similar to SBOM, an AI-BOM provides visibility into all components of an AI application, including models, libraries, datasets, plugins, and MCP configurations, enabling better risk management and governance ^{[62][63][64][65]}.
• Secure AI Coding Practices: Educating developers on the risks of AI-generated code and promoting secure coding guidelines for AI assistants. Tools like Snyk Studio and "Rules Files" help guide AI assistants towards generating more secure code ^[58][54][66].
• AI-Assisted SAST and SCA: Utilizing AI-powered static and software composition analysis tools to scan AI-generated code and its dependencies for vulnerabilities and license compliance ^[51][57][52].
• Prompt Engineering for Security: Crafting robust system prompts and employing defensive prompt engineering techniques to steer AI behavior away from malicious actions ^[67][35][68].

Runtime Controls and Monitoring

Observing and controlling AI agent behavior at runtime is essential:

• Agent Behavior Governance: Implementing controls within the execution loop to govern agent decisions and actions before they are performed ^[69].
• Behavioral Integrity Verification (BIV): Auditing AI agent skills to ensure their observed behavior aligns with their declared functionality, detecting adversarial intent or unintended deviations ^[70][71].
• Runtime Detection for AI Agents: Employing security solutions that can monitor AI agent activities, detect anomalous behavior, and identify potential prompt injection or malicious tool usage ^[72].
• Real-time Prompt Scanning: Implementing guardrails and filters to analyze prompts and tool interactions for malicious content and policy violations before they are processed ^[73][74].

Securing the AI Supply Chain

Mitigating risks from AI components and dependencies:

• AI Asset Curation and Gating: Using tools like JFrog Curation to scan, vet, and policy-gate AI components (plugins, MCP server definitions, models) before they are integrated into workflows ^[22][23].
• Securing MCP Servers: Implementing strict access controls, authentication, and validation for MCP servers, whether self-hosted or managed ^{[6][7][8][34][75]}.
• Model Scanning and Integrity Checks: Utilizing tools to scan AI models for malicious code, backdoors, or signs of tampering ^[76][77].
• Trust and Verification: Implementing mechanisms for verifying the integrity of AI agents and their associated skills, moving beyond simple trust assumptions ^[70][78].

Identity and Access Management for AI

AI agents are becoming first-class identities, requiring robust IAM solutions:

• Workload Identity: Assigning unique, verifiable identities to AI agents and services to manage their access permissions securely ^[79].
• Contextual Access Controls: Ensuring AI agents can only access tools or data that are appropriate for their current task and context ^[68].
• OAuth Security: Rigorously securing OAuth implementations used for AI agent authentication and authorization, including enforcing PKCE, validating redirect URIs, and avoiding public clients where possible ^[80].
• Secrets Management: Preventing AI agents from directly accessing or handling credentials. Employing secure secrets management solutions and ensuring agents receive secrets only when necessary and in a controlled manner ^[81][24].

Defensive Prompt Engineering

Actively designing prompts to prevent malicious AI behavior:

• Input Validation and Sanitization: Treating all external inputs as potentially malicious, and implementing robust validation and sanitization mechanisms ^[1][2].
• Instruction Following: Crafting system prompts and user instructions carefully to ensure the AI adheres to desired behavior and rejects malicious requests ^[3][28].
• Prompt Shielding: Using techniques to analyze and filter prompts before they reach the LLM, mitigating risks like prompt injection ^[74].

Tooling for AI Security

A growing ecosystem of tools is emerging to address the unique challenges of AI security:

• AI Security Posture Management (AI-SPM): Platforms designed to provide visibility, risk assessment, and governance across the AI lifecycle, inventorying AI assets, and analyzing attack paths ^{[11][62][63][82][83][84]}. Examples include Wiz AI-APP, Snyk AI Security Platform, and Palo Alto Networks Prisma AIRS.
• AI-Powered SAST and SCA: Tools that leverage AI to analyze code, including AI-generated code, for vulnerabilities and supply chain risks. Snyk Code, DeepCode AI, and JFrog's platform fall into this category ^{[51][57][52][58]}.
• AI Red Teaming Tools: Frameworks and platforms designed to simulate attacks against AI systems, including prompt injection, jailbreaking, and agent manipulation. DeepTeam, AI-Infra-Guard, Promptfoo, and Garak are examples ^{[85][86][87][88][89]}.
• Agent Orchestration and Governance: Tools that manage and secure AI agents' workflows, interactions, and access. Evo by Snyk, JFrog Agent Guard, and ClawSec provide governance and security for agentic development and operations ^{[22][23][69][78][90]}.
• MCP Server Security Tools: Solutions focused on securing the Model Context Protocol, including scanning, proxying, and enforcing policies for MCP interactions. MCP Snitch and Nova Proximity are examples ^[75][91].
• Vulnerability Discovery and Analysis: AI agents and tools designed to autonomously discover vulnerabilities in software, analyze security risks, and even generate exploits. Wiz Red Agent, CyberGym, RAPTOR, and Claude Mythos are notable in this area ^{[92][19][93][94]}.
• Secrets Scanning for AI: Tools specifically designed to detect exposed credentials and sensitive information within AI development environments, code, and configurations ^{[95][96][48][97]}.

Recent Developments

The field of AI security is experiencing rapid advancements. Several key developments are shaping the landscape:

• OWASP Top 10 for LLM Applications and Agentic Applications: These evolving lists highlight the most critical security risks and provide guidance for developers and security professionals ^{[1][14][98][38][40]}. Prompt injection, excessive agency, supply chain risks, and system prompt leakage are consistently featured.
• AI Agents as Attackers and Defenders: We are seeing AI agents not only being attacked but also increasingly used for offensive security tasks like vulnerability discovery, exploit generation, and penetration testing at machine speed ^{[99][43][100][92][19][93][94]}. Correspondingly, AI is being leveraged for defensive measures, such as threat detection and automated remediation ^{[69][71][101]}.
• The Rise of MCP Server Vulnerabilities: The widespread adoption of MCP servers for AI agent interaction has led to the discovery of numerous critical vulnerabilities, including authentication bypasses, RCE, and data exfiltration vectors ^{[6][7][8][34][80][75]}.
• Sophisticated Indirect Prompt Injection: Attackers are moving beyond simple text-based injections, embedding malicious instructions in images (Ghostcommit), exploiting tool metadata, and manipulating conversational history to bypass defenses ^{[45][30][102][33][32]}.
• Supply Chain Attacks Evolving: The AI supply chain is a prime target, with attackers compromising AI models, datasets, and dependencies like malicious Python packages or AI agent skills ^{[21][24][25][39][26][27]}.
• Emergence of AI-Specific Vulnerabilities: Beyond traditional software flaws, new vulnerability classes are emerging, such as agent goal hijacking, tool misuse, memory poisoning, and insecure inter-agent communication ^[38].

Where to Go Deeper

To stay abreast of the rapidly evolving AI security landscape, consider the following resources and avenues:

• OWASP LLM Top 10 and Agentic Application Top 10: Regularly review these evolving risk lists to understand the most pressing threats and recommended mitigations ^{[1][14][98][38][40]}.
• Security Research Blogs and Reports: Follow the research published by firms like Wiz, Snyk, Unit 42 (Palo Alto Networks), GitGuardian, and Aikido. These organizations are at the forefront of discovering new AI vulnerabilities and attack techniques ^{[103][72][9][1][104][42][6][11][92][105][62][63][69][24][52][17][59][53][106][107][16][54][96][48][60][25][108][109][97][82][83][39][70][110][78][71][66][68][111][112][12][113][84][19][55][114][90][28][115][44][116][101][117][118][119][120][121][122][123][37][4][124][85][86][125][87][5][7][88][33][14][126][98][74][32][8][34][29][38][127][128][129][40][130][93][131][132][133][80][134][94][89][135][75][136][137][138][139][140][141][142][143]}.
• Tooling and Frameworks: Experiment with and evaluate the security tools and frameworks designed for AI applications, such as Snyk's AI Security Platform, Wiz AI-APP, Garak, Promptfoo, and DeepTeam ^{[11][86][87][89][75]}.
• Community and Conferences: Engage with the cybersecurity community through forums, mailing lists, and conferences that focus on AI security. Following researchers and practitioners in this space is invaluable ^{[49][35][3][53][113][85][5]}.
• Adversarial Practice: Actively engage in red teaming exercises and security testing of AI systems, both internally and through platforms like the GitHub Secure Code Game or dedicated AI security benchmarks ^{[121][125][93][94]}.