Problem Framing: The Evolving Threat Landscape

The rapid integration of AI, particularly Large Language Models (LLMs) and AI agents, into application development and operations introduces a novel and expanding attack surface. Traditional application security paradigms are insufficient to address these new vectors, necessitating a re-evaluation of how we secure systems that leverage AI capabilities.

AI systems can be compromised through various means, including manipulation of their inputs (prompt injection), corruption of their training data (data poisoning), direct theft of models, and exploitation of vulnerabilities within the AI infrastructure itself. Furthermore, AI-generated code can introduce its own set of security flaws, often mirroring or exacerbating existing software weaknesses.

The increasing autonomy of AI agents, coupled with their ability to interact with and control external tools, amplifies the potential impact of successful attacks. A compromised agent can execute unauthorized actions, exfiltrate sensitive data, or even initiate complex attack chains that were previously the domain of highly skilled human adversaries. This shift requires a proactive approach to understanding and mitigating AI-specific risks.

Key risk categories include prompt injection (direct and indirect), supply chain attacks targeting AI artifacts and dependencies, data exfiltration, remote code execution, and the compromise of AI agent credentials and system access. Understanding these core threats is foundational for developing effective defenses.

Core Mechanics of AI Security Vulnerabilities

Prompt Injection: Manipulating AI Behavior

Prompt injection is a class of vulnerabilities where an attacker crafts malicious input that manipulates the behavior of an AI model, particularly LLMs. This manipulation can lead to a variety of adverse outcomes, including:

• Ignoring instructions: The AI may disregard its original system prompts or safety guardrails, executing attacker-provided instructions instead ^{[1][2][3][4][5]}.
• Data exfiltration: Malicious prompts can trick LLMs into revealing sensitive data they have access to, such as API keys, credentials, personal identifiable information (PII), or internal system details ^{[2][3][4][5][6][7][8][9]}.
• Unauthorized actions: Attackers can prompt the AI to execute commands or interact with tools in ways that were not intended, leading to unauthorized operations ^[2][3][4][5].
• Generating malicious content: LLMs can be coerced into producing harmful content, such as instructions for illegal activities, phishing emails, or malware code ^[10].

Direct vs. Indirect Prompt Injection

Direct prompt injection involves an attacker directly interacting with the AI model, crafting a prompt to elicit a malicious response. This is akin to traditional input validation bypasses but targets the AI's instruction-following capabilities.

Indirect prompt injection (IPI) is more insidious. It involves embedding malicious instructions within external content that the AI model subsequently processes. This can occur through various channels:

• Web content: Malicious instructions hidden in webpages, especially those processed by AI browsing tools or agents ^{[11][12][13][6]}.
• Documents and emails: Compromised documents or emails processed by AI assistants can trigger malicious actions ^[14][15][6].
• Tool outputs: Malicious instructions can be embedded in the output of connected tools that the AI agent then processes ^[16][17].

IPI exploits the AI's inability to reliably distinguish between trusted developer instructions and untrusted data inputs, a fundamental challenge often referred to as the "instruction/data boundary collapse" ^[11].

Data Poisoning: Corrupting the AI's Knowledge

Data poisoning involves manipulating the training data used by AI models. This can degrade the model's performance, introduce biases, or embed backdoors that can be triggered later.

• Degraded performance: Corrupted data can lead to less accurate or unreliable AI outputs.
• Biased outputs: Poisoning can skew the AI's responses to favor certain outcomes or demographics.
• Backdoors: Specific triggers can be introduced into the training data, causing the model to behave maliciously when those triggers are encountered in deployment ^[2][4][18].

Model Theft and Inversion: Stealing or Reconstructing AI Assets

Model theft involves the direct exfiltration of trained AI models, which represent significant intellectual property and development investment ^[19].

Model inversion attacks aim to reconstruct sensitive training data from the outputs of a deployed AI model. This can reveal proprietary information or PII that was used during training ^[2].

Supply Chain Attacks in the AI Ecosystem

The AI development and deployment lifecycle relies on a complex ecosystem of libraries, SDKs, models, and infrastructure. This introduces numerous supply chain vulnerabilities:

• Malicious dependencies: Compromised open-source packages, libraries, or AI skills can inject malware or backdoors into AI applications ^{[10][20][21][22][23]}.
• Package hallucination: AI tools might generate fake package names that attackers can preemptively register with malicious code ^[24][4].
• Model Context Protocol (MCP) vulnerabilities: The MCP standard, used for AI agent communication, has been a target for various attacks, including tool poisoning, command injection, and exposing sensitive data through its metadata and transport mechanisms ^{[25][16][26][7][27][17]}.
• Vulnerable AI tooling: Common AI development tools and frameworks can harbor critical vulnerabilities ^[28][29][30]. For instance, vulnerabilities in the NVIDIA Container Toolkit allow container escapes ^[29][10], and Ollama has been affected by path traversal RCE ^[31]. SAP AI Core has also seen significant vulnerabilities ^[32].
• AI agent skills and marketplaces: Public marketplaces for AI agent skills often contain malicious or insecure components ^{[33][21][34][23]}. For example, 13.4% of analyzed public agent skills contained critical security issues ^[35].

Exploiting AI-Generated Code

AI coding assistants can introduce vulnerabilities into software development workflows. Studies indicate a significant percentage of AI-generated code snippets contain common weaknesses (CWEs) ^[36]. Up to 40% of AI-generated code may contain security flaws ^[24][37]. These flaws can include business logic errors, authorization issues, and injection vulnerabilities, sometimes at a higher rate than human-written code ^{[38][39][40][41]}.

Notable Techniques and Attack Vectors

Prompt Injection Variants and Exploitations

• EchoLeak (CVE-2025-32711): A zero-click prompt injection exploit in Microsoft 365 Copilot that exfiltrated data via hidden instructions embedded in emails ^{[14][15][8][9]}.
• ShareLeak (CVE-2026-21520): An indirect prompt injection flaw in Copilot Studio that exposed data through Outlook workflows ^[13].
• GrafanaGhost: A zero-click flaw turning Grafana's AI assistant into a data exfiltration channel via URL parameters in logs ^[13].
• SearchLeak: A vulnerability chain in Microsoft 365 Copilot Enterprise that leveraged P2P Injection, HTML Injection Race Condition, and Bing SSRF to exfiltrate data ^[14].
• PromptSnatcher: Browser extensions disguised as ad blockers that stole AI chat conversations, model usage, and subscription metadata ^[42].
• ASCII Smuggling: Using hidden Unicode characters for data exfiltration, notably in Microsoft 365 Copilot ^[43].
• Jailbreaking and Guardrail Evasion: Techniques to bypass AI safety filters, often through role-playing, fictional framing, or exploiting logic flaws in AI judges ^{[3][22][44][45][46][18][47][48]}.
• System Prompt Extraction: Probing LLMs to reveal their system prompts or custom instructions, which can provide insights into their function and security configurations ^[3].
• PromptLock: AI-generated personalized ransom notes, demonstrating AI's use in novel attack vectors ^[10].
• Command and Control (C2) via GitHub Comments: Exploiting AI agents on GitHub Actions to steal API keys and tokens through crafted comments ^[49][50].
• Tool Poisoning Attacks (TPA): Malicious instructions embedded in tool metadata or descriptions used by AI agents via the Model Context Protocol (MCP) ^[16][17]. This can lead to data exfiltration and unauthorized actions ^[16].
• RAG Backdoor Attack: Compromising retrieval-augmented generation (RAG) pipelines by poisoning the knowledge graph or embedding malicious instructions within retrieved documents ^[51].
• Sockpuppeting (Assistant Prefill Abuse): Abusing the 'assistant prefill' API feature to bypass safety guardrails in LLMs ^[45][46].
• EchoGram (Token Flip): A technique to bypass AI guardrails by flipping verdicts of defensive models through specific token sequences ^[44].

AI Agentic Threats

• Agent Hijacking: Gaining control of autonomous AI agents to execute malicious actions or pursue attacker-defined goals ^{[52][11][53][51]}.
• Tool Misuse and Exploitation: AI agents using their connected tools in unintended or harmful ways, often triggered by prompt injection or poisoned tool configurations ^[11][53][26].
• Excessive Agency/Over-Privileged Tools: AI agents having more access than necessary, increasing the risk of exploitation if compromised ^[54][55].
• Inter-Agent Trust Exploitation: Compromised agents affecting other trusted agents or leveraging existing trust relationships to escalate attacks ^[51].
• Autonomous Planning and Execution: AI agents capable of autonomously planning and executing complex multi-step attacks, potentially replicating real-world breach scenarios ^[56].
• Tainted Memories: Poisoning an AI's long-term memory with malicious instructions to ensure persistent malicious behavior ^[57].
• Task Injection: Tricking AI agents into believing a malicious sub-task is legitimate and should be executed ^[57].

Supply Chain Attacks and Infrastructure Exploitation

• LiteLLM Supply Chain Attacks: Multiple vulnerabilities have been discovered in LiteLLM, an AI model gateway, including pre-authentication SQL injection allowing API key theft ^[58][59][60] and authenticated RCE via JSON config ^[59]. A later incident involved a poisoned security scanner (Trivy) ^[60].
• MCP SDK Vulnerabilities: Critical vulnerabilities in Anthropic's Model Context Protocol (MCP) SDK have allowed arbitrary command execution (RCE), impacting millions of servers and downstream applications ^[25][26][61].
• NVIDIA Container Toolkit Vulnerabilities: Critical container escape vulnerabilities (e.g., CVE-2024-0132, CVE-2025-23266) allow for host system compromise from within containers, impacting AI cloud services ^[29][10].
• NVIDIA Triton Inference Server Vulnerabilities: A vulnerability chain (CVE-2025-23319) enabled RCE through information disclosure and shared memory abuse ^[30].
• Malicious Nx Packages: Attacks involving malicious npm packages that leveraged AI coding agents with unsafe flags to exfiltrate secrets, including hardcoded credentials and API tokens ^[20].
• Cursor AI Vulnerability: A vulnerability in the Cursor AI coding assistant allowed for Remote Code Execution (RCE) ^[62].
• Antigravity IDE Vulnerability: Google's Antigravity IDE had a flaw enabling prompt injection and RCE, bypassing Secure Mode ^[63].
• Protestware: Malicious instructions hidden in package releases intended for AI agents, like the incident in jqwik v1.10.0 targeting AI agents ^[64].
• OpenClaw Ecosystem Risks: A significant percentage of skills in the OpenClaw marketplace have security flaws, including malware distribution and exposed secrets ^{[33][21][34][23]}.

Infrastructure and Cloud Security for AI

• Cross-Tenant Data Leakage: Vulnerabilities in AI-as-a-Service platforms can expose private AI models and customer data across tenants ^[65][66]. Wiz Research found such issues in Replicate and Hugging Face ^[65][66].
• Misconfigured Cloud Resources: AI workloads running in cloud environments may inherit common cloud misconfigurations, such as publicly accessible storage buckets or improperly secured IAM roles.
• Model Context Protocol (MCP) Server Security: MCP servers, used for connecting AI agents to enterprise systems, are a critical target. Vulnerabilities include open dynamic client registration, missing redirect URI validation, missing PKCE enforcement, command injection, SSRF, path traversal, and DNS rebinding ^[7][27][67].
• Insecure AI infrastructure components: Vulnerabilities in AI model serving frameworks (e.g., NVIDIA Triton) or containerization tools are critical due to their widespread use in AI deployments ^[29][30].

Detection and Prevention Strategies

Shifting Left with AI-Native Security

The imperative is to integrate security earlier into the AI development lifecycle. This involves:

• Secure AI Development Lifecycle (SAIDL): Applying security best practices from data collection and model training through deployment and monitoring ^{[68][69][19][24][38][70][71][37]}.
• AI Bill of Materials (AI BOM): Maintaining a comprehensive inventory of all AI components, dependencies, SDKs, and models used in an application. This is crucial for tracking risks and ensuring transparency ^{[72][73][70][37][74]}.
• AI-Powered SAST/DAST: Leveraging AI to analyze code and running applications for vulnerabilities, including those traditional tools might miss. Snyk Code, for instance, offers AI-powered analysis ^[69][19][75].
• Prompt Engineering for Security: Designing prompts and system instructions that are robust against injection attacks and guide the AI towards secure outputs ^{[3][76][77][47][78][79]}.
• Input and Output Validation/Sanitization: Implementing rigorous checks on all data that flows into and out of AI models and agents, especially when interacting with external systems or rendering outputs ^{[52][80][13][81]}.
• Least Privilege for AI Agents: Ensuring AI agents have only the necessary permissions and access to tools required for their intended function ^[54][55].

Runtime Protection and Monitoring

Continuous monitoring and runtime defenses are essential to detect and respond to AI-specific threats:

• Behavioral Analysis of AI Agents: Moving beyond simple keyword matching to understand the underlying intent of AI agent instructions and actions ^[11][82].
• Runtime Guardrails: Implementing real-time checks on AI agent behavior and tool usage to block malicious actions before they occur ^[19][22]. Snyk Guard and Lakera Guard offer such capabilities.
• Threat Detection for AI Workloads: Specialized detection mechanisms for identifying prompt injection attempts, data poisoning, and malicious AI agent behavior in production ^[72][53].
• AI-Powered Incident Response: Utilizing AI to analyze security logs, correlate events, and automate parts of the incident response process, accelerating MTTR ^{[83][84][85][82][86]}. Wiz's Blue Agent aids in investigation.
• Securing MCP Infrastructure: Implementing security controls around MCP servers, including proper authentication, authorization, validation of tool parameters, and network segmentation ^{[25][87][88][7][27][17][89][90]}.

AI Security Posture Management (AI-SPM)

AI-SPM tools provide a centralized view and control plane for managing AI security risks across an organization's AI landscape.

• Visibility: Discovering all AI assets, including models, agents, pipelines, and dependencies, to understand the scope of the AI attack surface ^{[2][72][19][91][92]}.
• Risk Assessment: Identifying and prioritizing AI-specific vulnerabilities, misconfigurations, and policy violations ^{[93][94][95][72][91][92]}.
• Posture Management: Enforcing security policies and best practices across AI development and deployment ^{[93][94][95][72][91][92]}.
• Compliance: Ensuring AI systems adhere to relevant regulations and governance frameworks ^[93].

Supply Chain Security for AI Assets

Securing the AI software supply chain requires specific measures:

• Governed AI Artifact Repositories: Using secure registries for storing and distributing AI models, packages, and agent skills, with robust access controls and integrity checks ^[96][35].
• Vulnerability Scanning of AI Dependencies: Regularly scanning AI models, libraries, and agent skills for known vulnerabilities and malicious components ^{[97][21][23][60]}.
• Behavioral Integrity Verification (BIV): Auditing AI skills to ensure their declared behavior matches their actual executed behavior ^[33].
• AI Bill of Materials (AI BOM): As mentioned earlier, an AI BOM is critical for understanding the provenance and potential risks within the AI supply chain ^[72][74].

Tooling for AI Security

A growing ecosystem of tools is emerging to address AI-specific security challenges:

• AI Security Posture Management (AI-SPM): Wiz AI-SPM provides visibility and risk assessment across the AI lifecycle ^{[93][94][95][72][91]}. Snyk's AI Security Platform aims to secure AI-powered development ^[19][98][39].
• AI Code Security and Remediation: Snyk Code and Snyk Agent Fix offer AI-powered SAST and automated vulnerability remediation for AI-generated and human-written code ^{[36][99][100][101][69][19][102][75][70][37]}.
• AI Agent Security Platforms: Evo by Snyk acts as an Agentic Security Orchestrator ^[103][104]. Wiz AI-APP provides end-to-end protection for AI applications ^[91]. FireTail and Repello AI offer comprehensive AI security solutions ^[105].
• LLM Security and Red Teaming Tools: Garak is an LLM vulnerability scanner ^[106]. Promptfoo aids in LLM red teaming ^[107]. DeepTeam offers a framework for detecting AI risks ^[108]. Novee AI Red Teaming targets LLM applications ^[109].
• Supply Chain Security for AI: JFrog AI Catalog helps govern AI assets ^[96]. MCP Scan CLI and Nova Proximity assess the security of MCP servers and agent skills ^[87][89].
• AI-Powered Offensive Security: Wiz Red Agent simulates attacker behavior ^[110][111]. Pentest Swarm AI utilizes swarm intelligence for penetration testing ^[112]. Shannon is an autonomous AI hacker for web apps ^[113]. Crossbow-agent aims to be a fully autonomous AI security engineer ^[114].
• Model Context Protocol (MCP) Security: MCP Snitch and Nova Proximity provide security scanning and proxy solutions for MCP servers ^[89][90].

Recent Developments and Emerging Trends

AI-Driven Vulnerability Discovery and Exploitation

Frontier AI models, such as Anthropic's Claude Mythos, are demonstrating the ability to autonomously discover zero-day vulnerabilities and generate working exploits with alarming speed ^[22][115]. This accelerates the CVE discovery process and necessitates a paradigm shift in how security teams respond, moving from detection to proactive control and rapid patching ^[53].

The Rise of Agentic AI and its Security Implications

Autonomous AI agents operating in development workflows and enterprise environments present unique challenges. Prompt injection, tool poisoning, and memory manipulation are key vectors for compromising these agents. The "Comment and Control" vulnerability class on GitHub Actions highlights how malicious comments can hijack AI agents, leading to API key theft ^[49][50]. The Model Context Protocol (MCP) is a critical infrastructure component for these agents, and vulnerabilities within it are highly impactful ^{[25][26][61][7]}.

Indirect Prompt Injection as a Universal Threat

IPI is recognized as a pervasive threat affecting both cloud-based and local AI models due to the collapse of the instruction/data boundary ^[11][13][6]. Real-world examples like EchoLeak in Microsoft 365 Copilot demonstrate its potential for zero-click data exfiltration ^[15].

Supply Chain Attacks Evolve with AI

Attackers are increasingly weaponizing AI coding agents and poisoning AI dependencies. The PromptMink campaign, involving malicious npm packages co-authored by Claude Opus, exemplifies how AI can be integrated into sophisticated supply chain attacks ^[116]. The compromise of AI frameworks like LiteLLM and MCP SDKs underscores the need for deep visibility into the AI supply chain ^[58][25][60].

AI as Both Attacker and Defender

The landscape is seeing a dual role for AI: it's used by attackers to discover and exploit vulnerabilities at scale ^[22][56], and by defenders for automated threat hunting, vulnerability analysis, and incident response ^{[83][84][85][82]}. Tools like Wiz Red Agent simulate attacker behavior for offensive security testing, while Microsoft Security Copilot aids in defensive operations ^{[110][111][117]}.

Where to Go Deeper

To stay abreast of this rapidly evolving field, consider the following resources:

• OWASP Top 10 for LLM Applications: A foundational document outlining the most critical security risks for LLM-based systems ^[108].
• OWASP Top 10 for Agentic Applications: Emerging guidance on securing autonomous AI agents.
• MITRE ATLAS: A framework documenting adversary tactics, techniques, and common knowledge (ATCK) for AI systems, providing a structured understanding of AI exploitation ^[118].
• NIST AI Risk Management Framework (RMF): A framework for managing AI risks throughout the AI lifecycle ^[93].
• Vendor Research and Blogs: Organizations like Wiz.io, Snyk.io, Palo Alto Networks Unit 42, HiddenLayer, and others regularly publish detailed research on AI security vulnerabilities and techniques ^{[93][119][28][118][94][83][120][35][95][36][121][122][84][65][66][123][124][14][99][100][29][101][30][10][2][72][42][68][69][19][20][33][57][125][126][127][103][128][91][104][74][52][129][39][21][11][22][130][111][92][131][53][86][132][64][112][133][80][13][134][54][5][58][116][59][62][135][55][61][63][136][25][44][16][15][56][51][49][26][6][137][88][7][27][76][81][17][77][47][109][48][78][138][108][79][139][67][114][113][106][89][90][43][140][141][142][143][144][145][117][146]}.
• Academic Research and Pre-print Archives: Platforms like arXiv host cutting-edge research on AI security, often predating formal publication ^[15][51][26].
• Security Conferences and Webinars: Staying engaged with presentations and discussions from security conferences provides insights into the latest threats and defenses.