Dynamic environment for evaluating prompt-injection attacks and defenses on tool-using agents.
Nasr, Carlini, Tramèr and 11 co-authors apply gradient, reinforcement learning, search and human red-teaming attacks to 12 published defenses. Most defenses originally reported near-zero attack success, but the adaptive attacks exceed 90% success against most, and human red-teamers succeeded on every challenge in the subset of defenses they were given.
Meta open-sources LlamaFirewall, combining PromptGuard 2 (a jailbreak and injection detector), AlignmentCheck (a chain-of-thought auditor for goal hijacking) and CodeShield (static analysis of generated code). On AgentDojo, Meta reports that the combination cut attack success from 17.63% to 1.75% while utility fell from 47.73% to 42.68%.
Debenedetti and colleagues (Google, Google DeepMind, ETH Zurich) propose CaMeL, which extracts control flow from the trusted user query so untrusted data cannot change which actions run, and attaches capabilities to data to block unauthorized flows. On AgentDojo the first version reported 67% of tasks solved with provable security; the June 2025 revision, with newer models, reports 77% versus 84% for an undefended system.
NIST released the 2025 edition of its adversarial machine learning taxonomy, co-authored with the UK AI Security Institute and US AI Safety Institute staff. Unlike the 2023 edition, it includes a section on the security of agents, noting that tool-using agents are exposed to direct and indirect prompt injection and that hijacking can lead to arbitrary code execution or data exfiltration.
NIST's AI safety institute technical staff (renamed the Center for AI Standards and Innovation in June 2025) extended AgentDojo and red-teamed agents built on the upgraded Claude 3.5 Sonnet. On held-out Workspace tasks, attack success rose from 11% for the strongest baseline attack to 81% for the strongest newly developed attack, and across five injection tasks from 57% to 80% when each attack was tried 25 times. The team released an Inspect-based AgentDojo port and ran the red teaming with the UK AI Security Institute.
Debenedetti and colleagues (ETH Zurich, Invariant Labs) release AgentDojo, a dynamic environment with 97 realistic user tasks across workspace, banking, travel and Slack suites and 629 security test cases. It measures both utility and targeted attack success, and reports that existing attacks break some security properties but not all. It became the standard testbed used by CaMeL, US AISI/CAISI, LlamaFirewall and adaptive-attack studies.