Oct 3, 2024
Agent Security Bench formalizes attacks and defenses across ten LLM agent scenarios
Zhang and colleagues release Agent Security Bench (ASB), covering 10 scenarios, 10 agents and over 400 tools, and benchmark attack and defense methods, including prompt injection, memory poisoning and a new backdoor attack, across 13 LLMs (23 methods in the first version, 27 in the ICLR 2025 version). They report a highest average attack success rate of 84.30% and limited effectiveness of current defenses. The paper was accepted at ICLR 2025.