Agentic Cyber Explorerby Fide AI · PreviewPreview
Chronicle/Defense & research

Agent Security Bench formalizes attacks and defenses across ten LLM agent scenarios

DefenseBenchmarkSignificance assistant-drafted

Zhang and colleagues release Agent Security Bench (ASB), covering 10 scenarios, 10 agents and over 400 tools, and benchmark attack and defense methods, including prompt injection, memory poisoning and a new backdoor attack, across 13 LLMs (23 methods in the first version, 27 in the ICLR 2025 version). They report a highest average attack success rate of 84.30% and limited effectiveness of current defenses. The paper was accepted at ICLR 2025.

Why it matters

ASB broadens agent security benchmarking beyond indirect injection to memory poisoning and backdoors in the agent loop.

Key facts

As stated in the sources, with where to find them.

  • ICLR 2025 version (arXiv v4, May 2025): 10 scenarios, 10 agents, 400+ tools, 27 attack/defense methods (10 prompt injection attacks, memory poisoning, a Plan-of-Thought backdoor, 4 mixed attacks, 11 defenses), 7 metrics, 13 LLM backbones.Abstract (v4)
  • First version (October 3, 2024): 23 attack/defense methods (one mixed attack, 10 defenses), 8 metrics, 13 LLM backbones, nearly 90,000 test cases.Abstract (v1)
  • Highest average attack success rate across evaluated attacks was 84.30% (same figure in v1 and v4).Abstract

Findings that cite this record

Key questions this bears on

Sources

Related records