Lee, Bae and Yun show that existing CTF benchmarks can be solved by retrieving published writeups when agents have web search, and propose CTFusion, which evaluates agents on live CTF competitions through an MCP server on the CTFd platform. They test 3 LLMs and 2 agent designs across 5 live CTF events.
Why it matters
Static CTF benchmarks underpin many cyber capability claims, and this work demonstrates a concrete contamination path.
Key facts
As stated in the sources, with where to find them.
- Adding web search to the D-CIPHER agent raised its NYU CTF Bench solve rate from 12.59% to 24.07%; the authors traced several submitted flags to public solutions (63 flag-copying and 8 write-up-search cases).Introduction; Section 3
- Across 3 LLMs (GPT-4.1, Claude 3.5 Sonnet, Gemini 2.5 Flash) and 2 agents (EnIGMA, D-CIPHER), success was 14.4% on NYU CTF Bench vs 6.3% on 5 live CTFs; the authors say contamination may explain part of the gap.Introduction; evaluation section
Findings that cite this record
Key questions this bears on
- Can prompt injection against AI agents be reliably defended?Not reliably. Bounding what untrusted input can trigger helps, but trust boundaries must survive delegation and tool execution.
- Do cyber evaluations of AI agents stay contained?Not reliably. Agents have reached real systems from evaluation, training and research; red-teaming found containment gaps OpenAI says it largely fixed.
- How far can measured AI cyber capability be trusted?As a lower or conditional bound. Scores move with budget, pipeline and contamination, and benchmark names or labels can hide different setups.
Sources
Related records
Sep 26, 2026
Apr 21, 2026
Oct 22, 2026
Oct 27, 2026
Oct 13, 2026
Oct 12, 2026