PatchBench, from the University of Maryland's AI Security Lab, evaluates 11 patching agents, including the top three AIxCC systems, on 213 C/C++ tasks whose true fixes lie outside the crash stack, using vulnerability transplant and code mutation to limit memorization. It finds that accepting a patch because the original proof-of-concept no longer crashes inflates solve rates by 1.83x on average, and that about 25% of agent patches closely resemble historical developer fixes.
Why it matters
It directly challenges how AI vulnerability-repair results, including competition results, are validated.
Key facts
As stated in the sources, with where to find them.
- Across 11 agents, PoC-only validation inflates patching solve rates by 1.83x on average.arXiv abstract
- On average 25% of agent patches show substantial similarity to historical developer patches.arXiv abstract
- 213 tasks from 32 C/C++ GitHub projects; a task counts as solved only if it passes PoC, replay (security), unit test and output-state checks.Dataset card; repository README
- Task container images total about 870 GB and additional corpora about 22 GB; metadata is MIT-licensed.Repository README; dataset card
Findings that cite this record
Key questions this bears on
- Do cyber evaluations of AI agents stay contained?Not reliably. Agents have reached real systems from evaluation, training and research; red-teaming found containment gaps OpenAI says it largely fixed.
- Is AI shifting the balance between finding and fixing vulnerabilities?Discovery is ahead. Verification, triage, coordinated disclosure, safe repair and getting fixes to users remain bottlenecks between AI findings and protection.
Sources
Related records
Feb 7, 2026
Oct 25, 2026
Oct 13, 2026
Oct 12, 2026
Oct 5, 2026
Sep 28, 2026