PatchBench, from the University of Maryland's AI Security Lab, evaluates 11 patching agents, including the top three AIxCC systems, on 213 C/C++ tasks whose true fixes lie outside the crash stack, using vulnerability transplant and code mutation to limit memorization. It finds that accepting a patch because the original proof-of-concept no longer crashes inflates solve rates by 1.83x on average, and that about 25% of agent patches closely resemble historical developer fixes.
Why it matters
It directly challenges how AI vulnerability-repair results, including competition results, are validated.
Key facts
As stated in the sources, with where to find them.
- Across 11 agents, PoC-only validation inflates patching solve rates by 1.83x on average.arXiv abstract
- On average 25% of agent patches show substantial similarity to historical developer patches.arXiv abstract
- 213 tasks from 32 C/C++ GitHub projects; a task counts as solved only if it passes PoC, replay (security), unit test and output-state checks.Dataset card; repository README
- Task container images total about 870 GB and additional corpora about 22 GB; metadata is MIT-licensed.Repository README; dataset card
Findings that cite this record
Key questions this bears on
- Is AI shifting the balance between finding and fixing vulnerabilities?Discovery is ahead. AI finds real vulnerabilities faster than they are fixed, and simple checks overstate how often AI patches work.
Sources
Related records
Feb 7, 2026
Aug 8, 2025
Apr 29, 2025
Aug 11, 2024
Aug 4, 2024
Aug 9, 2023