Meta introduced AutoPatchBench, part of CyberSecEval 4, with 136 fuzzing-identified C/C++ vulnerabilities and verified fixes, plus a 113-case Lite subset with single-function root causes. Patches are checked by build and crash reproduction, then fuzzing and white-box differential testing; Meta's reference agent generated crash-stopping patches in about 60% of cases, but only 5-11% passed the stricter checks.
Why it matters
It showed early that crash-only acceptance greatly overstates how often AI-generated security patches are actually correct.
Key facts
As stated in the sources, with where to find them.
- 136 fuzzing-identified C/C++ vulnerabilities with verified fixes; AutoPatchBench-Lite has 113.Benchmark description
- On AutoPatchBench-Lite, Meta's reference patch generator with each tested LLM produced a build- and crash-passing patch for about 60% of samples, but only about 5-11% of samples passed fuzzing and differential testing (Gemini 1.5 Pro: 61.1% vs 5.3%).A case study with AutoPatchBench-Lite, Figure 2
- Manual validation of 44 Llama 4 Maverick patches that passed fuzzing found differential testing had 84.1% accuracy, 100% recall and 41.7% precision.Case study, Table 1
Findings that cite this record
No tracked finding cites this record yet.
Key questions this bears on
- Is AI shifting the balance between finding and fixing vulnerabilities?Discovery is ahead. AI finds real vulnerabilities faster than they are fixed, and simple checks overstate how often AI patches work.
Sources
Related records
May 20, 2025
Sep 3, 2026
Aug 12, 2026
May 12, 2026
Apr 21, 2026
Feb 7, 2026