Meta introduced AutoPatchBench, part of CyberSecEval 4, with 136 fuzzing-identified C/C++ vulnerabilities and verified fixes, plus a 113-case Lite subset with single-function root causes. Patches are checked by build and crash reproduction, then fuzzing and white-box differential testing; Meta's reference agent generated crash-stopping patches in about 60% of cases, but only 5-11% passed the stricter checks.
Why it matters
It showed early that crash-only acceptance greatly overstates how often AI-generated security patches are actually correct.
Key facts
As stated in the sources, with where to find them.
- 136 fuzzing-identified C/C++ vulnerabilities with verified fixes; AutoPatchBench-Lite has 113.Benchmark description
- On AutoPatchBench-Lite, Meta's reference patch generator with each tested LLM produced a build- and crash-passing patch for about 60% of samples, but only about 5-11% of samples passed fuzzing and differential testing (Gemini 1.5 Pro: 61.1% vs 5.3%).A case study with AutoPatchBench-Lite, Figure 2
- Manual validation of 44 Llama 4 Maverick patches that passed fuzzing found differential testing had 84.1% accuracy, 100% recall and 41.7% precision.Case study, Table 1
Findings that cite this record
No tracked finding cites this record yet.
Key questions this bears on
- Is AI shifting the balance between finding and fixing vulnerabilities?Discovery is ahead. Verification, triage, coordinated disclosure, safe repair and getting fixes to users remain bottlenecks between AI findings and protection.
Sources
Related records
Oct 13, 2026
Oct 12, 2026
Sep 28, 2026
Oct 16, 2026
May 20, 2025
Oct 12, 2026