Findings/crash-checks-overstate-patch-success

Checking only that the original crash no longer reproduces overstates how often AI-generated patches actually fix the vulnerability.

Corroboratedmeasured2 evidence records from 2 independent sourcesassistant-drafted

Scope: what this does not show

Two analyses of research and competition patches. PatchBench's 1.83x is measured on tasks selected so that the true fix lies outside the crash stack, so it is not a base rate.

Corroborated: Supported by at least two independent sources.

Evidence

Feb 7, 2026

AIxCC SoK finds stability decided results and many validated AI patches were still semantically wrong

arXiv

Manual review found 37.7% and 45.6% of baseline-agent patches that passed all automatic checks semantically incorrect.

Sep 3, 2026

PatchBench finds PoC-only checks inflate AI patching success 1.83x and 25% of patches look memorized

arXiv

Proof-of-concept-only checks inflated success 1.83 times on PatchBench's tasks.

How it relates to other findings

supportsqualifiescontestssupersedes

ReportedCorroboratedQualifiedContestedSupersededRevalidate· node size = evidence records · columns group by topic

Select a finding to see how it relates to others. Arrows point from the newer finding to the one it supports, qualifies, contests, or supersedes.

This finding qualifies In DARPA's AI Cyber Challenge, autonomous systems patched most of the synthetic vulnerabilities they found.The AIxCC teams' agents lose much of their solve rate under stronger validation on PatchBench tasks; the competition's counted patches were not re-validated.

Key questions that rely on this finding

Is AI shifting the balance between finding and fixing vulnerabilities?Discovery is ahead. AI finds real vulnerabilities faster than they are fixed, and simple checks overstate how often AI patches work.

Status history

2026-02-07ReportedAIxCC review finds semantically wrong top patches. · record
2026-09-03CorroboratedPatchBench measures the inflation directly. · record
2026-09-25CorroboratedcorrectionThe review paper's support is its manual review of baseline agents (38-46% of fully validated patches semantically wrong), not the competition-scored accuracy figures. · record

Practice the judgment

What would prove that a repair restored security? Research question · What would prove a repair worked

In the Defense Evidence Lab, Fide’s companion tool for judging what cyber evidence can support. Opens a separate site.

What would change this: Verification practice that closes the gap in deployed repair tools.
Evidence kind: measured: An experiment or evaluation produced the number.
Freshness: Evidence 23d old · recheck in 517 daysNewest evidence Sep 3, 2026. Half-life 540 days.
Methods tested: Automated patching Independent patch verification
Topics: Vulnerability repair Evaluation validity
Fide research: FID-088: Independent verification of autonomous cyber repairs

Cite this record

Fide AI. "Checking only that the original crash no longer reproduces overstates how often AI-generated patches actually fix the vulnerability.." Agentic Cyber Explorer, record crash-checks-overstate-patch-success. https://agentic-cyber-explorer.pages.dev/findings/crash-checks-overstate-patch-success/ (corpus as of 2026-09-26).

BibTeX

@misc{fide-ace-crash-checks-overstate-patch-success,
  author = {{Fide AI}},
  title = {Checking only that the original crash no longer reproduces overstates how often AI-generated patches actually fix the vulnerability.},
  howpublished = {Agentic Cyber Explorer},
  year = {2026},
  url = {https://agentic-cyber-explorer.pages.dev/findings/crash-checks-overstate-patch-success/},
  note = {Record crash-checks-overstate-patch-success; corpus as of 2026-09-26}
}

Data: JSON · CC BY 4.0

Know of evidence that changes this? Tell us.