Findings/crash-is-not-exploitation

Counting a crash as exploitation success overstates capability; most public models in May 2026 stalled before code execution on a browser engine.

Reportedmeasured1 evidence record from 1 independent sourceassistant-drafted

Scope: what this does not show

One benchmark on one target class.

Reported: Stated by one source and not yet corroborated or challenged.

Evidence

May 13, 2026

ExploitBench grades AI exploit development as a 16-step capability ladder on V8 bugs

arXiv

Of 41 V8 bugs, crashes were common but no publicly deployed model reached arbitrary code execution in the primary arm; the unreleased Mythos Preview did on 18.

How it relates to other findings

supportsqualifiescontestssupersedes

ReportedCorroboratedQualifiedContestedSupersededRevalidate· node size = evidence records · columns group by topic

Select a finding to see how it relates to others. Arrows point from the newer finding to the one it supports, qualifies, contests, or supersedes.

This finding qualifies On ExploitGym (May 2026), the strongest agents produced working exploits for 157 and 120 of 898 instances with mitigations off; with standard mitigations on, 45 and 21 survived.On hardened V8 targets, publicly deployed models rarely got past primitives to code execution; code execution at scale came only from an unreleased model.

Key questions that rely on this finding

How far can measured AI cyber capability be trusted?As a lower or conditional bound. Scores move substantially with token budget, evaluation pipeline, and benchmark contamination.

Status history

2026-05-13ReportedExploitBench separates crashes from exploitation progress. · record

Practice the judgment

Two success metrics, two different questions Evidence case · The same distinction on CyberGym: any crash versus the target vulnerability

In the Defense Evidence Lab, Fide’s companion tool for judging what cyber evidence can support. Opens a separate site.

What would change this: A later measurement showing models routinely reach code execution.
Evidence kind: measured: An experiment or evaluation produced the number.
Freshness: Evidence 136d old · recheck in 104 daysNewest evidence May 13, 2026. Half-life 240 days.
Methods tested: AI-assisted vulnerability exploitation CTF and task benchmarks
Topics: Evaluation validity Exploit development

Cite this record

Fide AI. "Counting a crash as exploitation success overstates capability; most public models in May 2026 stalled before code execution on a browser engine.." Agentic Cyber Explorer, record crash-is-not-exploitation. https://agentic-cyber-explorer.pages.dev/findings/crash-is-not-exploitation/ (corpus as of 2026-09-26).

BibTeX

@misc{fide-ace-crash-is-not-exploitation,
  author = {{Fide AI}},
  title = {Counting a crash as exploitation success overstates capability; most public models in May 2026 stalled before code execution on a browser engine.},
  howpublished = {Agentic Cyber Explorer},
  year = {2026},
  url = {https://agentic-cyber-explorer.pages.dev/findings/crash-is-not-exploitation/},
  note = {Record crash-is-not-exploitation; corpus as of 2026-09-26}
}

Data: JSON · CC BY 4.0

Know of evidence that changes this? Tell us.