Findings/frontier-models-produce-working-exploits

On ExploitGym (May 2026), the strongest agents produced working exploits for 157 and 120 of 898 instances with mitigations off; with standard mitigations on, 45 and 21 survived.

Qualifiedmeasured3 evidence records from 3 independent sourcesassistant-drafted
Scope: what this does not show

One benchmark with an LLM judge, run with lab safeguards disabled; the top model (Claude Mythos Preview) was unreleased, and most other models fell to zero with mitigations on. Anthropic's September 2026 Opus 5.5 system card reports higher ExploitGym counts for a released model (289 of 869 instances within two hours, mitigations disabled, safeguards off) with a different instance count, and Anthropic says many of its harnesses were rewritten for that card, so its figures are not directly comparable with the May 2026 ones. Epoch AI (2026-10-29) separately reports GLM 5.3 building file-reading exploit chains, not full code execution, for six Firefox releases offline, using fixes public before they reached users, at a marginal cost that depends on how earlier research spending is counted.

Qualified: Still standing, but later work narrows how far it applies.

Evidence

How it relates to other findings

supportsqualifiescontestssupersedes
ReportedCorroboratedQualifiedContestedSupersededRevalidate· node size = evidence records · columns group by topic

Select a finding to see how it relates to others. Arrows point from the newer finding to the one it supports, qualifies, contests, or supersedes.

Key questions that rely on this finding

Status history

  1. 2026-05-11ReportedExploitGym results. · record
  2. 2026-09-08QualifiedLater work shows cyber benchmark scores depend heavily on pipeline choices. · record
  3. 2026-09-25QualifiedcorrectionThe pipeline audit covered eight knowledge and multiple-choice benchmarks, not ExploitGym. The qualification rests on ExploitBench, where no publicly deployed model reached code execution on V8. · record