One benchmark with an LLM judge, run with lab safeguards disabled; the top model (Claude Mythos Preview) was unreleased, and most other models fell to zero with mitigations on. Anthropic's September 2026 Opus 5.5 system card reports higher ExploitGym counts for a released model (289 of 869 instances within two hours, mitigations disabled, safeguards off) with a different instance count, and Anthropic says many of its harnesses were rewritten for that card, so its figures are not directly comparable with the May 2026 ones. Epoch AI (2026-10-29) separately reports GLM 5.3 building file-reading exploit chains, not full code execution, for six Firefox releases offline, using fixes public before they reached users, at a marginal cost that depends on how earlier research spending is counted.
Qualified: Still standing, but later work narrows how far it applies.
Evidence
Table 3: Mythos Preview 157 and GPT-5.5 120 of 898. Table 5: with mitigations re-enabled, 45 and 21 remained.
Anthropic reports a released model, Claude Opus 5.5, exploiting 289 of 869 ExploitGym instances within two hours and 300 within six, mitigations disabled.
Epoch AI reports that GLM 5.3, in an offline setup with researcher-supplied tools and materials, built file-reading exploit chains for six Firefox releases, each also using a then-unpublished flaw Mozilla patched in Firefox 159.0.
How it relates to other findings
Select a finding to see how it relates to others. Arrows point from the newer finding to the one it supports, qualifies, contests, or supersedes.
- Counting a crash as exploitation success overstates capability; most public models in May 2026 stalled before code execution on a browser engine. qualifies this findingOn hardened V8 targets, publicly deployed models rarely got past primitives to code execution; code execution at scale came only from an unreleased model.
Key questions that rely on this finding
- How far can measured AI cyber capability be trusted?As a lower or conditional bound. Scores move with budget, pipeline and contamination, and benchmark names or labels can hide different setups.
- Is AI shifting the balance between finding and fixing vulnerabilities?Discovery is ahead. Verification, triage, coordinated disclosure, safe repair and getting fixes to users remain bottlenecks between AI findings and protection.
Status history
- 2026-05-11ReportedExploitGym results. · record
- 2026-09-08QualifiedLater work shows cyber benchmark scores depend heavily on pipeline choices. · record
- 2026-09-25QualifiedcorrectionThe pipeline audit covered eight knowledge and multiple-choice benchmarks, not ExploitGym. The qualification rests on ExploitBench, where no publicly deployed model reached code execution on V8. · record