Authors affiliated with XOR, Protege and CISPA evaluate attack, repair and subsequent attacks in five nonpublic software environments using researcher-reviewed deterministic graders. Repair scores exceed attack scores in two environments and fall below them in three, while subsequent tests expose failures after an initial exploit is stopped. The selected environments, differing outcome definitions and dependent interactions do not establish a general attacker or defender advantage.
It tests whether a repair remains secure against further attempts and reduces dependence on public exploit-and-fix material without claiming training-data contamination has been eliminated.
Key facts
As stated in the sources, with where to find them.
- Five nonpublic environments comprise four investigated flaws and one constructed protocol variant, including privately disclosed vulnerabilities that remained unpatched. Scores come from researcher-developed and reviewed deterministic graders, not LLM judges.Abstract; Introduction
- Repair scores exceed attack scores in two nonpublic environments and fall below them in three. Attack and repair criteria differ, so a task-score difference is not a scale-calibrated capability advantage.Abstract; Section VII, Comparability and generalization
- In repeated interactions requesting GPT-5.6 Sol at high reasoning effort, with a broader collection also including Opus 5, another exploit succeeds in 92 of 524 defender test intervals after the initial exploit is stopped. The dependent intervals include intermediate software states, not just final submissions, and are not independent patch-success trials.Abstract; Section VI; Section VII, Measurement and review
- The authors compare with a group of 209 disclosed memory-safety vulnerabilities and a separate collection of cryptographic challenges. Differences cannot isolate prior exposure from task difficulty, system choices and scoring. Absence of public flaw-specific solutions does not prove absence from model training data.Sections III–VII, Limits and interpretation
Findings that cite this record
No tracked finding cites this record yet.
Key questions this bears on
- Is AI shifting the balance between finding and fixing vulnerabilities?Discovery is ahead. Verification, triage, coordinated disclosure and safe repair remain bottlenecks between AI findings and deployed fixes.
- How are attackers using AI agents in real operations?To run parts of intrusions at scale: vendors report a state-aligned actor and criminal groups using open-source agent frameworks and commercial models.