The CyberBinGym preprint evaluates agents given stripped vulnerable and patched binaries for 100 tasks from 53 projects. On the non-hardened baseline, Codex with GPT-5.6 recovers vulnerabilities on 88% of tasks and completes exploitation on 20%. The known-vulnerability patch-gap tests use trusted-access models with safeguards disabled; hardened configurations are evaluated separately.
Why it matters
It measures the gap between identifying a bug from a binary patch and producing a verified exploit during delayed deployment.
Key facts
As stated in the sources, with where to find them.
- The 100 tasks cover 53 projects. Agents receive stripped vulnerable and patched binaries without source code or vulnerability descriptions.Abstract; benchmark construction
- Recovery uses differential crash checks; exploitation requires end-to-end flag retrieval and trajectory audits to reject reward hacking. Codex with GPT-5.6 scores 88% and 20%, respectively, on the non-hardened baseline with a nominal 1,000-call budget.Evaluation protocol; main results
- Providing a patched binary raises exploitation from 10% to 25% on a matched 20-task subset. This comparison uses that subset rather than the full benchmark denominator.Abstract; patched-binary ablation
- In a selected hardened rerun, Codex with GPT-5.6 exploits two of its 20 previously successful tasks. Search variation and service failures mean the aggregate difference does not isolate hardening’s causal effect. Main experiments use trusted-access models with safeguards disabled.Hardening experiment; Appendix A.8
Findings that cite this record
No tracked finding cites this record yet.
Key questions this bears on
- How far can measured AI cyber capability be trusted?As a lower or conditional bound. Scores move with budget, pipeline and contamination, and benchmark names or labels can hide different setups.
- How are attackers using AI agents in real operations?To run large parts of intrusions under human direction, including espionage, credential theft and extortion; reported autonomy and success vary.
Sources
Related records
Oct 13, 2026
Oct 5, 2026
Sep 28, 2026
Nov 5, 2026
Oct 22, 2026
Oct 20, 2026