Chronicle/Capability & gating

CyberBinGym separates binary vulnerability recovery from working exploitation

CapabilityBenchmarkSignificance assistant-drafted

The CyberBinGym preprint evaluates agents given stripped vulnerable and patched binaries for 100 tasks from 53 projects. On the non-hardened baseline, Codex with GPT-5.6 recovers vulnerabilities on 88% of tasks and completes exploitation on 20%. The known-vulnerability patch-gap tests use trusted-access models with safeguards disabled; hardened configurations are evaluated separately.

Why it matters

It measures the gap between identifying a bug from a binary patch and producing a verified exploit during delayed deployment.

Key facts

As stated in the sources, with where to find them.

  • The 100 tasks cover 53 projects. Agents receive stripped vulnerable and patched binaries without source code or vulnerability descriptions.Abstract; benchmark construction
  • Recovery uses differential crash checks; exploitation requires end-to-end flag retrieval and trajectory audits to reject reward hacking. Codex with GPT-5.6 scores 88% and 20%, respectively, on the non-hardened baseline with a nominal 1,000-call budget.Evaluation protocol; main results
  • Providing a patched binary raises exploitation from 10% to 25% on a matched 20-task subset. This comparison uses that subset rather than the full benchmark denominator.Abstract; patched-binary ablation
  • In a selected hardened rerun, Codex with GPT-5.6 exploits two of its 20 previously successful tasks. Search variation and service failures mean the aggregate difference does not isolate hardening’s causal effect. Main experiments use trusted-access models with safeguards disabled.Hardening experiment; Appendix A.8

Findings that cite this record

No tracked finding cites this record yet.

Key questions this bears on

Sources

Related records