Researchers at Georgian and the University of Toronto ran Vercel Labs' open-source DeepSec harness with 24 model–runner configurations on a production TypeScript codebase whose vulnerabilities they say were non-public at scan time. No configuration covered all 265 confirmed security root causes, and moving from a minimal prompt to an engineered prompt with CodeQL-derived call-graph context raised criticality-weighted yield by 20–86% for three smaller or cheaper models but by 2% for GPT-5.5. The paper is published in the 2026 CAMLIS proceedings (PMLR 349).
CyberGym
Large benchmark of real vulnerabilities from OSS-Fuzz for evaluating agents on vulnerability reproduction.
Records citing CyberGym
Anthropic's Frontier Red Team reports that Zhipu AI's open-weight GLM-5.3 developed end-to-end exploits in 50 of 410 ExploitBench attempts, against 56 of 410 for Claude Mythos Preview, and full control-flow hijacks in 4% of trials on a 100-task subset of Anthropic's internal binary-exploitation benchmark, against 6% for Mythos Preview. In a simulated harmful-request test, Anthropic reports the model's refusals were bypassed in 64% to 100% of trials using a cover story, prefilled reasoning, or an abliterated copy of the weights, techniques it says did not work on safeguarded Claude models. All figures are Anthropic's own and were produced on setups the post describes only in part.
Google introduced Gemini 3.8 Flash Cyber, a cybersecurity-tuned model with more permissive cyber mitigations, available only to trusted defenders through a new Fairwind Program. Google says it prioritized vulnerability fixing over exploitation and reports 47.2% pass@1 on Collinear's CWE-Bench patching benchmark, over 70% on an internal 20-language discovery benchmark, and 2.6 times more correct Chrome patches than larger commercial models.
Abdelnabi, Hicks, Rieck and Sadeghi argue that security evaluations of agents face three problems: agents can break the benchmark environment instead of solving the task, static benchmarks such as CyberGym and Cybench age as vulnerabilities are patched or leak, and stochastic behavior, agent-written code and external dependencies make single runs unreliable. They propose stronger environment isolation, canary tokens to detect cheating, continually updated or live benchmarks, reporting worst-case results and variance, and benchmark introspection, which they call a holistic first step.
Anthropic launched Project Glasswing with AWS, Apple, Broadcom, Cisco, CrowdStrike, Google, JPMorganChase, the Linux Foundation, Microsoft, NVIDIA and Palo Alto Networks to use the unreleased Claude Mythos Preview for defensive security work, extending access to over 40 more organizations that maintain critical software. Anthropic committed up to $100M in usage credits and $4M in donations to open-source security groups, and reports Mythos Preview found thousands of high-severity vulnerabilities, including in every major operating system and browser.
Anthropic reports that a small team focused Claude Sonnet 4.5 training on finding and patching vulnerabilities and on testing simulated security infrastructure, while avoiding enhancements that clearly favour offence. It reports Sonnet 4.5 results on Cybench and CyberGym, a preliminary patching study in which 15% of patches were judged semantically equivalent to human references, and invites work on SOC and SIEM automation.