CyberGym

Large benchmark of real vulnerabilities from OSS-Fuzz for evaluating agents on vulnerability reproduction.

Records citing CyberGym

Oct 20, 2026
24 model setups: an engineered prompt with code-graph context lifts smaller models' vulnerability finding, not a frontier model's
CapabilityPaperUniversity of Toronto

Researchers at Georgian and the University of Toronto ran Vercel Labs' open-source DeepSec harness with 24 model–runner configurations on a production TypeScript codebase whose vulnerabilities they say were non-public at scan time. No configuration covered all 265 confirmed security root causes, and moving from a minimal prompt to an engineered prompt with CodeQL-derived call-graph context raised criticality-weighted yield by 20–86% for three smaller or cheaper models but by 2% for GPT-5.5. The paper is published in the 2026 CAMLIS proceedings (PMLR 349).

Sep 29, 2026
Anthropic assesses open-weight GLM-5.3: exploit results near Mythos Preview, safeguards bypassed in 64% to 100% of simulated trials
CapabilityEvaluation reportAnthropic, US Center for AI Standards and Innovation, Moonshot AI

Anthropic's Frontier Red Team reports that Zhipu AI's open-weight GLM-5.3 developed end-to-end exploits in 50 of 410 ExploitBench attempts, against 56 of 410 for Claude Mythos Preview, and full control-flow hijacks in 4% of trials on a 100-task subset of Anthropic's internal binary-exploitation benchmark, against 6% for Mythos Preview. In a simulated harmful-request test, Anthropic reports the model's refusals were bypassed in 64% to 100% of trials using a cover story, prefilled reasoning, or an abliterated copy of the weights, techniques it says did not work on safeguarded Claude models. All figures are Anthropic's own and were produced on setups the post describes only in part.

Sep 2, 2026
Google releases Gemini 3.8 Flash Cyber for trusted defenders, emphasizing automated patching
DefenseTool releaseGoogle, Google DeepMind, Collinear

Google introduced Gemini 3.8 Flash Cyber, a cybersecurity-tuned model with more permissive cyber mitigations, available only to trusted defenders through a new Fairwind Program. Google says it prioritized vulnerability fixing over exploitation and reports 47.2% pass@1 on Collinear's CWE-Bench patching benchmark, over 70% on an internal 20-language discovery benchmark, and 2.6 times more correct Chrome patches than larger commercial models.

May 21, 2026
Position paper argues agent security benchmarks suffer from hackable environments, staleness and runtime noise
DefensePaperarXiv

Abdelnabi, Hicks, Rieck and Sadeghi argue that security evaluations of agents face three problems: agents can break the benchmark environment instead of solving the task, static benchmarks such as CyberGym and Cybench age as vulnerabilities are patched or leak, and stochastic behavior, agent-written code and external dependencies make single runs unreliable. They propose stronger environment isolation, canary tokens to detect cheating, continually updated or live benchmarks, reporting worst-case results and variance, and benchmark introspection, which they call a holistic first step.

Apr 7, 2026
Anthropic launches Project Glasswing to give defenders early access to Claude Mythos Preview
DefenseProgramAnthropic, Amazon Web Services, Apple

Anthropic launched Project Glasswing with AWS, Apple, Broadcom, Cisco, CrowdStrike, Google, JPMorganChase, the Linux Foundation, Microsoft, NVIDIA and Palo Alto Networks to use the unreleased Claude Mythos Preview for defensive security work, extending access to over 40 more organizations that maintain critical software. Anthropic committed up to $100M in usage credits and $4M in donations to open-source security groups, and reports Mythos Preview found thousands of high-severity vulnerabilities, including in every major operating system and browser.

Oct 3, 2025
Anthropic says it trained Claude Sonnet 4.5 for defensive vulnerability finding and patching
DefensePaperAnthropic, HackerOne, CrowdStrike

Anthropic reports that a small team focused Claude Sonnet 4.5 training on finding and patching vulnerabilities and on testing simulated security infrastructure, while avoiding enhancements that clearly favour offence. It reports Sonnet 4.5 results on Cybench and CyberGym, a preliminary patching study in which 15% of patches were judged semantically equivalent to human references, and invites work on SOC and SIEM automation.