Private benchmark of 120 repository audit-and-repair tasks with injected vulnerabilities, graded by security and functionality verifiers.
Google announces Gemini 4 Carbon with vendor-reported scores of 88.2% on SEC-bench Pro, 54.8% on ExploitGym and 71.0% on CWE-bench, evaluated without guardrails. It describes phased access for selected cyber defenders through Fairwind, with broader product availability planned after further safeguard work.
Collinear researchers introduce CWE-bench, a private benchmark of 120 repository audit-and-repair tasks built on real open-source repositories with injected vulnerabilities across 73 CWE types, eight languages and all ten OWASP Top 10:2025 categories; agents receive a threat model but not the weakness, its location or the number of vulnerable sites. A patch passes only if a programmatic verifier finds the exploits blocked and existing functionality tests passing, and the authors check the verifier against a three-judge agentic panel. They report that Grok 4.7, Gemini 4 Argon and GPT-6 Astra tie at 68% pass@1, that Claude Opus 5.5 reaches 67%, and that pass@4 for the tied agents ranges from 74% to 81%; the results are author-run on a task set outsiders cannot inspect.