Topics/Defense

Vulnerability repair

Automated patching and how to verify that a patch worked.

40 records3 findings1 openings15 benchmarks and toolsLatest record
Start here

Finding and fixing vulnerabilities with AI

AI systems now find real vulnerabilities at scale. Whether their fixes work is less clear.

  1. Google's Big Sleep agent finds exploitable stack buffer underflow in SQLite before release
    The first public real-world find.
  2. AIxCC final: Team Atlanta wins as systems patch 43 of 54 found synthetic bugs and find 18 real ones
    The competition that tested find-and-fix end to end.
  3. Checking only that the original crash no longer reproduces overstates how often AI-generated patches actually fix the vulnerability.
    Why patch counts overstate success.
  4. AI-driven vulnerability discovery produces reports faster than maintainers can triage and patch them.
    The resulting backlog.
  5. What evidence should gate an AI-generated patch?
    The open question.
RangeLanes
36 of 40 records in view

Use the arrow keys to move between records, Home and End to jump to the first and last, and Enter to select one.

Agents find real bugsAgents in real operationsGated capability, incidents in the labAttackCapabilityDefensePolicyJan 25Jul 25Jan 26Jul 26
Full record · drag to choose a range
202420252026

Select a mark to read the record. Mark size shows editorial significance. Hollow marks are dated to the month. Era bands are editorial labels.

Records in view

36 records · newest first
Nov 2026
Nov 13, 2026
Project Zero starts Full 90 disclosure and schedules shorter future deadlines
PolicyGuidanceGoogle Project Zero

Project Zero replaced its disclosure policy with a fixed Full 90 schedule on November 13, citing faster AI-enabled discovery and patch analysis. Full 60 is scheduled for February 2, 2027, and Full 30 for January 3, 2028. These later deadlines are announced changes, not rules already in force in November 2026.

Nov 10, 2026
AlphaProof Code checks specified Signal security properties at machine-code level
DefensePaperGoogle DeepMind, Google

Google DeepMind describes AlphaProof Code, combining human-approved specifications with agent-generated proofs checked by Lean. Its libsignal study proves specified properties of PQXDH and the Triple Ratchet at the AArch64 machine-code level. Assurance depends on the specifications, formal semantics and other stated trust assumptions; it does not cover every property of the Signal application.

Nov 4, 2026
Observational Glasswing study reports more disclosures with limited outcome visibility
DefensePaperQueen Mary University of London

Thomas Mosk’s observational preprint estimates more than doubled public vulnerability disclosures for Glasswing-access firms relative to comparison firms. Measured remediation did not slow among fixed-and-disclosed flaws. A suggestive decline in the exploited share and no statistically detectable count change do not establish fewer attacks; non-random access and limited disclosure visibility constrain the conclusions.

Nov 4, 2026
OpenAI reports Linux discovery, exploit validation and candidate patch pipeline
DefenseEvaluation reportOpenAI

OpenAI describes a Linux research pipeline that produced 2,308 candidate findings, dynamically reproduced 1,724 and developed 948 local privilege-escalation exploits on its stock desktop target. It generated candidate patches for review, rather than reporting that all findings were novel or all fixes had shipped. Hardened-target results are a separate cohort.

Nov 3, 2026
CWE-bench v2 releases inspectable security-repair tasks alongside held-out tests
DefenseBenchmarkCollinear

Collinear released CWE-bench v2 with 100 public repository security-repair tasks covering 91 distinct CWEs, alongside held-out evaluation. Tasks check security and existing functionality, with programmatic graders and human review. Public-set results and the separate combined leaderboard use different denominators, and neither is a direct continuation of v1 scores.

Nov 2, 2026
Trail of Bits reports Codex-assisted backports delivered for 149 Linux CVEs
DefenseEvaluation reportTrail of Bits, OpenAI

Trail of Bits reports that its Codex workflow helped deliver fixes for 149 distinct CVEs into stable Linux kernel releases by October 29, with additional package work for Fedora and Ubuntu. Agents prepare backports and evidence; engineers review and submit them to maintainers. Reported human-time savings exclude setup, regression testing and maintainer acceptance.

Oct 2026
Oct 28, 2026
Google expands Gemini 4 Carbon availability after its phased cyber announcement
CapabilityEvaluation reportGoogle DeepMind

Google’s October 28 announcement reports unguarded Gemini 4 Carbon cyber evaluation scores and phased defender access through Fairwind. Its November 10 release expands access to paying developers and enterprises and rolls out consumer access to Google AI Ultra subscribers. Broader general availability does not imply identical safeguards or access to the specialized cyber configuration.

Oct 25, 2026
CyberDual evaluates AI attackers planting memory-safety defects against AI code reviewers on real maintenance contexts
DefensePaperUniversity of Illinois Urbana-Champaign, Arizona State University

Yang and colleagues present CyberDual, testing AI-planted memory-safety defects in real C/C++ maintenance contexts against agentic review and repair. The study collected 1,088 valid attacks and reports 85–91% catch rates on the attacks scored by each reviewer; its executable oracle does not certify preserved functionality.

Oct 21, 2026
Trail of Bits releases patching and regression-review plugins after finding 21 regressions in merged patches
DefenseTool releaseTrail of Bits, OpenAI

Trail of Bits releases patching-bugs and regression-hunt, plugins developed through its Patch the Planet work with OpenAI. It reports 21 confirmed regressions among more than 130 merged pull requests and preliminary testing in which regression-hunt produced 25% fewer false-positive findings than a baseline Codex review. The comparison used those 21 regressions, and stricter evidence requirements also left some known regressions unconfirmed; patching-bugs itself had not yet received the planned evaluation.

Oct 21, 2026
OpenAI describes coordinating PHP's disclosure of two critical flaws AI review brought to light, and proposes a response framework
DefenseVulnerability disclosureOpenAI, Wiz, Cloudflare

OpenAI describes coordinating fixes for two critical PHP flaws after Calif identified one using AI and Wiz independently found both; PHP released fixes on October 21, and OpenAI proposed a disclosure framework. Follow-up evidence includes CISA’s October 26 listing of both flaws as known exploited and Palo Alto Networks’ report of active exploitation against internet-exposed management interfaces. Wiz’s October 27 report describes declining observed exposure and failed chains in its own captured attempts; this does not establish an absence of exploitation elsewhere or a global compromise count.

Oct 13, 2026
CyberArena benchmark pits 12 frontier agents against each other in live attack-and-defense matches over 466 games
CapabilityBenchmarkUC Berkeley, UC Santa Cruz, Purdue University

A team from 2077AI and several universities presents CyberArena, in which two model-harness entrants host identical vulnerable services in isolated sandboxes and, over three 20-minute rounds, try to capture rotating flags from the opponent while patching their own service under health checks. They report standings for 12 agents over 466 valid games on 40 challenges, with GPT-5.6 Sol at an expected score of 78.6 and Claude Opus 5 at 73.8, and say contested play lowers capture rates relative to solo play. An LLM-run audit of all trajectories reports recurring defender errors such as partial patches and repair claims resting on stale evidence; the authors note it is not human verification of every finding.

Oct 12, 2026
Collinear's CWE-bench: 120 private audit-and-repair tasks, with three of eleven agents tied at 68% pass@1
DefenseBenchmarkCollinear

Collinear researchers introduce CWE-bench, a private benchmark of 120 repository audit-and-repair tasks built on real open-source repositories with injected vulnerabilities across 73 CWE types, eight languages and all ten OWASP Top 10:2025 categories; agents receive a threat model but not the weakness, its location or the number of vulnerable sites. A patch passes only if a programmatic verifier finds the exploits blocked and existing functionality tests passing, and the authors check the verifier against a three-judge agentic panel. They report that Grok 4.7, Gemini 4 Argon and GPT-6 Astra tie at 68% pass@1, that Claude Opus 5.5 reaches 67%, and that pass@4 for the tied agents ranges from 74% to 81%; the results are author-run on a task set outsiders cannot inspect.

Oct 8, 2026
Anthropic launches critical-infrastructure support and free open-source scanning
DefenseProgramAnthropic

Anthropic launches its Cyber Mission with a Critical Infrastructure Defense Program and an opt-in OSS Scanner offering free periodic security scans to open-source maintainers. Scanner reports are model-generated and sent without human review; Anthropic continues human-verified disclosure for projects unable to absorb that volume. The announcement describes efforts and expected performance rather than an independently measured reduction in security risk.

Oct 7, 2026
Microsoft FORGE reports scaled vulnerability discovery and validation bottlenecks
DefenseEvaluation reportMicrosoft

Microsoft’s FORGE reports distinguish internally validated vulnerability findings from maintainer acceptance and shipped fixes. A November 11 Linux follow-up traces 26,502 raw findings to 621 PoC-backed cases, 59 upstream reports, 33 assigned CVEs and 38 reports with fixes merged into mainline. Expert triage, deduplication and maintainer review took months; these cohort-specific figures should not be added to earlier program totals.

Oct 5, 2026
Nonpublic vulnerability tests find attack–repair results depend on the target
DefensePaperCISPA Helmholtz Center for Information Security

Authors affiliated with XOR, Protege and CISPA evaluate attack, repair and subsequent attacks in five nonpublic software environments using researcher-reviewed deterministic graders. Repair scores exceed attack scores in two environments and fall below them in three, while subsequent tests expose failures after an initial exploit is stopped. The selected environments, differing outcome definitions and dependent interactions do not establish a general attacker or defender advantage.

Sep 2026
Sep 30, 2026
Google DeepMind says Gemini 4 Argon is rolling out to trusted cyber defenders and will ship to them without cyber guardrails
CapabilityAccess programGoogle DeepMind, Google, Wiz

Google DeepMind announced Gemini 4 Argon on 2026-09-30 and says it is rolling out to a set of trusted cyber defenders through the Fairwind Program, and says it will release the model without cyber guardrails for those defenders and Google's internal teams. Google claims the model can autonomously find, validate and patch critical vulnerabilities, ties for first on CWE-bench v1 at 68%, and improves on Gemini 3.8 Flash Cyber in vulnerability discovery, without giving numbers for the comparison. It says broad release awaits further safeguard work in four areas (misuse, prompt injection, misalignment and hardening), citing its Frontier Safety Framework for the misuse safeguards; all of these are Google-reported claims that have not been independently verified.

Sep 28, 2026
Artificial Analysis launches a Cyber Index and industry alliance scoring models on source-level vulnerability finding and fixing
DefenseBenchmarkArtificial Analysis, Collinear, IBM

Artificial Analysis announced a Cyber Index Alliance, with Collinear AI, IBM, NVIDIA and Vercel as launch partners, and a v1 Cyber Index that combines three source-code evaluations: repository audit and patch (CWE-Bench-AA, 120 private tasks), finding expert-verified vulnerabilities (DeepsecBench-AA) and discovering, reproducing and patching memory-safety bugs (CyberGym-E2E-AA, 131 tasks). Artificial Analysis says it excludes exploit building, runs all three on its open-source Stirrup harness, and reports safety refusals separately from scores. The article reports failure-mode statistics and says several frontier models refuse almost every CyberGym-E2E-AA task; per-model index scores appear only in charts.

Sep 23, 2026
OpenAI extends its Daybreak cyber-defense access to the Government of Ukraine for civilian infrastructure
CapabilityAccess programOpenAI, ENISA, Ukraine Ministry of Digital Transformation

OpenAI announced on 2026-09-23 that it will give the Government of Ukraine access to its Daybreak program, working with Ukraine's Ministry of Digital Transformation, so Ukrainian teams can find software vulnerabilities and develop and test fixes for civilian infrastructure. OpenAI describes Daybreak as giving defenders access to advanced AI for authorized security work, and says it has provided its cyber models to defenders in France, Germany, Poland and elsewhere in Europe. It reports that ENISA used the models to identify vulnerabilities in software used across EU institutions and that CERT Polska used them to help find six router-software vulnerabilities; these outcomes are OpenAI's account.

Sep 3, 2026
PatchBench finds PoC-only checks inflate AI patching success 1.83x and 25% of patches look memorized
DefenseBenchmarkUniversity of Maryland AI Security Lab

PatchBench, from the University of Maryland's AI Security Lab, evaluates 11 patching agents, including the top three AIxCC systems, on 213 C/C++ tasks whose true fixes lie outside the crash stack, using vulnerability transplant and code mutation to limit memorization. It finds that accepting a patch because the original proof-of-concept no longer crashes inflates solve rates by 1.83x on average, and that about 25% of agent patches closely resemble historical developer fixes.

Sep 2, 2026
Google releases Gemini 3.8 Flash Cyber for trusted defenders, emphasizing automated patching
DefenseTool releaseGoogle, Google DeepMind, Collinear

Google introduced Gemini 3.8 Flash Cyber, a cybersecurity-tuned model with more permissive cyber mitigations, available only to trusted defenders through a new Fairwind Program. Google says it prioritized vulnerability fixing over exploitation and reports 47.2% pass@1 on Collinear's CWE-Bench patching benchmark, over 70% on an internal 20-language discovery benchmark, and 2.6 times more correct Chrome patches than larger commercial models.

Jul 2026
Jul 7, 2026
European Commission presents EU Action Plan on Cybersecurity and Artificial Intelligence
PolicyProgramEuropean Commission, ENISA

The Commission presented an action plan responding to advanced AI models that can both improve and undermine cybersecurity. It plans an EU capacity to evaluate AI models, a European blueprint for structured access to advanced AI capabilities developed with ENISA, a secure ENISA-JRC platform to test AI for cybersecurity, AI-assisted vulnerability fixing, and a campaign to secure critical open-source software. ENISA published its own recommendations for the frontier AI era the same day.

Jun 2026
Jun 22, 2026
Five Eyes cyber agency heads tell leaders AI is shifting cyber risk on a timescale of months
PolicyGuidanceAustralian Signals Directorate (ACSC), Communications Security Establishment, GCSB

The heads of the Five Eyes cyber agencies issued a joint statement that AI is rapidly transforming cyber risk and that organizations must act within months, not years. They ask leaders to reduce attack surface, accelerate patching as exploitation windows shorten, replace unsupported legacy systems, strengthen identity controls, and prepare for incidents.

Jun 2, 2026
Executive Order 14409 creates classified cyber benchmarking for covered frontier models and a clearinghouse
PolicyRegulationThe White House, US Department of the Treasury, NSA

Executive Order 14409 directs Treasury, NSA and CISA to develop a classified benchmarking process to assess advanced cyber capabilities of AI models and designate covered frontier models, with a voluntary framework for pre-release government and trusted-partner access. It also orders an AI cybersecurity clearinghouse to coordinate vulnerability scanning, validation and remediation with industry, and states it does not create mandatory licensing or pre-clearance.

May 2026
May 22, 2026
Glasswing discovery and disclosure updates distinguish reported bugs from known upstream patches
DefenseProgramAnthropic, Cloudflare, Mozilla

Anthropic’s May Glasswing update reports high- and critical-severity discovery outpacing triage and patching. Its November 2 disclosure dashboard records 5,087 reports across 680 projects and 629 known upstream patches across all severities and multiple Claude models. Different cohorts, triage routes and incomplete patch visibility prevent treating these snapshots as a comparable patch-rate trend.

Mar 2026
Mar 9, 2026
OSS-CRS makes AIxCC reasoning systems runnable locally; OpenSSF adopts it as a sandbox project
DefenseTool releaseGeorgia Institute of Technology, Microsoft, Team Atlanta

Researchers led by Georgia Tech released OSS-CRS, a locally deployable framework for running and combining AIxCC cyber reasoning systems, noting that all seven open-sourced finalist systems depended on competition cloud infrastructure that no longer exists. Porting the winning Atlantis system, they found 10 previously unknown bugs (three high severity) in 8 OSS-Fuzz projects; OpenSSF welcomed OSS-CRS into its AI/ML Security Working Group in April 2026.

Mar 6, 2026
OpenAI relaunches Aardvark as Codex Security, reporting 1.2M commits scanned and 14 CVEs
DefenseTool releaseOpenAI, OpenSSH, GnuTLS

OpenAI renamed Aardvark to Codex Security and opened a research preview to ChatGPT Pro, Enterprise, Business and Edu customers. OpenAI reports that in 30 days it scanned over 1.2 million commits in its beta cohort and flagged 792 critical and 10,561 high-severity findings, that beta changes cut false positives by more than 50%, and that its open-source reports led to 14 CVEs.

Feb 2026
Feb 20, 2026
Anthropic releases Claude Code Security in limited preview to scan code and propose patches
DefenseTool releaseAnthropic

Anthropic released Claude Code Security as a limited research preview for Enterprise and Team customers, with expedited free access for open-source maintainers. The tool reasons about data flow across a codebase, re-examines each finding in a multi-stage verification pass, assigns severity and confidence ratings, and proposes patches that are applied only with human approval.

Feb 7, 2026
AIxCC SoK finds stability decided results and many validated AI patches were still semantically wrong
DefensePaperGeorgia Institute of Technology, Texas A&M University, DARPA

A systematization-of-knowledge paper by organizers and competitors analyzes AIxCC's design, the seven finalist architectures and results beyond the scoreboard. It reports that system stability and accuracy penalties decided rankings, that LLM-based systems found vulnerabilities a fuzzing baseline missed, and that among patches passing all automatic validation, manual review found semantic errors in 38-46% from baseline agents; the top two systems had 83.8% and 79.2% competition-scored patch accuracy.

Feb 5, 2026
Anthropic reports over 500 human-validated high-severity open-source vulnerabilities found with Claude Opus 4.6
DefensePaperAnthropic, Ghostscript, OpenSC

Anthropic's Frontier Red Team reports that Claude Opus 4.6, run in a VM with standard tools but no custom harness, found and validated more than 500 high-severity vulnerabilities in open-source software, focusing on memory corruption that can be confirmed with sanitizers. Every bug was validated before reporting, initially by Anthropic researchers who also wrote patches and later with external researchers; examples include Ghostscript, OpenSC and CGIF.

Oct 2025
Oct 30, 2025
OpenAI announces Aardvark, a GPT-5 agent that finds, validates and proposes patches for vulnerabilities
DefenseTool releaseOpenAI

OpenAI announced Aardvark, a GPT-5-powered agent in private beta that builds a threat model of a repository, scans commits, tries to trigger suspected flaws in a sandbox, and attaches Codex-generated patches for human review. OpenAI reports 92% recall on known and synthetically introduced vulnerabilities in its 'golden' repositories and ten CVEs from open-source scanning, and planned pro-bono scanning for some non-commercial projects.

Oct 6, 2025
Google DeepMind introduces CodeMender, an agent that patches and hardens code, with 72 upstreamed fixes
DefenseTool releaseGoogle DeepMind, Google

Google DeepMind introduced CodeMender, an agent built on Gemini Deep Think models that combines static and dynamic analysis, fuzzing, differential testing and SMT solvers with LLM-based critique to generate and validate security patches. DeepMind reports 72 security fixes upstreamed to open-source projects over six months, all reviewed by human researchers before submission; in May 2026 Google said it would fold CodeMender into its enterprise agent platform.

Oct 3, 2025
Anthropic says it trained Claude Sonnet 4.5 for defensive vulnerability finding and patching
DefensePaperAnthropic, HackerOne, CrowdStrike

Anthropic reports that a small team focused Claude Sonnet 4.5 training on finding and patching vulnerabilities and on testing simulated security infrastructure, while avoiding enhancements that clearly favour offence. It reports Sonnet 4.5 results on Cybench and CyberGym, a preliminary patching study in which 15% of patches were judged semantically equivalent to human references, and invites work on SOC and SIEM automation.

Aug 2025
Aug 8, 2025
AIxCC final: Team Atlanta wins as systems patch 43 of 54 found synthetic bugs and find 18 real ones
DefenseCompetitionDARPA, ARPA-H, Team Atlanta

DARPA reports that seven finalist cyber reasoning systems analyzed over 54 million lines of code, found 54 unique synthetic vulnerabilities in 63 challenges and patched 43, and found 18 real non-synthetic vulnerabilities with 11 patches. Team Atlanta won $4 million, Trail of Bits $3 million and Theori $1.5 million; DARPA and ARPA-H added $1.4 million for real-world integration and four systems were open-sourced on the day.

Jun 2025
Jun 13, 2025
SEC-bench automatically builds real vulnerability tasks and finds agents patch at most 34%
DefenseBenchmarkUniversity of Illinois Urbana-Champaign, Purdue University

SEC-bench uses multi-agent scaffolding to construct reproducible vulnerability instances with test environments and validated patches from real projects, at about $0.87 per instance. The authors report that LLM agents reached at most 18.0% on proof-of-concept generation and 34.0% on vulnerability patching.

May 2025
May 21, 2025
BountyBench measures AI agents on detect, exploit and patch tasks from real bug bounties
DefenseBenchmarkStanford University, UC Berkeley

BountyBench, from Stanford-led researchers, builds 40 bug bounties across 25 real-world systems into 120 Detect, Exploit and Patch tasks with dollar values attached. In the first version the best Detect score was 5%, while OpenAI Codex CLI and Claude Code scored 90% and 87.5% on Patch, well above their Exploit scores. A July 2025 revision with more agents reported Codex CLI with o3-high at 12.5% on Detect and 90% on Patch.

Apr 2025
Apr 29, 2025
Meta releases AutoPatchBench to test AI repair of fuzzing-found C/C++ vulnerabilities
DefenseBenchmarkMeta

Meta introduced AutoPatchBench, part of CyberSecEval 4, with 136 fuzzing-identified C/C++ vulnerabilities and verified fixes, plus a 113-case Lite subset with single-function root causes. Patches are checked by build and crash reproduction, then fuzzing and white-box differential testing; Meta's reference agent generated crash-stopping patches in about 60% of cases, but only 5-11% passed the stricter checks.

Findings

Research openings

Benchmarks and tools